Include certain url structures as filter

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
Guyserbun007
Posts: 1
Joined: Tue Aug 19, 2014 9:43 pm

Include certain url structures as filter

Post by Guyserbun007 »

I would like to scrape the valid urls from a site following a specific formats, as follows:

1) http://www.example.com/201(****), where (****) could be any character and any length.
For examples: http://www.example.com/2014/12/06 or http://www.example.com/2013/joo/put%324-242

2) Furthermore, I don't want pages that either i) not follow that structure (i.e. http://www.example.com/2000/12/06) or ii) follow that structure but is not a valid page

How can I specify that in the filter before the scan?

Thanks.

User avatar
Support
Site Admin
Posts: 1892
Joined: Sun Oct 02, 2011 10:49 am

Re: Include certain url structures as filter

Post by Support »

Only valid pages will be listed in the structure. Here is what it looks like in the Filters...
Attachments
2014-08-19_215039.png
2014-08-19_215039.png (78.99 KiB) Viewed 7842 times
Your support team.
http://SoftByteLabs.com

Post Reply