Help with filter - can't get list of files in a

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
patmc4fun
Posts: 2
Joined: Mon Aug 13, 2012 3:16 pm

Help with filter - can't get list of files in a

Post by patmc4fun » Mon Aug 13, 2012 5:08 pm

I am trying to download an archive of daily pictures from http://www.staremagazine.com/images/temp/_today/ . Unfortunately, if I try to get a list of all the files in that directory, I get the error "The website declined to show this webpage". Yet, when I specify a particular file like: http://www.staremagazine.com/images/tem ... .09.01.jpg , where there is a specific file named... I can find that file.

The filenames are all logical, date based file names. Is there any way to create a filter based on a date-based filename structure... where it can increment the date on the file? I've tried setting the scan for whole site, and just for that directory, but I keep getting the error when I try any kind of search for all the files in the directory. Is there any way to get a list and/or download all the files at that location? Sorry if I'm missing something basic... this is all kind of new to me.

Thanks!

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: Help with filter - can't get list of files in a

Post by Support » Mon Aug 13, 2012 5:19 pm

Here is a script that will pull all the images for 2012 (from jan 1 to dec 31). In the Filters window, click on the Expert button and paste the following script in the editor, then start the scan, or, first change the year, month, day range.

Code: Select all

case ScannerEvent of

	Starting:
	begin
		for y = 2012 to 2012 do
			for m = '01' to '12' range('0'..'9') do
				for d = '01' to '31' range('0'..'9') do begin
					aLink = 'http://www.staremagazine.com/images/temp/_today/STAREMagazine.com_'+y+'.'+m+'.'+d+'.jpg';
					Scanlink(aLink); // add the link to the scan queue.
				end;
	end;

	BeforeAdding:
	begin
		AcceptEvent = (DocumentType ~= 'image/'); // add any kind of images.
	end;

else
	AcceptEvent = No;

end;
Your support team.
http://SoftByteLabs.com

patmc4fun
Posts: 2
Joined: Mon Aug 13, 2012 3:16 pm

Re: Help with filter - can't get list of files in a

Post by patmc4fun » Mon Aug 13, 2012 5:30 pm

Wow! It works like a charm! Thank you for the VERY quick response... It is greatly appreciated!

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: Help with filter - can't get list of files in a

Post by Support » Mon Aug 13, 2012 5:40 pm

You are welcome
Your support team.
http://SoftByteLabs.com

quinto1
Posts: 3
Joined: Thu Nov 08, 2012 11:33 am

Re: Help with filter - can't get list of files in a

Post by quinto1 » Thu Nov 08, 2012 11:34 am

Hi - thanks for the code... which product are you using to perform this? thanks

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: Help with filter - can't get list of files in a

Post by Support » Thu Nov 08, 2012 11:48 am

This code is to be used in BlackWidow.
Your support team.
http://SoftByteLabs.com

quinto1
Posts: 3
Joined: Thu Nov 08, 2012 11:33 am

Re: Help with filter - can't get list of files in a

Post by quinto1 » Thu Nov 08, 2012 2:08 pm

thanks.... would Blackwidow be able to identify unlisted directory structures (URLs) as well? for instance:

COuld it identify folder structures below something like...

http://espn.com/images


would it be able to pull results like:

http://espn.com/images/1
http://espn.com/images/2/2010

etc..etc..

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: Help with filter - can't get list of files in a

Post by Support » Thu Nov 08, 2012 2:19 pm

No, unless you know the file names. BW can only follow links it finds, otherwise, you'll have to give it a list of links.
Your support team.
http://SoftByteLabs.com

quinto1
Posts: 3
Joined: Thu Nov 08, 2012 11:33 am

Re: Help with filter - can't get list of files in a

Post by quinto1 » Thu Nov 08, 2012 3:03 pm

how would i go about grabbing every ".JPG" using an expert filter?

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: Help with filter - can't get list of files in a

Post by Support » Thu Nov 08, 2012 3:08 pm

You do not need to use the Expert Filter to grab all the JPG from a site. Simply add a filter as to what you want to add into the structure, in this case, a regular expression of \.jpg$ will do just that.
Your support team.
http://SoftByteLabs.com

Post Reply