How to fusker in new version??

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

How to fusker in new version??

Post by stjernquist »

Hmmm.... I just updated my BW to ver. 6.

Now I can't figure out how to fusker anymore.

The old version had the link-creator which could be used for fuskering.

I can't find anything similar in this version.

How would you make a fusker of e.g. this string:

http://www.site/image.php3?id=113367240 ... 1245988000}

It is a fusker for html and it does if you hit the right url have a mime of a jpeg picture hidden within the url.

So the extension is not in the fusker but in the mime.

How would I do this in version 6?? :?: :?: :!:

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

The way it works in v6 is by using the Expert filter. In the Filters window, click on the Expert button in the top/right corner.

Now, you can clear the script editor window, and paste the following in it...

Code: Select all

case ScannerEvent of

  Starting:
  begin
    for x = 1245985000 to 1245988000 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
    end;
  end;

  BeforeAdding:
  begin
    AcceptEvent = (DocumentType ~= 'image');
  end;

else
  AcceptEvent = No;

end;
Make sure to edit the URL so it's the correct one, then click on Scanner and start the scan. Any of the links that is an image will be added to the Structure list.

Please let me know if this is what you needed or not.
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

:D :D
Thanks Admin,

This script works fine :)

However I do have some issues that you might can help with.

I intent to use the script to scan a wide range of urls. In ver 5 I could filter out the 'bad' urls, meaning the urls without images.
I could put in a size filter, that filtered out files smaller than e.g. 10 kb. How do I do this in ver 6?

In ver 5 it was also relatively simple to make steps in the fusker e.g; {x,1167623836,1199159836,9}
How do I make steps in ver 6??

And if possible can you provide a script that does the follwoing: http://www.site/image.php3?id=113367240 ... }(scanning 400 of 3600 urls)( then steps 79200 urls (urls) and makes the operation {x,1199228401,1199235601,9} (scanning 400 of 600 urls, then steps another 79200 urls and starts scanning the next 3600 urls with a step of 9?)and so on until end url that would be 1201741201 in this case.

I am not sure wheter this is understandable, english is not my prime language. But as you probably already know I am trying to fusker
epoch time. So what I basically want is to fusker the hours from e.g. 00.00.01 - 02.00.01 with a stepcount of 9. then skip the next 22 hours and then fusker the 2 hours with a step of 9, then skip another 22 hours and so on for a month. So in this case I would reduce my calculated randomness of the time interval from 111600 urls to only 12400 urls (or if I would have to fusker all urls 2678400!!! :o ).

I hope you can see what I want to do and that this is possible. :?

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

Hi,

To also filter out smaller images, change the following line...

Code: Select all

AcceptEvent = (DocumentType ~= 'image');
for this and modify the file size if you want...

Code: Select all

AcceptEvent = (DocumentType ~= 'image') and (DocumentSize > 10000);

To step as per your example {x,1167623836,1199159836,9}, change the following line...

Code: Select all

for x = 1245985000 to 1245988000 do begin
for the following...

Code: Select all

for x = 1167623836 to 1199159836 step 9 do begin
Let me check your next question and I'll post the answer in a new reply...
Your support team.
http://SoftByteLabs.com

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote:And if possible can you provide a script that does the follwoing: http://www.site/image.php3?id=113367240 ... }(scanning 400 of 3600 urls)( then steps 79200 urls (urls) and makes the operation {x,1199228401,1199235601,9} (scanning 400 of 600 urls, then steps another 79200 urls and starts scanning the next 3600 urls with a step of 9?)and so on until end url that would be 1201741201 in this case.

I am not sure wheter this is understandable, english is not my prime language. But as you probably already know I am trying to fusker
epoch time. So what I basically want is to fusker the hours from e.g. 00.00.01 - 02.00.01 with a stepcount of 9. then skip the next 22 hours and then fusker the 2 hours with a step of 9, then skip another 22 hours and so on for a month. So in this case I would reduce my calculated randomness of the time interval from 111600 urls to only 12400 urls (or if I would have to fusker all urls 2678400!!! :o ).

Sure, I can do that. But first let me ask, is there a pattern for the big numbers? Is it like yyyymmddhh or something like that?
Your support team.
http://SoftByteLabs.com

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote:And if possible can you provide a script that does the follwoing: http://www.site/image.php3?id=113367240 ... }(scanning 400 of 3600 urls)( then steps 79200 urls (urls) and makes the operation {x,1199228401,1199235601,9} (scanning 400 of 600 urls, then steps another 79200 urls and starts scanning the next 3600 urls with a step of 9?)and so on until end url that would be 1201741201 in this case.

I am not sure wheter this is understandable, english is not my prime language. But as you probably already know I am trying to fusker
epoch time. So what I basically want is to fusker the hours from e.g. 00.00.01 - 02.00.01 with a stepcount of 9. then skip the next 22 hours and then fusker the 2 hours with a step of 9, then skip another 22 hours and so on for a month. So in this case I would reduce my calculated randomness of the time interval from 111600 urls to only 12400 urls (or if I would have to fusker all urls 2678400!!! :o ).

You can write some test condition in the loop. Change the following...

Code: Select all

  Starting:
  begin
    for x = 1245985000 to 1245988000 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
    end;
  end;
for somehting like this...

Code: Select all

  Starting:
  begin

    x = 1199142001;
    for c = 1 to 400 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

    x = x + 79200;
    for c = 1 to 400 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

    x = x + 79200;
    for c = 1 to 3600 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

  end;
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:
stjernquist wrote:And if possible can you provide a script that does the follwoing: http://www.site/image.php3?id=113367240 ... }(scanning 400 of 3600 urls)( then steps 79200 urls (urls) and makes the operation {x,1199228401,1199235601,9} (scanning 400 of 600 urls, then steps another 79200 urls and starts scanning the next 3600 urls with a step of 9?)and so on until end url that would be 1201741201 in this case.

I am not sure wheter this is understandable, english is not my prime language. But as you probably already know I am trying to fusker
epoch time. So what I basically want is to fusker the hours from e.g. 00.00.01 - 02.00.01 with a stepcount of 9. then skip the next 22 hours and then fusker the 2 hours with a step of 9, then skip another 22 hours and so on for a month. So in this case I would reduce my calculated randomness of the time interval from 111600 urls to only 12400 urls (or if I would have to fusker all urls 2678400!!! :o ).

Sure, I can do that. But first let me ask, is there a pattern for the big numbers? Is it like yyyymmddhh or something like that?

Hmmm.... I am not excatly sure what you means with pattern - I put in this link, here you can see the pattern and also additional info about the format and the epoch time.

http://www.esqsoft.com/javascript_examp ... -epoch.htm

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:Hi,

To also filter out smaller images, change the following line...

Code: Select all

AcceptEvent = (DocumentType ~= 'image');
for this and modify the file size if you want...

Code: Select all

AcceptEvent = (DocumentType ~= 'image') and (DocumentSize > 10000);

To step as per your example {x,1167623836,1199159836,9}, change the following line...

Code: Select all

for x = 1245985000 to 1245988000 do begin
for the following...

Code: Select all

for x = 1167623836 to 1199159836 step 9 do begin
Let me check your next question and I'll post the answer in a new reply...

First: The stepcount seems to be working great - thanks a lot! :D

However the filefiltering is not so great - I still see urls containg less than 10 kb - it valids the url even though the url is 0 kb. The mime extension for this is text/html and I only want the mime containing image/jpeg - so I need some additional help here :?:

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

Code: Select all

AcceptEvent = (DocumentType ~= 'image/jpeg') and (DocumentSize > 10000);
The above will get you only the jpg. If you get images less than 10kb is because the website is not telling us what the size is!
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:
stjernquist wrote:And if possible can you provide a script that does the follwoing: http://www.site/image.php3?id=113367240 ... }(scanning 400 of 3600 urls)( then steps 79200 urls (urls) and makes the operation {x,1199228401,1199235601,9} (scanning 400 of 600 urls, then steps another 79200 urls and starts scanning the next 3600 urls with a step of 9?)and so on until end url that would be 1201741201 in this case.

I am not sure wheter this is understandable, english is not my prime language. But as you probably already know I am trying to fusker
epoch time. So what I basically want is to fusker the hours from e.g. 00.00.01 - 02.00.01 with a stepcount of 9. then skip the next 22 hours and then fusker the 2 hours with a step of 9, then skip another 22 hours and so on for a month. So in this case I would reduce my calculated randomness of the time interval from 111600 urls to only 12400 urls (or if I would have to fusker all urls 2678400!!! :o ).

You can write some test condition in the loop. Change the following...

Code: Select all

  Starting:
  begin
    for x = 1245985000 to 1245988000 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
    end;
  end;
for somehting like this...

Code: Select all

  Starting:
  begin

    x = 1199142001;
    for c = 1 to 400 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

    x = x + 79200;
    for c = 1 to 400 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

    x = x + 79200;
    for c = 1 to 3600 do begin
      lnk = 'http://www.site/image.php3?id=1133672408.'+x;
      Scanlink(lnk);
      x = x + 9;
    end;

  end;
Now this seem to work to some extent. However, it seems like the script in this form only creates 4400 urls to scan which would equal like 11 days instead of a month. I can't see any maximum no. for the range e.g from 1199142001 to 1282705201 so how do I control this feature??

Finally I must express my gratitude for your very fast answer which is what I call super-support ;)

Thanks a lot admin!!

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote:Now this seem to work to some extent. However, it seems like the script in this form only creates 4400 urls to scan which would equal like 11 days instead of a month. I can't see any maximum no. for the range e.g from 1199142001 to 1282705201 so how do I control this feature??
There is no ending number, just a starting number and a count from that.

x = 1199142001; is the starting number.
for c = 1 to 400 do begin is the count, so the ending will be 1199142001 + 400 = 1199142401

Then at the end of that loop, we have x = x + 79200; which adds 79200 to our previous result of 1199142401, so x is now 1199221601 and once more, we scan another 400 from that number.

All you have to do is change the numbers to what you need. I'm still not sure what 1199142001 means, is it days? hours?
stjernquist wrote:Finally I must express my gratitude for your very fast answer which is what I call super-support
You are welcome :D
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:
stjernquist wrote:Now this seem to work to some extent. However, it seems like the script in this form only creates 4400 urls to scan which would equal like 11 days instead of a month. I can't see any maximum no. for the range e.g from 1199142001 to 1282705201 so how do I control this feature??
There is no ending number, just a starting number and a count from that.

x = 1199142001; is the starting number.
for c = 1 to 400 do begin is the count, so the ending will be 1199142001 + 400 = 1199142401

Then at the end of that loop, we have x = x + 79200; which adds 79200 to our previous result of 1199142401, so x is now 1199221601 and once more, we scan another 400 from that number.

All you have to do is change the numbers to what you need. I'm still not sure what 1199142001 means, is it days? hours?
stjernquist wrote:Finally I must express my gratitude for your very fast answer which is what I call super-support
You are welcome :D

:shock: WoW you are fast!

My english is not good enough to explain the epoch system for you.

But if you go to this url : http://www.esqsoft.com/javascript_examp ... -epoch.htm

Here is a nice explanation and also some javascripts that might interest a programmer like yourself. :ugeek:


However I am still experiencing problems with your script.

BW tells me that there is only 4400 in the range provided with my script.

I probably made some kind of error. Here is my testscript:

[BlackWidow v6.00 filters]
URL = http://upload.www.site/fetchimg.php3?id ... 1226807257
[x] Expert mode
[ ] Slow down by 10:60 seconds
20 threads
[expert script]
Starting;
begin

x = 1226807131;
for c = 1 to 400 do begin
lnk = 'http://www.site/image.php3?id=1133672408.'+x;
Scanlink(lnk);
x = x + 9;
end;

x = x + 79200;
for c = 1 to 400 do begin
lnk = 'http://www.site/image.php3?id=1133672408.'+x;
Scanlink(lnk);
x = x + 9;
end;

x = x + 79200;
for c = 1 to 3600 do begin
lnk = 'http://www.site/image.php3?id=1133672408.'+x;
Scanlink(lnk);
x = x + 9;
end;

end;

BeforeAdding;
begin
AcceptEvent = (DocumentType ~= 'image') and (DocumentSize > 10000);
end;

else;
AcceptEvent = No
[end expert script]
[end]


Maybe you from this can see the error??


And I am not quite sure you really understood what I want:

1 sec. in epoch is a count of 1
so 1 hour will be the count of 3600 units/urls

So if I want to fusker e.g the hours between midnight and 2. am this will require a scan of 7200 urls.

However, I only want to see if I get any hits and I do not need to test every second. I am satisfied with every 9 second. If I get anything interesting here, I can always rescan with different settings getting all images. This will reduce an hour from 3600 scans to 400.

After this I want to go 22 hours ahead so I start a new scan at midnight the following day (that is the step of 3600 x 22 = 79200) and so on.

It is very good there is no ending number - this is the best!

Now I will make a test to see if I can filter the htms out of my downloads.

I look forward to hear from you and do hope you understand my poor english :roll:

Kind regards

Stjernquist

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Hmmm... I am still having filter problems :(

BW still downloads all urls unless the size and format, here both html and jpg (and yes I did try both jpeg and jpg in filter options)

A strange thing is:

If I do fusker in a website like : http://members.home.nl/bdr/files/fusker.html the fileextension will look like this: 1133672408.1226807131.jpg, which is usefull because the .1226807131 tells me the date and time of the image.

In BW when downloading I get this format: pcimage.php3.3A7630F6.jpg

It is probably some kind of autogenerated filenaming.

Is there anyway to bypass this?? (Its not critical, it would just be nice because I prefer the other format)


I pm you the full url I am trying to fusker - that might help you out in helping me :P

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote::shock: WoW you are fast!

My english is not good enough to explain the epoch system for you.

But if you go to this url : http://www.esqsoft.com/javascript_examp ... -epoch.htm

Yes, I know exactly what you mean now. So I made you a script that all you have to do is enter the start date/time and the end date/time and how many seconds to step, and the script will do the rest. I think this will work for you. Let me know?

Code: Select all

YearStart   = 2009;
MonthStart  = 6;
DayStart    = 25;
HourStart   = 21;
MinuteStart = 0;
SecondStart = 0;

YearEnd   = 2009;
MonthEnd  = 6;
DayEnd    = 25;
HourEnd   = 22;
MinuteEnd = 0;
SecondEnd = 0;

EpochStep = 9;

BaseURL = 'http://www.site/image.php3?id=1133672408.';

//-----------------------------------------------------------------------------------

function NumDaysInMonth(m,y);
begin
  case m of
    4,6,9,11:
      Result = 30;
    2: begin
      Result = 29;
      if (not((y mod 4 = 0)and(y mod 100 <> 0)) or (y mod 400 = 0)) then Result--;
    end;
  else
    Result = 31;
  end;
end;

//-----------------------------------------------------------------------------------

function ToEpoch(y,m,d,h,n,s);
begin
  Result = 0;
  for x = 1970 to y-1 do
    for c = 1 to 12 do
      Result += NumDaysInMonth(c,x) * 86400;
  for c = 1 to m-1 do
    Result += NumDaysInMonth(c,y) * 86400;
  Result += d*86400 + h*3600 + n*60 + s;
end;

//-----------------------------------------------------------------------------------

EpochStart = ToEpoch(YearStart,MonthStart,DayStart,HourStart,MinuteStart,SecondStart);
EpochEnd = ToEpoch(YearEnd,MonthEnd,DayEnd,HourEnd,MinuteEnd,SecondEnd);

//-----------------------------------------------------------------------------------

case ScannerEvent of

  Starting:
  begin
    for x = EpochStart to EpochEnd step EpochStep do begin
      lnk = BaseURL + x;
      Scanlink(lnk);
    end;
  end;

  BeforeAdding:
  begin
    AcceptEvent = (DocumentType ~= 'image/jpeg') and (DocumentSize > 10*1024);
  end;

else
  AcceptEvent = No;

end;

//-----------------------------------------------------------------------------------
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

:D :D :D :D

We are almost there.

This new script is so much better - it seems to be able to fusker real fast!

However, now it fusker all hours of the day and not only 2 hours with a 9 count step.

Is it possible you forgot to add the 79200 count step?? Or would this just not be possible to do - I mean having 2 different, but yet depending on each other, steps in the same script??

And.... it still downloads all kind of files and sizes, but that is not a big issue. I can select download after done scanning. Is there an easy way to just filter the images from the htms after scanning and before downloading??

Again - thanks a lot for your effort and your dedication to this project :D

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote::D :D :D :D

We are almost there.

This new script is so much better - it seems to be able to fusker real fast!

However, now it fusker all hours of the day and not only 2 hours with a 9 count step.

Is it possible you forgot to add the 79200 count step?? Or would this just not be possible to do - I mean having 2 different, but yet depending on each other, steps in the same script??

And.... it still downloads all kind of files and sizes, but that is not a big issue. I can select download after done scanning. Is there an easy way to just filter the images from the htms after scanning and before downloading??

Again - thanks a lot for your effort and your dedication to this project :D

Kind regards

Stjernquist

Not a problem. So you need to step 9 for 2 hours (3600), and then step 79200?
Give me the real numbers you need to scan and I'll make it work.

The reason it's picking up all images is because you need to use the URL of the actual image location instead, I sent you a PM with the link.
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Okay - here is an example of what I want to fusker.

The range would be june 2007, between the hours of 00.00.01 and 02.00.00 with a step of 9 secs.

In epoch the numbers will be:
From 1180648801 to 1183154401

Is this what you need??

Btw. changing the url to the real img.url doesn't seem to help me!

I still get all urls??

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote:Okay - here is an example of what I want to fusker.

The range would be june 2007, between the hours of 00.00.01 and 02.00.00 with a step of 9 secs.

In epoch the numbers will be:
From 1180648801 to 1183154401

Is this what you need??

Btw. changing the url to the real img.url doesn't seem to help me!

I still get all urls??

Kind regards

Stjernquist

Yes, but if you need to scan 2 hours, that's 7200 secs, and stepping by 9 (7200/9=800) means 800 URL will be scanned, so the ending number should be 1180648801+7200=1180656001 instead of 1183154401, right?
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:
stjernquist wrote:Okay - here is an example of what I want to fusker.

The range would be june 2007, between the hours of 00.00.01 and 02.00.00 with a step of 9 secs.

In epoch the numbers will be:
From 1180648801 to 1183154401

Is this what you need??

Btw. changing the url to the real img.url doesn't seem to help me!

I still get all urls??

Kind regards

Stjernquist

Yes, but if you need to scan 2 hours, that's 7200 secs, and stepping by 9 (7200/9=800) means 800 URL will be scanned, so the ending number should be 1180648801+7200=1180656001 instead of 1183154401, right?
No - that would be wrong. In that case I would only scan the a range of files between june 1st 2007 at 00.00.01 and june 01. 2007 at 02.00.01. This would be fine for the first scan but with a span/step of 9 units so only 400 urls would be scanned. After this, it should jump 79200 units and start a new scan from this point being 1180735201 scanning every 9 urls/units untill 1180742401, then jump 79200 urls and scan every 9th url from the point 1180821601 untill 1180828801 etc. This process should be proceeded as many times as I wish, in the example it would be 30 times equals the days of the month june. This way I would have been "not so randomly" fuskering 12000 urls covering the every 9th second between june 1st 00.00.01untill june 1st 02.00.01 2007 and june 30 00.00.01 untill june 30 02.00.01. If the code is right in this example I would end at exatcly this epoch no. = 1183161601 !!! :!: So again - the starting no. would be 1180648801 and the end 1183154401 - but only scanning 12000 selected urls in this range.

I do hope I made my self clear this time. If you use the epoch converter from the link: http://www.esqsoft.com/javascript_examp ... -epoch.htm it might help you understaqnd my needs.

Damn, this is complicated ! :oops:

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

stjernquist wrote:No - that would be wrong. In that case I would only scan the a range of files between june 1st 2007 at 00.00.01 and june 01. 2007 at 02.00.01. This would be fine for the first scan but with a span/step of 9 units so only 400 urls would be scanned. After this, it should jump 79200 units and start a new scan from this point being 1180735201 scanning every 9 urls/units untill 1180742401, then jump 79200 urls and scan every 9th url from the point 1180821601 untill 1180828801 etc. This process should be proceeded as many times as I wish, in the example it would be 30 times equals the days of the month june. This way I would have been "not so randomly" fuskering 12000 urls covering the every 9th second between june 1st 00.00.01untill june 1st 02.00.01 2007 and june 30 00.00.01 untill june 30 02.00.01. If the code is right in this example I would end at exatcly this epoch no. = 1183161601 !!! :!: So again - the starting no. would be 1180648801 and the end 1183154401 - but only scanning 12000 selected urls in this range.

I do hope I made my self clear this time. If you use the epoch converter from the link: http://www.esqsoft.com/javascript_examp ... -epoch.htm it might help you understaqnd my needs.

Damn, this is complicated ! :oops:

Kind regards

Stjernquist
Yes, complicated :D :D But we'll get it right.

ok, now I fully understand. There is only one glitch.

june 1st 00.00.01 2007 in epoch is 1180674000
and
june 1st 02.00.01 2007 in epoch is 1180681200

so that's 2 hours (1180681200 - 1180674000 = 7200)
Now, 7200/9 = 800 not 400. If you want 400 then we have to step 18

The rest I understand fully. I sent you a PM with a script, did it work correctly for the images using the link?

So I'll makde the script to do...

Start at 1180674000 and end at 1180674800 (2 hours after) but skip 9 on every turn, scanning 800 links.
Now add 22 hours (79200 ssecs) to the epoch, so now it's at 1180754000 and end at 1180761200 (2 hours after) but skip 9 on every turn, scanning 800 links.
And now for the last turn, add 22 hours (79200 ssecs) to the epoch and scan 3600 links but still step 9?
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:
stjernquist wrote:No - that would be wrong. In that case I would only scan the a range of files between june 1st 2007 at 00.00.01 and june 01. 2007 at 02.00.01. This would be fine for the first scan but with a span/step of 9 units so only 400 urls would be scanned. After this, it should jump 79200 units and start a new scan from this point being 1180735201 scanning every 9 urls/units untill 1180742401, then jump 79200 urls and scan every 9th url from the point 1180821601 untill 1180828801 etc. This process should be proceeded as many times as I wish, in the example it would be 30 times equals the days of the month june. This way I would have been "not so randomly" fuskering 12000 urls covering the every 9th second between june 1st 00.00.01untill june 1st 02.00.01 2007 and june 30 00.00.01 untill june 30 02.00.01. If the code is right in this example I would end at exatcly this epoch no. = 1183161601 !!! :!: So again - the starting no. would be 1180648801 and the end 1183154401 - but only scanning 12000 selected urls in this range.

I do hope I made my self clear this time. If you use the epoch converter from the link: http://www.esqsoft.com/javascript_examp ... -epoch.htm it might help you understaqnd my needs.

Damn, this is complicated ! :oops:

Kind regards

Stjernquist
Yes, complicated :D :D But we'll get it right.

ok, now I fully understand. There is only one glitch.

june 1st 00.00.01 2007 in epoch is 1180674000
and
june 1st 02.00.01 2007 in epoch is 1180681200

so that's 2 hours (1180681200 - 1180674000 = 7200)
Now, 7200/9 = 800 not 400. If you want 400 then we have to step 18

The rest I understand fully. I sent you a PM with a script, did it work correctly for the images using the link?

So I'll makde the script to do...

Start at 1180674000 and end at 1180674800 (2 hours after) but skip 9 on every turn, scanning 800 links.
Now add 22 hours (79200 ssecs) to the epoch, so now it's at 1180754000 and end at 1180761200 (2 hours after) but skip 9 on every turn, scanning 800 links.
And now for the last turn, add 22 hours (79200 ssecs) to the epoch and scan 3600 links but still step 9?
Sorry I made an error - yes that will be 800 scans per day and the final of scanned urls in june will then reach 24000 and not 12000!! The last scan, however, also needs to be 7200 step 9 - and this operation should be repeated 30/31 times to cover a whole month.

And yes - the script you sent works faboulously - it just needs the fine tuning.

Thanks a lot mate :P

Kind regards

Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

Great, so here is the complete script. I added a loop to scan for 31 days, so you only need to have the 2 code lines instead of adding 60 of them, 2 per day! Just change 31 in the loop to the number of days you want to scan.

Code: Select all

// -------------------------
// Edit starting at line 26
// -------------------------

function ScanEpoch(cnt, stp);
begin
  for x = 1 to cnt do begin
    lnk = @BaseURL + @Epoch;
    @ListOfLinks = @ListOfLinks + lnk + #13#10;
    @Epoch += stp;
	end;
end;

function StepEpoch(stp);
begin
  @Epoch += stp;
end;

case ScannerEvent of

  Starting:
  begin
    ListOfLinks = '';

//------ Edit this block------
    BaseURL = 'Enter the URL without the epoch number';

    Epoch = 1325543931; // the epoch number to start

    for HowManyDays = 1 to 31 do begin
      ScanEpoch(7200, 9); // scan 800 links stepping 9
      StepEpoch(79200);  // step 79200
    end;
//----------------------------

    for each line in ListOfLinks as aLink do
      ScanLink(aLink);
  end;

  BeforeAdding:
  begin
    AcceptEvent = (DocumentType ~= 'image');
  end;

else
  AcceptEvent = No;

end;
Your support team.
http://SoftByteLabs.com

stjernquist
Posts: 26
Joined: Sun Jan 01, 2012 1:48 pm

Re: How to fusker in new version??

Post by stjernquist »

Support wrote:Great, so here is the complete script. I added a loop to scan for 31 days, so you only need to have the 2 code lines instead of adding 60 of them, 2 per day! Just change 31 in the loop to the number of days you want to scan.

Code: Select all

// -------------------------
// Edit starting at line 26
// -------------------------

function ScanEpoch(cnt, stp);
begin
  for x = 1 to cnt do begin
    lnk = @BaseURL + @Epoch;
    @ListOfLinks = @ListOfLinks + lnk + #13#10;
    @Epoch += stp;
	end;
end;

function StepEpoch(stp);
begin
  @Epoch += stp;
end;

case ScannerEvent of

  Starting:
  begin
    ListOfLinks = '';

//------ Edit this block------
    BaseURL = 'Enter the URL without the epoch number';

    Epoch = 1325543931; // the epoch number to start

    for HowManyDays = 1 to 31 do begin
      ScanEpoch(7200, 9); // scan 800 links stepping 9
      StepEpoch(79200);  // step 79200
    end;
//----------------------------

    for each line in ListOfLinks as aLink do
      ScanLink(aLink);
  end;

  BeforeAdding:
  begin
    AcceptEvent = (DocumentType ~= 'image');
  end;

else
  AcceptEvent = No;

end;



Hmmm.... something seems to go wrong.

When I edit the script with the correct baseurl and then try to run it in the spider, it seems to go idle.

Now it has been "working" for very long time without any scan starts.

I did try to restart the program all over - same result :|

I wonder what can be the result of this???

Kind regards
Stjernquist

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

oh! a bug!

In the code function ScanEpoch(cnt, stp);

I have for x = 1 to cnt do begin but it should be for x = 1 to cnt step stp do begin
Your support team.
http://SoftByteLabs.com

User avatar
Support
Site Admin
Posts: 1877
Joined: Sun Oct 02, 2011 10:49 am

Re: How to fusker in new version??

Post by Support »

Here is the new code that works and generate huge amount of links much faster...

Code: Select all

// -------------------------
// Edit starting at line 26
// -------------------------

function ScanEpoch(cnt, stp);
begin
  for x = 1 to cnt step stp do begin
    lnk = @BaseURL + @Epoch;
    @ListOfLinks = @ListOfLinks + lnk + #13#10;
    @Epoch += stp;
  end;
end;

function StepEpoch(stp);
begin
  @Epoch += stp;
end;

case ScannerEvent of

  Starting:
  begin
    ListOfLinks = '';

//------ Edit this block------
    BaseURL = 'Enter the URL without the epoch number';

    Epoch = 1325543931; // the epoch number to start

    for HowManyDays = 1 to 31 do begin

      ScanEpoch(7200, 9); // scan 800 links stepping 9
      StepEpoch(79200);   // step 79200

			// do not modify
      for each line in ListOfLinks as aLink do
        ScanLink(aLink);
      ListOfLinks = '';
    end;
//----------------------------

  end;

  BeforeAdding:
  begin
    AcceptEvent = (DocumentType ~= 'image');
  end;

else
  AcceptEvent = No;

end;
Your support team.
http://SoftByteLabs.com

Post Reply