I need help with one website

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

I need help with one website

Post by braciata »

Hi! I used BW for some websites and it worked fine cause they were built in easy html.
Now I came across with this one http://miditune.com/
I would like to download the .mid files but I am not able.
Any help? Thank you
Fabrizio
User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support »

When you click on the green download button in the browser, it sends a POST to the server, like a Submit button when you register, that's why BW can't see the MIDI file. Only our BrownRecluse software can do this because it's programmable and can send POST request, BW only can do GET requests from links found on the page.
Your support team.
http://SoftByteLabs.com
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

Thanks for the reply
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

After your answer I downloaded BrownRecluse but I am newbie and I couldn't figure out a right code. I checked also the examples (I loved the one for the images) but nothing for 'post' method. Do you have some other example for my case? Thank you
User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support »

Here is a BrownRecluse script to download the mid files from the site. Set the download folder on line 5 and set the URL on line 4...

Code: Select all

PerlRegEx = Yes;
Output.Clear;

MidiURL = 'http://miditune.com/';
DownloadFolder = ScriptPath + 'midi download';

Link = New(URL);
fl = New(File);
rx = New(RegEx);
sk = New(Stack);

Abort = False;

DownloadFolder.Trailing('\');
Link.Get(MidiURL);
//Output(Link.Data);

rx.Data = Link.Data;
rx.Mask = 'href="(http\:\/\/miditune.com/midi-[^"]+\.html)">';
rx.Reset;

while rx.Match do
  sk.Push(rx.Value[1]);

unless sk.Empty do begin

  lnk = sk.Pop;
  //Output(lnk);
  Link.Get(lnk);
  //Output(Link.Data);

  frmID = WildGet(Link.Data, 'name="id" value=''([^'']+)''');
  frmSS = WildGet(Link.Data, 'name="session_id" value=''([^'']+)''');

  Link.Referer = Link.Location;
  Link.Location = 'http://miditune.com/download.php';
  Link.Post('id='+frmID+'&session_id='+frmSS);

  Flname = DownloadFolder + WildGet(Link.Headers, 'filename="([^"]+)"');
  //Output(Flname); Terminate;

  fl.Open(Flname);
  fl.Truncate;
  fl.Write(Link.Data);
  fl.Close;

  Output('Downloaded: '+Flname);

  if Abort then
    Break;

end;

function OnStop();
begin
  Result = @Abort;
  @Abort = True;
end;
Your support team.
http://SoftByteLabs.com
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

Omg, thank you very much. I will try to study it
User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support »

Here is a new script that will download all of the midi files from the entire site...

Code: Select all

PerlRegEx = Yes;
Output.Clear;

DownloadFolder = ScriptPath + 'midi download';

Link = New(URL);
fl = New(File);
rx = New(RegEx);

Abort = False;

DownloadFolder.Trailing('\');

for c = 'A' to 'Z' range('A','Z') do begin

  Link.Get('http://miditune.com/list/midi-'+c+'.html');
  //Output(Link.Data);Terminate;

    rx.Data = Link.Data;
    rx.Mask = 'href=(http[^<>]+)';
    rx.Reset;

    while rx.Match do begin

      lnk = rx.Value[1];
      lnk.Replace('\.mid$', '.html');
      //Output(lnk); Terminate;
      Link.Get(lnk);
      //Output(Link.Data);

      frmID = WildGet(Link.Data, 'name="id" value=''([^'']+)''');
      frmSS = WildGet(Link.Data, 'name="session_id" value=''([^'']+)''');

      Link.Referer = Link.Location;
      Link.Location = 'http://miditune.com/download.php';
      Link.Post('id='+frmID+'&session_id='+frmSS);

      Flname = DownloadFolder + WildGet(Link.Headers, 'filename="([^"]+)"');
      //Output(Flname); Terminate;

      fl.Open(Flname);
      fl.Truncate;
      fl.Write(Link.Data);
      fl.Close;

      Output('Downloaded: '+Flname);

      if Abort then
        Break;

    end;

end;

function OnStop();
begin
  Result = @Abort;
  @Abort = True;
end;
Your support team.
http://SoftByteLabs.com
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

Perfect! Thank you so much
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

Here I am again!
When it comes ATB - 9PM Till I Come.mid BR stops.
I am not sure but I think because the url is different > http://miditune.com/midi-17680-download ... -come.html
9PM vs 9pm
User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support »

they both work for me, how long did you leave it when it appeared stopped?
Your support team.
http://SoftByteLabs.com
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

After it downloaded 42 midi files from page A it found that file and it stops.
User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support »

I guess that explains it...
2018-08-13_154812.png
2018-08-13_154812.png (43.16 KiB) Viewed 16733 times
Your support team.
http://SoftByteLabs.com
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata »

:o thats why
Post Reply