I need help with one website

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

I need help with one website

Post by braciata » Fri Aug 10, 2018 5:59 pm

Hi! I used BW for some websites and it worked fine cause they were built in easy html.
Now I came across with this one http://miditune.com/
I would like to download the .mid files but I am not able.
Any help? Thank you
Fabrizio

User avatar
Support
Site Admin
Posts: 1731
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support » Fri Aug 10, 2018 6:10 pm

When you click on the green download button in the browser, it sends a POST to the server, like a Submit button when you register, that's why BW can't see the MIDI file. Only our BrownRecluse software can do this because it's programmable and can send POST request, BW only can do GET requests from links found on the page.
Your support team.
http://SoftByteLabs.com

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Fri Aug 10, 2018 6:13 pm

Thanks for the reply

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Sat Aug 11, 2018 3:19 am

After your answer I downloaded BrownRecluse but I am newbie and I couldn't figure out a right code. I checked also the examples (I loved the one for the images) but nothing for 'post' method. Do you have some other example for my case? Thank you

User avatar
Support
Site Admin
Posts: 1731
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support » Sat Aug 11, 2018 8:08 pm

Here is a BrownRecluse script to download the mid files from the site. Set the download folder on line 5 and set the URL on line 4...

Code: Select all

PerlRegEx = Yes;
Output.Clear;

MidiURL = 'http://miditune.com/';
DownloadFolder = ScriptPath + 'midi download';

Link = New(URL);
fl = New(File);
rx = New(RegEx);
sk = New(Stack);

Abort = False;

DownloadFolder.Trailing('\');
Link.Get(MidiURL);
//Output(Link.Data);

rx.Data = Link.Data;
rx.Mask = 'href="(http\:\/\/miditune.com/midi-[^"]+\.html)">';
rx.Reset;

while rx.Match do
  sk.Push(rx.Value[1]);

unless sk.Empty do begin

  lnk = sk.Pop;
  //Output(lnk);
  Link.Get(lnk);
  //Output(Link.Data);

  frmID = WildGet(Link.Data, 'name="id" value=''([^'']+)''');
  frmSS = WildGet(Link.Data, 'name="session_id" value=''([^'']+)''');

  Link.Referer = Link.Location;
  Link.Location = 'http://miditune.com/download.php';
  Link.Post('id='+frmID+'&session_id='+frmSS);

  Flname = DownloadFolder + WildGet(Link.Headers, 'filename="([^"]+)"');
  //Output(Flname); Terminate;

  fl.Open(Flname);
  fl.Truncate;
  fl.Write(Link.Data);
  fl.Close;

  Output('Downloaded: '+Flname);

  if Abort then
    Break;

end;

function OnStop();
begin
  Result = @Abort;
  @Abort = True;
end;
Your support team.
http://SoftByteLabs.com

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Sun Aug 12, 2018 5:08 am

Omg, thank you very much. I will try to study it

User avatar
Support
Site Admin
Posts: 1731
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support » Mon Aug 13, 2018 1:11 pm

Here is a new script that will download all of the midi files from the entire site...

Code: Select all

PerlRegEx = Yes;
Output.Clear;

DownloadFolder = ScriptPath + 'midi download';

Link = New(URL);
fl = New(File);
rx = New(RegEx);

Abort = False;

DownloadFolder.Trailing('\');

for c = 'A' to 'Z' range('A','Z') do begin

  Link.Get('http://miditune.com/list/midi-'+c+'.html');
  //Output(Link.Data);Terminate;

    rx.Data = Link.Data;
    rx.Mask = 'href=(http[^<>]+)';
    rx.Reset;

    while rx.Match do begin

      lnk = rx.Value[1];
      lnk.Replace('\.mid$', '.html');
      //Output(lnk); Terminate;
      Link.Get(lnk);
      //Output(Link.Data);

      frmID = WildGet(Link.Data, 'name="id" value=''([^'']+)''');
      frmSS = WildGet(Link.Data, 'name="session_id" value=''([^'']+)''');

      Link.Referer = Link.Location;
      Link.Location = 'http://miditune.com/download.php';
      Link.Post('id='+frmID+'&session_id='+frmSS);

      Flname = DownloadFolder + WildGet(Link.Headers, 'filename="([^"]+)"');
      //Output(Flname); Terminate;

      fl.Open(Flname);
      fl.Truncate;
      fl.Write(Link.Data);
      fl.Close;

      Output('Downloaded: '+Flname);

      if Abort then
        Break;

    end;

end;

function OnStop();
begin
  Result = @Abort;
  @Abort = True;
end;
Your support team.
http://SoftByteLabs.com

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Mon Aug 13, 2018 1:26 pm

Perfect! Thank you so much

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Mon Aug 13, 2018 3:23 pm

Here I am again!
When it comes ATB - 9PM Till I Come.mid BR stops.
I am not sure but I think because the url is different > http://miditune.com/midi-17680-download ... -come.html
9PM vs 9pm

User avatar
Support
Site Admin
Posts: 1731
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support » Mon Aug 13, 2018 3:31 pm

they both work for me, how long did you leave it when it appeared stopped?
Your support team.
http://SoftByteLabs.com

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Mon Aug 13, 2018 3:35 pm

After it downloaded 42 midi files from page A it found that file and it stops.

User avatar
Support
Site Admin
Posts: 1731
Joined: Sun Oct 02, 2011 10:49 am

Re: I need help with one website

Post by Support » Mon Aug 13, 2018 3:49 pm

I guess that explains it...
2018-08-13_154812.png
2018-08-13_154812.png (43.16 KiB) Viewed 3174 times
Your support team.
http://SoftByteLabs.com

braciata
Posts: 8
Joined: Fri Aug 10, 2018 5:54 pm

Re: I need help with one website

Post by braciata » Mon Aug 13, 2018 3:51 pm

:o thats why

Post Reply