Having trouble downloading part of a website

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
Peacemaker
Posts: 4
Joined: Thu Feb 20, 2014 7:52 pm

Having trouble downloading part of a website

Post by Peacemaker »

I just downloaded BW yesterday. I'm trying to download a part of the website BibleHub.com. I started out by scanning only the parts of the website that I wanted, because the whole site is very large and scanning the whole site would take way too much time. So, once I got everything scanned that I wanted, I clicked on the Structure part of the BW program. Then, I highlighted all the folders that I wanted and clicked Download. By the way, the scan said that it scanned 32,000 files and that the size of the finished product would be close to a gigabyte. But, after the BW download finished, I checked the folder it downloaded to and only found 4700 files totaling 120 megabytes. I tried to redownload everything, checking the box to ignore duplicates, but the folder still has the same amount of files inside. I made a BW6 file of the 32,000 files it scanned, which I have attached to this message. Maybe you can figure out what's going on. Thanks.

P.S. The program works fast. I did all of this in one day. I've tried other programs(HTTrack, WinWSD Website Downloader) and they take a lot, lot longer.
Attachments
BibleHub Structure BW6.bw6
(60.18 MiB) Downloaded 1193 times
User avatar
Support
Site Admin
Posts: 1899
Joined: Sun Oct 02, 2011 10:49 am

Re: Having trouble downloading part of a website

Post by Support »

Before I start the download to test, I just want to make sure of one thing. When you select the folder in the left pan of the structure, all the files the selected folders contain will be listed in the right pan. Have you selected all the files before download? how many selected files did it show in the download window before it actually start the download? and when the download start, how many files are in the queue (shown in the bottom status bar)?
Your support team.
http://SoftByteLabs.com
Peacemaker
Posts: 4
Joined: Thu Feb 20, 2014 7:52 pm

Re: Having trouble downloading part of a website

Post by Peacemaker »

Alright, so, to answer your questions, I went ahead and captured four screenshots, so you can see what I did the first and second time I tried downloading the files. The 1st screenshot is where I selected all the folders on the left side. The 2nd screenshot is of the popup window that comes up after you click on download on the left side. And the 3rd screenshot is of the BW program downloading files. This is the third time I have downloaded files, so the scanned files number is pretty high now. And the 4th screenshot shows how many files were downloaded into the destination folder. I hope that answers your questions.

One more thing... I tried checking the box for "Download while scanning" on this third try, but it hasn't made any difference. After getting more than two-thirds of the way through, I still have the same number of files in the destination folder.

Alright, the "remaining files" counter just counted down to zero and then started counting up in negative numbers. It's still doing that right now. There's no change to the number of files in the destination folder. I think I'm going to just stop it. I don't think it's supposed to be counting in negative numbers, although it does show a negative sign, so that must be built into the program. Is that something you want the BW program to be doing? It's currently at -3160. Still no change in the number of files downloaded. Alright, I'm going to stop it.
Attachments
4th screenshot
4th screenshot
ScreenHunter_05 Feb. 21 02.37.jpg (181.78 KiB) Viewed 13789 times
3rd screenshot
3rd screenshot
ScreenHunter_04 Feb. 21 02.00.jpg (195.2 KiB) Viewed 13791 times
2nd screenshot
2nd screenshot
ScreenHunter_03 Feb. 21 01.50.jpg (225.93 KiB) Viewed 13791 times
1st screenshot
1st screenshot
ScreenHunter_02 Feb. 21 01.46.jpg (278.4 KiB) Viewed 13791 times
User avatar
Support
Site Admin
Posts: 1899
Joined: Sun Oct 02, 2011 10:49 am

Re: Having trouble downloading part of a website

Post by Support »

You did not select "Preserve directory structure" and "ignore duplicates" is selected, so it does not save same file name files! That's why.
Your support team.
http://SoftByteLabs.com
Peacemaker
Posts: 4
Joined: Thu Feb 20, 2014 7:52 pm

Re: Having trouble downloading part of a website

Post by Peacemaker »

UPDATE:
After failing to get anything to happen earlier, I tried rescanning everything, but this time I checkmarked the "Download while scanning" before I started scanning. I also checkmarked the "Preserve directory structure" and "Ignore duplicates". This time, I'm pretty sure all of the files were downloaded. I have 32,445 of them. The problem I'm having now is that the files aren't linked together and the webpages I downloaded aren't set up in the same way as the online webpages. I'm just trying to go from one verse in the Bible to another by clicking on a link, and that doesn't work. Can you help me figure out what's going on? Thanks. Victor
Peacemaker
Posts: 4
Joined: Thu Feb 20, 2014 7:52 pm

Re: Having trouble downloading part of a website

Post by Peacemaker »

I just got your most recent message after I sent my last one. So, now that I've downloaded all 32,445 of the files, is it possible that I've messed up the entire thing by having the "Ignore duplicate files" box checkmarked? Is that why my files aren't linked together and the page doesn't set up properly?
User avatar
Support
Site Admin
Posts: 1899
Joined: Sun Oct 02, 2011 10:49 am

Re: Having trouble downloading part of a website

Post by Support »

Yes it is possible because BW will nor re-download files that already exist and have the same file size. What I suggest you do is delete the download folder and re-download them again to ensure they are the same as the ones on the site.
Your support team.
http://SoftByteLabs.com
Post Reply