A possible problem

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

A possible problem

Post by pilotmike55 »

I am running BlackWidow v6.3 I have noticed that when I am scanning a site that randomly the program will freeze during the scan and the only thing I can do is to end the process in task manager. Because of this I routinely pause the scan, save the structure, then continue the scan. If the program freezes I end the process then load in the last saved structure for that scan.

I have also noticed that the number of remaining links to scan will be at particular number, let's say 10,000, when I say save it. If I reload that file and start the scan again, a few links will scan then the scan ends as if all of the links have been scanned and I cannot continue.

I am currently running Windows 10 and have ran BW in Windows 7 compatibility mode with the same results

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

I'm not sure the nature of the freeze. BW uses your IE extension so it has to come from that. When you save the structure and reload it, make sure to resubmit the links in the Error tab because they were canceled.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

When I reload the file there is nothing under the link errors tab.

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

ok, how many connections do you have it set to?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

I have it currently set to 6 connections. I have tried 12 but had the same results

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

Try 4 because that's the standard setting for a web server. The freezing could be that it detect you are scanning the site and freeze the packet putting your scan in a loop. What you can do is use 12 connections but use the pause 1-1 to pause each one for 1 sec before continuing.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

I have used the pause 1-3 previously and had the same issue.

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

ok, is you IE up to date? this happened when people don't want to use IE and use Firefox for example, not updating their IE.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

I am currently running IE11

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

on win10/64?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

I have IE11 on Windows 10. But there is also Microsoft Edge

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

I don't know what to say, it could be many things such as your service provider etc. Do you scan while downloading? or only scan and download when the scan is done? do you need a login to the site? if so, maybe it ask to login from time to time and that login window is behind the BW window freezing BW until you close the login window?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

I have downloaded a couple of other programs and they do not have the freezing issue that I experience with BlackWidow. Unfortunately they do not have the programmability that BlackWidow has.

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

That's because none of them use IE like BW does. The advantage on using IE is because of the customization you can have, whatever your settings are in IE, that's what BW will use. The others have limited functionalities. In your case that may be irrelevant, it depend on the site you try to scan. I haven't had this problem of freezing and no one else have complained about BW freezing so it's very hard for me to know why it does. Could be a hardware issue as well, like a bad memory, or simply it's the fault of IE directly.

How long does it scan before the freeze? did you notice a pattern in time, number of files etc?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

There is no pattern to the freezing, it could be days, hours, or minutes. I thought at first it was do to links such as statcounter.com but I modified the filter to skip that page.

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

Although it seems to occur more when scan has been running for a while and the file size is large rather when first starting the scan and the file size is small

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

Are you scanning the whole site or are you trying to scan particular files such as images. Is a login required? if so you might be scanning the logout link too.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

A login is not required. I am scanning various blogs in tumblr.com. The URLs are typically http://blogname.tumblr.com/archive

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

ok, so you should set BW to scan nothing and add filters to only follow the URLs of interest, in your example URL, they are all http://blogname.tumblr.com/post/586793124 where the number is any, so the "What to add" RegEx mask for this is

Code: Select all

blogname\.tumblr\.com/post/\d+$
and this will make it only scan the pages of the links listed on the start page, no other links like login, logout, join etc. I assume you want to download those pages right?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

That is correct

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

ok, have you tried the filter I suggested?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 »

Yes I have and it still freezes. I have built a Windows 7 virtual machine and it still freezes.

User avatar
Support
Site Admin
Posts: 1896
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support »

Virtual still uses the same PC so it won't help! I'm not sure what to suggest at this point other than let me know your settings, copy them from the filters and paste them here and I'll try it myself and see where the problem is, and fix it of course.
Your support team.
http://SoftByteLabs.com

Post Reply