A possible problem

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

A possible problem

Post by pilotmike55 » Tue Jul 19, 2016 4:37 pm

I am running BlackWidow v6.3 I have noticed that when I am scanning a site that randomly the program will freeze during the scan and the only thing I can do is to end the process in task manager. Because of this I routinely pause the scan, save the structure, then continue the scan. If the program freezes I end the process then load in the last saved structure for that scan.

I have also noticed that the number of remaining links to scan will be at particular number, let's say 10,000, when I say save it. If I reload that file and start the scan again, a few links will scan then the scan ends as if all of the links have been scanned and I cannot continue.

I am currently running Windows 10 and have ran BW in Windows 7 compatibility mode with the same results

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Tue Jul 19, 2016 4:41 pm

I'm not sure the nature of the freeze. BW uses your IE extension so it has to come from that. When you save the structure and reload it, make sure to resubmit the links in the Error tab because they were canceled.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Tue Jul 19, 2016 6:43 pm

When I reload the file there is nothing under the link errors tab.

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Tue Jul 19, 2016 6:47 pm

ok, how many connections do you have it set to?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Tue Jul 19, 2016 7:06 pm

I have it currently set to 6 connections. I have tried 12 but had the same results

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Tue Jul 19, 2016 7:17 pm

Try 4 because that's the standard setting for a web server. The freezing could be that it detect you are scanning the site and freeze the packet putting your scan in a loop. What you can do is use 12 connections but use the pause 1-1 to pause each one for 1 sec before continuing.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Tue Jul 19, 2016 7:44 pm

I have used the pause 1-3 previously and had the same issue.

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Tue Jul 19, 2016 7:53 pm

ok, is you IE up to date? this happened when people don't want to use IE and use Firefox for example, not updating their IE.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Wed Jul 20, 2016 8:11 am

I am currently running IE11

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Wed Jul 20, 2016 9:30 am

on win10/64?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Wed Jul 20, 2016 10:52 am

I have IE11 on Windows 10. But there is also Microsoft Edge

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Wed Jul 20, 2016 11:44 am

I don't know what to say, it could be many things such as your service provider etc. Do you scan while downloading? or only scan and download when the scan is done? do you need a login to the site? if so, maybe it ask to login from time to time and that login window is behind the BW window freezing BW until you close the login window?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sat Jul 23, 2016 8:46 am

I have downloaded a couple of other programs and they do not have the freezing issue that I experience with BlackWidow. Unfortunately they do not have the programmability that BlackWidow has.

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Sat Jul 23, 2016 12:52 pm

That's because none of them use IE like BW does. The advantage on using IE is because of the customization you can have, whatever your settings are in IE, that's what BW will use. The others have limited functionalities. In your case that may be irrelevant, it depend on the site you try to scan. I haven't had this problem of freezing and no one else have complained about BW freezing so it's very hard for me to know why it does. Could be a hardware issue as well, like a bad memory, or simply it's the fault of IE directly.

How long does it scan before the freeze? did you notice a pattern in time, number of files etc?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sat Jul 23, 2016 1:43 pm

There is no pattern to the freezing, it could be days, hours, or minutes. I thought at first it was do to links such as statcounter.com but I modified the filter to skip that page.

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sat Jul 23, 2016 1:54 pm

Although it seems to occur more when scan has been running for a while and the file size is large rather when first starting the scan and the file size is small

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Sat Jul 23, 2016 1:55 pm

Are you scanning the whole site or are you trying to scan particular files such as images. Is a login required? if so you might be scanning the logout link too.
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sat Jul 23, 2016 3:07 pm

A login is not required. I am scanning various blogs in tumblr.com. The URLs are typically http://blogname.tumblr.com/archive

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Sat Jul 23, 2016 3:24 pm

ok, so you should set BW to scan nothing and add filters to only follow the URLs of interest, in your example URL, they are all http://blogname.tumblr.com/post/586793124 where the number is any, so the "What to add" RegEx mask for this is

Code: Select all

blogname\.tumblr\.com/post/\d+$
and this will make it only scan the pages of the links listed on the start page, no other links like login, logout, join etc. I assume you want to download those pages right?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sun Jul 24, 2016 9:16 am

That is correct

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Sun Jul 24, 2016 11:14 am

ok, have you tried the filter I suggested?
Your support team.
http://SoftByteLabs.com

pilotmike55
Posts: 17
Joined: Wed Jan 27, 2016 4:11 pm

Re: A possible problem

Post by pilotmike55 » Sun Jul 24, 2016 5:39 pm

Yes I have and it still freezes. I have built a Windows 7 virtual machine and it still freezes.

User avatar
Support
Site Admin
Posts: 1679
Joined: Sun Oct 02, 2011 10:49 am

Re: A possible problem

Post by Support » Sun Jul 24, 2016 6:10 pm

Virtual still uses the same PC so it won't help! I'm not sure what to suggest at this point other than let me know your settings, copy them from the filters and paste them here and I'll try it myself and see where the problem is, and fix it of course.
Your support team.
http://SoftByteLabs.com

Post Reply