SoftByteLabs.com

Hi there, I need a solution which can scan htm html & other various html extensions. I need it to be able to scan in excess of over 100K files in each run without crashing and a ability to remove what looks like timestamped server generated code such as. <!-- text below generated by server. PLEASE R...

It can yes, especially when the site has many links pointing to a single link, it creates a lot of referring links. What I can do is create a small program that can remove all but one referrer from the .web file, it will be very small and load fast, or, you can edit the .web file (it's a text file)...

Sure, I can give that a try.

Hello, after 2 long hours, blackwidow finally manages to load this thing, is there any way to cut it down to size or make it load faster?

My system is more than capable of multitasking it while doing other stuff.

I've compressed it so you can try it for yourself.

I just got partway done with a site that has a little over 2k pages to download and I notice that the save file for this project is over 2GB?

2.22GB to be exact.

Why? How? Will it even be able to load that save file if I try to open it tomorrow?

I would like the following rules for blackwidow, thanks. I've tried but it stops after 1 download, claiming it's "done".

Thanks, I was overthinking this one a bit lol

Remove numbered listing from files. What settings should I use to remove the numbers in front of the titles? I have over 2400 titles to rename this way, thanks! 0001 - F-Zero for Game Boy Advance (J).zip 0002 - Super Mario Advance (J).zip 0003 - Battle Network Rockman EXE (J).zip 0004 - Kurukuru Kur...

Hi thanks, is it possible to setup macros? Like for instance I type in what I want to be removed in a macro and with a press of a button it's entered into the replace/remove fields.

I need to try to replace the space characters with -'s. Example:

Rayman 2 - The Great Escape (USA) (En,Fr,De,Es,It)
TO
Rayman-2-The Great Escape-(USA)-(En,Fr,De,Es,It)

Also in this case I noticed it would have put in two -'s instead of one, any tips?

Have an idea for a new feature for blackwidow, please keep them all in this thread thanks! I think a good option to be added (from httrack) would be a option to follow robots.txt rules or not. *Detection of if the scan is crashing the site's server or not & proper slowdowns applied if this is the ca...

I was wrong, it is downloading them properly as you said.

ok thanks. Is it possible to make it crawl the subdomain of img.gamefaqs.net? Tricky one there... at first I thought it ended in gamefaqs.com

nvm it seems to be grabbing from the img.gamefaqs.net

Hi thanks, it doesn't seem to be adding any yet... I can report that I had to slowdown the scan as it tried to become unresponsive. And yes my system is more than capable of handling it... but why nothing added? http://img.photobucket.com/albums/v48/scylla/specs_zpsa0149f8d.png http://i.imgur.com/Vs...

Also it's interesting to note that some images on gamefaqs are hosted on gamespot.. like so... http://img.gamefaqs.net/screens/1/1/2/gfs_67605_2_3_thm.jpg http://image.gamespotcdn.net/gamespot/images/box/5/1/7/928517_75786_thumb.jpg If it helps here is a gamefaqs scraper I think for xbmc written in ...

I wish for it to save only images @ gamefaqs(png, gif, jpg, jpeg, bmp & webp formats)... any other files, it can throw away without saving them. Can someone help me with settings to make the blackwidow do this? Thanks! This is from their sitemap page for their images. <h2>Images</h2> <ul> <li><a hre...

Thank you!

Yes those things. I'm not sure their exact name.

When it encounters pages with the little hash tags, it tends to grab a copy for every hashtag element thing it finds... help?

Example here.... there's 15 of the hashtag things so it'll grab 15 copies of that page.
http://www.world-of-nintendo.com/ninten ... ault.shtml

Yes the mime type was set correctly.... I manage several sites and do some maintenance on them now & then so having a browser set to open a htm/html/shtml/php file etc is pretty important to me but as your request I checked via regedit and here is result.

And what about the threads? Those are the main one's that give trouble. Also maybe you have different rule sets than me because mine isn't saving as .php.htm instead just saving it with apparently no extension like this 2468-Ultimecia-Vs-Sephiroth-Vs-Kuja-Vs-Kefka Unless of course it does encounter ...

even with what you suggested it still remains the same. Even further verified by opening the file in editplus... where it displays the correct file syntax colors to show opened/closed tags & other good stuff.

Unfortunately it doesn't seem to want to put the .html for me.

I'm wanting to know, how I make your software crawl download these pages when it encounters them?
Instead of saving it and calling the end result .file can we make it save it as .html?

http://www.finalfantasyforums.net/forum.php

There is one of the sites I'd like to crawl.

Well? Is there a way to do this?