google results pages

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
alpha2
Posts: 31
Joined: Tue Oct 30, 2012 8:24 am

google results pages

Post by alpha2 » Mon Nov 05, 2012 6:26 pm

Hi,

thanks for the hints so far. I would like to save the google results pages, so I can compare them time after time. Are there any standard settings I should take in order to save the google results pages locally for a certain query? I had problems, and it didn't work at all for me. But I thought, that google should be a standard, which is already known to you.

Regards,

Alpha

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: google results pages

Post by Support » Mon Nov 05, 2012 6:55 pm

Well, this will not work. When you scan google, you get different session ID in the links and it will also ask you for a captcha verification. Google don't like to be spidered!
Your support team.
http://SoftByteLabs.com

alpha2
Posts: 31
Joined: Tue Oct 30, 2012 8:24 am

Re: google results pages

Post by alpha2 » Tue Nov 06, 2012 1:38 am

Does this problem also exist, if I crawl only with one stream and slowly with long delays?
Speed is not the problem.

Is there any other search engine like bing or so, which can be spidered?

User avatar
Support
Site Admin
Posts: 1830
Joined: Sun Oct 02, 2011 10:49 am

Re: google results pages

Post by Support » Tue Nov 06, 2012 6:12 pm

Perhaps not. I'm not sure. All search engines have some ways to restrict spidering the data!
Your support team.
http://SoftByteLabs.com

Post Reply