How to download images only from gamefaqs.com

BlackWidow scans websites (it's a site ripper). It can download an entire website, or download portions of a site.
Post Reply
avalanch
Posts: 43
Joined: Fri Mar 16, 2012 11:31 am

How to download images only from gamefaqs.com

Post by avalanch »

I wish for it to save only images @ gamefaqs(png, gif, jpg, jpeg, bmp & webp formats)... any other files, it can throw away without saving them. Can someone help me with settings to make the blackwidow do this? Thanks!

This is from their sitemap page for their images.

<h2>Images</h2>
<ul>
<li><a href="/images/3do/">3DO</a></li>
<li><a href="/images/3ds/">3DS</a></li>
<li><a href="/images/arch/">Acorn Archimedes</a></li>
<li><a href="/images/avision/">Adventurevision</a></li>
<li><a href="/images/amiga/">Amiga</a></li>
<li><a href="/images/cd32/">Amiga CD32</a></li>
<li><a href="/images/cpc/">Amstrad CPC</a></li>
<li><a href="/images/android/">Android</a></li>
<li><a href="/images/apf1000/">APF-*1000/IM</a></li>
<li><a href="/images/appleii/">Apple II</a></li>
<li><a href="/images/arcade/">Arcade Games</a></li>
<li><a href="/images/a2k1/">Arcadia 2001</a></li>
<li><a href="/images/astrocade/">Astrocade</a></li>
<li><a href="/images/atari2600/">Atari 2600</a></li>
<li><a href="/images/atari5200/">Atari 5200</a></li>
<li><a href="/images/atari7800/">Atari 7800</a></li>
<li><a href="/images/atari8bit/">Atari 8-bit</a></li>
<li><a href="/images/ast/">Atari ST</a></li>
<li><a href="/images/pippin/">Bandai Pippin</a></li>
<li><a href="/images/bbc/">BBC Micro</a></li>
<li><a href="/images/bbs/">BBS Door</a></li>
<li><a href="/images/blackberry/">BlackBerry</a></li>
<li><a href="/images/loopy/">Casio Loopy</a></li>
<li><a href="/images/ecv/">Cassette Vision</a></li>
<li><a href="/images/cdi/">CD-I</a></li>
<li><a href="/images/channelf/">Channel F</a></li>
<li><a href="/images/colecovision/">Colecovision</a></li>
<li><a href="/images/c64/">Commodore 64</a></li>
<li><a href="/images/pet/">Commodore PET</a></li>
<li><a href="/images/cps/">CPS Changer</a></li>
<li><a href="/images/cvision/">CreatiVision</a></li>
<li><a href="/images/dreamcast/">Dreamcast</a></li>
<li><a href="/images/ds/">DS</a></li>
<li><a href="/images/dvd/">DVD Player</a></li>
<li><a href="/images/ereader/">e-Reader</a></li>
<li><a href="/images/cg2000/">EACA Colour Genie 2000</a></li>
<li><a href="/images/famicomds/">Famicom Disk System</a></li>
<li><a href="/images/flash/">Flash</a></li>
<li><a href="/images/fmtowns/">FM Towns</a></li>
<li><a href="/images/fm7/">FM-7</a></li>
<li><a href="/images/gameboy/">Game Boy</a></li>
<li><a href="/images/gba/">Game Boy Advance</a></li>
<li><a href="/images/gbc/">Game Boy Color</a></li>
<li><a href="/images/gamecom/">Game.com</a></li>
<li><a href="/images/gamecube/">GameCube</a></li>
<li><a href="/images/gamegear/">GameGear</a></li>
<li><a href="/images/genesis/">Genesis</a></li>
<li><a href="/images/gizmondo/">Gizmondo</a></li>
<li><a href="/images/gp32/">GP32</a></li>
<li><a href="/images/intellivision/">Intellivision</a></li>
<li><a href="/images/vc4000/">Interton VC4000</a></li>
<li><a href="/images/iphone/">iPhone/iPod</a></li>
<li><a href="/images/jaguar/">Jaguar</a></li>
<li><a href="/images/jaguarcd/">Jaguar CD</a></li>
<li><a href="/images/laser/">LaserActive</a></li>
<li><a href="/images/lynx/">Lynx</a></li>
<li><a href="/images/mac/">Macintosh</a></li>
<li><a href="/images/aquarius/">Mattel Aquarius</a></li>
<li><a href="/images/microvision/">Microvision</a></li>
<li><a href="/images/mobile/">Mobile</a></li>
<li><a href="/images/msx/">MSX</a></li>
<li><a href="/images/ngage/">N-Gage</a></li>
<li><a href="/images/pc88/">NEC PC88</a></li>
<li><a href="/images/pc98/">NEC PC98</a></li>
<li><a href="/images/neogeocd/">Neo-Geo CD</a></li>
<li><a href="/images/neo/">NeoGeo</a></li>
<li><a href="/images/ngpc/">NeoGeo Pocket Color</a></li>
<li><a href="/images/nes/">NES</a></li>
<li><a href="/images/n64/">Nintendo 64</a></li>
<li><a href="/images/n64dd/">Nintendo 64DD</a></li>
<li><a href="/images/nuon/">Nuon</a></li>
<li><a href="/images/odyssey/">Odyssey</a></li>
<li><a href="/images/odyssey2/">Odyssey^2</a></li>
<li><a href="/images/webonly/">Online/Browser</a></li>
<li><a href="/images/oric1/">Oric 1/Atmos</a></li>
<li><a href="/images/os2/">OS/2</a></li>
<li><a href="/images/ouya/">Ouya</a></li>
<li><a href="/images/palmos/">Palm OS Classic</a></li>
<li><a href="/images/palm-webos/">Palm webOS</a></li>
<li><a href="/images/pc/">PC</a></li>
<li><a href="/images/pcfx/">PC-FX</a></li>
<li><a href="/images/pinball/">Pinball</a></li>
<li><a href="/images/playdia/">Playdia</a></li>
<li><a href="/images/ps/">PlayStation</a></li>
<li><a href="/images/ps2/">PlayStation 2</a></li>
<li><a href="/images/ps3/">PlayStation 3</a></li>
<li><a href="/images/ps4/">PlayStation 4</a></li>
<li><a href="/images/vita/">PlayStation Vita</a></li>
<li><a href="/images/psp/">PSP</a></li>
<li><a href="/images/studio2/">RCA Studio II</a></li>
<li><a href="/images/redemption/">Redemption</a></li>
<li><a href="/images/saturn/">Saturn</a></li>
<li><a href="/images/sega32x/">Sega 32X</a></li>
<li><a href="/images/segacd/">Sega CD</a></li>
<li><a href="/images/sms/">Sega Master System</a></li>
<li><a href="/images/sg1000/">SG-1000</a></li>
<li><a href="/images/x1/">Sharp X1</a></li>
<li><a href="/images/x68000/">Sharp X68000</a></li>
<li><a href="/images/sinclair/">Sinclair ZX81/Spectrum</a></li>
<li><a href="/images/sordm5/">Sord M5</a></li>
<li><a href="/images/scv/">Super Cassette Vision</a></li>
<li><a href="/images/snes/">Super Nintendo</a></li>
<li><a href="/images/svision/">SuperVision</a></li>
<li><a href="/images/coco/">Tandy Color Computer</a></li>
<li><a href="/images/ti99/">TI-99/4A</a></li>
<li><a href="/images/turbocd/">Turbo CD</a></li>
<li><a href="/images/tg16/">TurboGrafx-16</a></li>
<li><a href="/images/unixlinux/">Unix/Linux</a></li>
<li><a href="/images/vectrex/">Vectrex</a></li>
<li><a href="/images/vic20/">VIC-20</a></li>
<li><a href="/images/virtualboy/">Virtual Boy</a></li>
<li><a href="/images/wii/">Wii</a></li>
<li><a href="/images/wii-u/">Wii U</a></li>
<li><a href="/images/windows-mobile/">Windows Mobile</a></li>
<li><a href="/images/wonderswan/">WonderSwan</a></li>
<li><a href="/images/wsc/">WonderSwan Color</a></li>
<li><a href="/images/xbox/">Xbox</a></li>
<li><a href="/images/xbox360/">Xbox 360</a></li>
<li><a href="/images/xboxone/">Xbox One</a></li>
<li><a href="/images/zeebo/">Zeebo</a></li>
<li><a href="/images/zod/">Zodiac</a></li>

avalanch
Posts: 43
Joined: Fri Mar 16, 2012 11:31 am

Re: How to download images only from gamefaqs.com

Post by avalanch »

Also it's interesting to note that some images on gamefaqs are hosted on gamespot.. like so...

http://img.gamefaqs.net/screens/1/1/2/g ... _3_thm.jpg
http://image.gamespotcdn.net/gamespot/i ... _thumb.jpg

If it helps here is a gamefaqs scraper I think for xbmc written in python.

https://code.google.com/p/xbmc-repo-ang ... per.py?r=2

User avatar
Support
Site Admin
Posts: 1892
Joined: Sun Oct 02, 2011 10:49 am

Re: How to download images only from gamefaqs.com

Post by Support »

Here are the filters. You can change the URL to that of a specific one, and you can also add more file extensions to the filters...

Code: Select all

[BlackWidow v6.00 filters]
URL = http://sitemap.gamefaqs.com/
[ ] Expert mode
[ ] Scan everything
[x] Scan whole site
Local depth: 0
[x] Scan external links
[ ] Only verify external links
External depth: 0
Default index page: 
Startup referrer: 
[ ] Slow down by 1:1 seconds
4 threads
[x] Follow /images/ using plain text
[x] Follow /images$ using regular expression
[x] Do not add thumb\. from URL using regular expression
[x] Add \.(png|gif|jpg|jpeg|bmp|webp)$ from URL using regular expression
[end]
Your support team.
http://SoftByteLabs.com

avalanch
Posts: 43
Joined: Fri Mar 16, 2012 11:31 am

Re: How to download images only from gamefaqs.com

Post by avalanch »

Hi thanks, it doesn't seem to be adding any yet... I can report that I had to slowdown the scan as it tried to become unresponsive. And yes my system is more than capable of handling it... but why nothing added?

Image

Image

User avatar
Support
Site Admin
Posts: 1892
Joined: Sun Oct 02, 2011 10:49 am

Re: How to download images only from gamefaqs.com

Post by Support »

It did the same to me. Could be the server slowing it down. If you scan any one of the /images/ URL like /images/3do/ it will work fine. I guess you just can't scan the whole site at once! Maybe you can using a single connection instead of 4 of them.
Your support team.
http://SoftByteLabs.com

avalanch
Posts: 43
Joined: Fri Mar 16, 2012 11:31 am

Re: How to download images only from gamefaqs.com

Post by avalanch »

ok thanks. Is it possible to make it crawl the subdomain of img.gamefaqs.net? Tricky one there... at first I thought it ended in gamefaqs.com

nvm it seems to be grabbing from the img.gamefaqs.net

User avatar
Support
Site Admin
Posts: 1892
Joined: Sun Oct 02, 2011 10:49 am

Re: How to download images only from gamefaqs.com

Post by Support »

It is set to crawl any URL that contain /images in it, including subdomains and even other sites so it should scan img.gamefaqs.net too. What do you mean by nvm?
Your support team.
http://SoftByteLabs.com

avalanch
Posts: 43
Joined: Fri Mar 16, 2012 11:31 am

Re: How to download images only from gamefaqs.com

Post by avalanch »

I was wrong, it is downloading them properly as you said. :mrgreen:

User avatar
Support
Site Admin
Posts: 1892
Joined: Sun Oct 02, 2011 10:49 am

Re: How to download images only from gamefaqs.com

Post by Support »

No problem :lol:
Your support team.
http://SoftByteLabs.com

Post Reply