Westbyte's IDA implements that option by actually changing the date of the downloaded file to match the original.
Here is a simple example that demonstrates the problem
Code: Select all
[BlackWidow v6.00 filters]
URL = https://www.library.vanderbilt.edu/govt/images/
[ ] Expert mode
[x] Scan everything
[ ] Scan whole site
Local depth: 0
[ ] Scan external links
[ ] Only verify external links
External depth: 0
Default index page:
Browser user agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)
Startup referrer:
[ ] Slow down by 2:4 seconds
4 threads
[end]