Search found 29 matches

by gcarmich
Fri Sep 06, 2013 8:05 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Or would it be better to output a separate file for each keyword/header result since there might be more than one result for each keyword/header query?
by gcarmich
Fri Sep 06, 2013 6:53 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

The script runs great but I am having trouble breaking out the results from the output file in an organized way. Would it be possible to output the file with column headings referring to each keyword and header result? if there is no result, the field would be empty?
by gcarmich
Tue Sep 03, 2013 8:27 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

I've noticed in the some of the data captured there is a line feed? and maybe a tab occasionally. Is there a way to encapsulate the output or remove the line feed/tab when the data is captured?
by gcarmich
Mon Aug 26, 2013 9:23 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

The script runs great but I am having trouble breaking out the results from the output file in an organized way. Would it be possible to output the file with column headings referring to each keyword and header result? if there is no result, the field would be empty?
by gcarmich
Wed Aug 21, 2013 9:06 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

If I wanted to capture phone numbers, URLs, and ZipCode in my script using: \((?<AreaCode>\d{3})\)\s*(?<Number>\d{3}(?:-|\s*)\d{4})(?x) # Phone numbers (?<Protocol>\w+):\/\/(?<Domain>[\w.]+\/?)\S*(?x) # URL (?<Zip>\d{5})-(?<Sub>\d{4})(?x) # Zip Codes Can I insert these expressions directly to the sc...
by gcarmich
Sun Aug 18, 2013 8:21 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

For the script in this thread, is there a way to retrieve the server IP address for each URL and place it in the .txt output file?
by gcarmich
Thu Aug 01, 2013 9:18 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Works great. thanks! I see the code button - I'll use it next time.

Can BrownRecluse compile and execute javascript code?
by gcarmich
Wed Jul 31, 2013 7:01 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

I made some slight modifications to the last script you provided by this thread. I wanted the output to be in a delimited format. I chose "|" as my delimiter because a "," caused issues in reading the output file into Excel (there is probably a better delimiter to use). I also changed the script to ...
by gcarmich
Tue Jul 30, 2013 6:55 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

From this URL: http://www.usaultimate.org/index.html if I look at the source on lines 5 and 6 I see: <title id="PageTitle">USA Ultimate | Home Page</title> <meta name="keywords" content="USA Ultimate,UPA,Ultimate,Disc,National Governing Body,Ultimate Players Association,US Ultimate,Spirit of the Gam...
by gcarmich
Tue Jul 30, 2013 1:37 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Using this script, how can I add pull of meta data from header- keywords and content- into the delimited output file?
by gcarmich
Thu Jul 25, 2013 10:12 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Could the script run a timer that forces that spider to go to the next URL once the timer expires? Would this circumvent the login popup?
by gcarmich
Wed Jul 24, 2013 1:53 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

If I uninstalled IE would that prevent the login window from coming up and interrupting the process?
by gcarmich
Wed Jul 24, 2013 1:10 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

thanks. Did you see my question about login screens?
by gcarmich
Wed Jul 24, 2013 12:55 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

How can I add "http://www." to the beginning of the URLs in the source .txt file before the spider tries to access the URL?
by gcarmich
Wed Jul 24, 2013 12:50 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

If the spider encounters a site with a login screen, is it possible to "cancel' the login and proceed to the next URL in the list?
by gcarmich
Wed Jul 10, 2013 11:19 pm
Forum: BrownRecluse
Topic: Extracting street address, phone number, and email from a page
Replies: 1
Views: 6175

Extracting street address, phone number, and email from a page

For the USA, is it possible to extract the street address, city, state, zip,(possibly phone and email) without knowing where this text exist on a webpage and return the extracted information in a text file? For example, have a spider read a text file of URLs. http://www.kennedy-center.org http://www...
by gcarmich
Thu Jul 04, 2013 7:47 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Is it possible to search for the keywords in both the link and the visible text on the page that references the link? Then return the link as it does is the existing code. Sometimes it says "Contacts" on the page but the link does not contain the word. Does that make sense?
by gcarmich
Sat Jun 29, 2013 7:23 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Perfect! Thank you
by gcarmich
Fri Jun 28, 2013 6:38 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

I mean that some of the pages have the keywords on them so at least a few of the URLs should have returned in the output.
by gcarmich
Fri Jun 28, 2013 2:21 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

The input file does have a single URL per line- same file I ran successfully with the earlier code. I left the third line as is. The keywords were on the URL referenced pages.
by gcarmich
Fri Jun 28, 2013 5:06 am
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

I ran it today but the code produced no output. I selected the source file when prompted - do I need to do any thing else?
by gcarmich
Wed Jun 19, 2013 12:08 pm
Forum: BrownRecluse
Topic: Website lookup from a list and link capture
Replies: 48
Views: 50376

Re: Website lookup from a list and link capture

Ok. Np. Thank you.