Search found 29 matches
- Fri Sep 06, 2013 8:05 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Or would it be better to output a separate file for each keyword/header result since there might be more than one result for each keyword/header query?
- Fri Sep 06, 2013 6:53 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
The script runs great but I am having trouble breaking out the results from the output file in an organized way. Would it be possible to output the file with column headings referring to each keyword and header result? if there is no result, the field would be empty?
- Tue Sep 03, 2013 8:27 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
I've noticed in the some of the data captured there is a line feed? and maybe a tab occasionally. Is there a way to encapsulate the output or remove the line feed/tab when the data is captured?
- Mon Aug 26, 2013 9:23 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
The script runs great but I am having trouble breaking out the results from the output file in an organized way. Would it be possible to output the file with column headings referring to each keyword and header result? if there is no result, the field would be empty?
- Wed Aug 21, 2013 9:06 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
If I wanted to capture phone numbers, URLs, and ZipCode in my script using: \((?<AreaCode>\d{3})\)\s*(?<Number>\d{3}(?:-|\s*)\d{4})(?x) # Phone numbers (?<Protocol>\w+):\/\/(?<Domain>[\w.]+\/?)\S*(?x) # URL (?<Zip>\d{5})-(?<Sub>\d{4})(?x) # Zip Codes Can I insert these expressions directly to the sc...
- Sun Aug 18, 2013 8:21 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
For the script in this thread, is there a way to retrieve the server IP address for each URL and place it in the .txt output file?
- Thu Aug 01, 2013 9:23 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
- Thu Aug 01, 2013 9:18 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Works great. thanks! I see the code button - I'll use it next time.
Can BrownRecluse compile and execute javascript code?
Can BrownRecluse compile and execute javascript code?
- Wed Jul 31, 2013 7:01 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
I made some slight modifications to the last script you provided by this thread. I wanted the output to be in a delimited format. I chose "|" as my delimiter because a "," caused issues in reading the output file into Excel (there is probably a better delimiter to use). I also ch...
- Tue Jul 30, 2013 6:55 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
From this URL: http://www.usaultimate.org/index.html if I look at the source on lines 5 and 6 I see: <title id="PageTitle">USA Ultimate | Home Page</title> <meta name="keywords" content="USA Ultimate,UPA,Ultimate,Disc,National Governing Body,Ultimate Players Association,US U...
- Tue Jul 30, 2013 1:37 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Using this script, how can I add pull of meta data from header- keywords and content- into the delimited output file?
- Thu Jul 25, 2013 10:12 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Could the script run a timer that forces that spider to go to the next URL once the timer expires? Would this circumvent the login popup?
- Wed Jul 24, 2013 2:37 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
- Wed Jul 24, 2013 1:53 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
If I uninstalled IE would that prevent the login window from coming up and interrupting the process?
- Wed Jul 24, 2013 1:10 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
thanks. Did you see my question about login screens?
- Wed Jul 24, 2013 12:55 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
How can I add "http://www." to the beginning of the URLs in the source .txt file before the spider tries to access the URL?
- Wed Jul 24, 2013 12:50 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
If the spider encounters a site with a login screen, is it possible to "cancel' the login and proceed to the next URL in the list?
- Wed Jul 10, 2013 11:19 pm
- Forum: BrownRecluse
- Topic: Extracting street address, phone number, and email from a page
- Replies: 1
- Views: 18266
Extracting street address, phone number, and email from a page
For the USA, is it possible to extract the street address, city, state, zip,(possibly phone and email) without knowing where this text exist on a webpage and return the extracted information in a text file? For example, have a spider read a text file of URLs. http://www.kennedy-center.org http://www...
- Thu Jul 04, 2013 4:33 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
- Thu Jul 04, 2013 7:47 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Is it possible to search for the keywords in both the link and the visible text on the page that references the link? Then return the link as it does is the existing code. Sometimes it says "Contacts" on the page but the link does not contain the word. Does that make sense?
- Sat Jun 29, 2013 7:23 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Perfect! Thank you
- Fri Jun 28, 2013 6:38 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
I mean that some of the pages have the keywords on them so at least a few of the URLs should have returned in the output.
- Fri Jun 28, 2013 2:21 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
The input file does have a single URL per line- same file I ran successfully with the earlier code. I left the third line as is. The keywords were on the URL referenced pages.
- Fri Jun 28, 2013 5:06 am
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
I ran it today but the code produced no output. I selected the source file when prompted - do I need to do any thing else?
- Wed Jun 19, 2013 12:08 pm
- Forum: BrownRecluse
- Topic: Website lookup from a list and link capture
- Replies: 48
- Views: 142530
Re: Website lookup from a list and link capture
Ok. Np. Thank you.