Software
From Online Research Methods comprehensive resource
There are several software available
Blinker
Cyclist
Curl
Microsoft Site Analyst
Nutch <http://lucene.apache.org/nutch/about.html#Overview>
Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc. For more information about Nutch, please see the Nutch wiki.
SocSciBot crawler - Wolverhampton <http://socscibot.wlv.ac.uk/>
SocSciBot is a Web site crawler designed for research purposes. Together with its supporting programs SocSciBot Tools and Cyclist, it can be used to conduct link analysis on a site or collection of sites, or to run a search engine on a collection of sites. These programs can also be used in teaching, to illustrate how link analysis and search engines work. Note that SocSciBot does not work well on web sites with non-ASCII URLs and Cyclist does not work well on web sites with non-ASCII text.
LexiURL link analyser - Wolverhampton <http://lexiurl.wlv.ac.uk/>
LexiURL URL list and Link list Analysis Software
LexiURL is free software designed to analyse lists of web page URLs and lists of hyperlinks. You will need to have a list of the links or URLs that you wish to analyse. Test examples are available for practice and the associated LexiURL Searcher software is available to help you create your URL or link lists. LexiURL can download all the pages an URL list, then extract and analyse their links. It performs simple analyses and more complex analyses, using the Alternative Document Model concept. Note that LexiURL does not work well with non-ASCII URLs.
LexiURL generates summary statistics based upon the lexical structure of URLs. These statistics primarily relate to URL Top-Level Domains (TLDs), web sites and full URLs. For example, the results may include the number of URLs from the UK TLD (with domain names ending in .uk), the Wolverhampton University Web site (with domain names ending in wlv.ac.uk) and the number of links pointing to the Wolverhampton University home page (http://www.wlv.ac.uk/). A sample set of statistics is available to illustrate the kind of information that may be produced.
The following list of uses of link or URL list analysis illustrates the wide potential of this technique and each application is (or will be) accompanied by a link to specific instructions on how to conduct that particular type of study.
Mozdeh blog analyser - RSS and blog analysis - Wolverhampton
