Labrador

Labrador is a distributed web crawler (or spider), written in Perl. Primarily, it has been designed to be integrated with the Terrier search framework, designed and implemented by the Information Retrieval Group at the Department of Computing Science, University of Glasgow

Labrador is a final-year Honours project by Craig Macdonald, supervised by Dr Iadh Ounis and Vassilis Plachouras.

Labrador is suitable, for crawling a single website or Intranet, upto restricted domain Internet crawls. A modular interface allows customisation of filtering and crawler strategies to suit the application.