The Pipeline needs your project: the more projects, the more efficiƫnt our infrastructure becomes.

Example project; Bavarian Food

One of the first Tasks for the Pipeline was contributed by the University of Passau (Germany). It want to detect (and extract) websites in Bayern (county of Bavaria) which relate to local food. In the first step, their Task contains all webpages written in German (under any domain) or in any language under TLD .bayern, which have an extracted text of at least 300 words. It also report discovered city-names.

Example project; link extract

For the use of other infrastructural projects, related to the Skrodon Project, there is link extractor Task. For each HTML response, <meta>, <link>, and href/src link data is collected for further processing. Links and data are nicely normalized.

Ideas for student projects

It is not hard to extract many useful facts from huge numbers of web-pages via the Crawl Pipeline, but it still has some challenges which make this into educational student projects.

  • Count distribution of web-server software;
  • Count distribution of authoring tools;
  • Estimate the percentage of internet which CommonCrawl collects monthly;
  • Detector for pages which need headless crawling;
  • Discover the languages available for certain page, from response headers and OpenGraph;
  • Build a simple searchable index based on OpenGraph and "classic" meta fields only;

You may ideas to add to this list. We will help the students to collect the right raw data.