crwlr online with Winfy

We have hosted the application crwlr in order to run this application in our online workstations with Wine or directly.

Quick description about crwlr:

This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could just load actually all links it is finding (and is allowed to load according to the robots.txt file), then it would just load the whole internet (if the URL(s) it starts with are no dead end). Or it can be restricted to load only links matching certain criteria (on same domain/host, URL path starts with "/foo",.) or only to a certain depth. A depth of 3 means 3 levels deep. Links found on the initial URLs provided to the crawler are level 1 and so on.

Features:

• Crawler Politeness (respecting robots.txt, throttling,.)
• Get absolute links from HTML documents
• Get sitemaps from robots.txt and get all URLs from those sitemaps
• Crawl (load) all pages of a website
• Use any HTTP methods (GET, POST,.) and send any headers or body
• Iterate over paginated list pages

Programming Language: PHP.
Categories: