go spider online with Winfy

We have hosted the application go spider in order to run this application in our online workstations with Wine or directly.

Quick description about go spider:

An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only. Spider gets a Request in Scheduler that has url to be crawled. Then Downloader downloads the result(html, json, jsonp, text) of the Request. The result is saved in Page for parsing in PageProcesser. Html parsing is based on goquery package. Json parsing is based on simple JSON package. Jsonp will converse to json. Text form represents plain text content without a parser. The PageProcesser moduler only parse results. The moduler gets results(key-value pairs) and URLs to be crawled next step. These key-value pairs will be saved in PageItems and urls will be pushed in Scheduler.

Features:

• Requires Go 1.2 or higher
• Concurrent
• Fit for vertical communities
• Flexible, Modular
• Native Go implementation
• Can be expanded to an individualized crawler easily

Programming Language: Go.
Categories: