crawler online with Winfy

We have hosted the application crawler in order to run this application in our online workstations with Wine or directly.

Quick description about crawler:

A toolchain for bringing web2 to web3. IPFS daemon should be launched. Download enwiki-latest-all-titles to crawler root dir. Basically, there are two main functions provided by crawler tool. The first one is to parse wiki titles and submit links between keywords and wiki pages. Also, crawler has separate command upload-duras-to-ipfs to upload files to local IPFS node. All DURAs are collected under single root unixfs directory.

Features:

• Note: Requires Go 1.12+
• IPFS daemon should be launched
• Download enwiki-latest-all-titles to crawler root dir
• Basically, there are two main functions provided by crawler tool
• Parse wiki titles and submit links between keywords and wiki pages
• Upload duras to IPFS

Programming Language: Go.
Categories: