see : Search Engine in Erlang
author: tmaciejewski
created: 2010-05-25 12:51:55
crawler erlang information-retrieval search-engine


arachni : Web Application Security Scanner Framework
author: Arachni
created: 2010-08-25 16:06:38
analysis arachni audit crawler detection dom hack hacking javascript modular penetration-testing ruby scanner scanners security-audit sql-injection vulnerability-detection web-application xss


node-crawler : Web Crawler/Spider for NodeJS + server-side jQuery ;-)
author: bda-research
created: 2010-11-25 11:47:19
cheerio crawler extract-data javascript jquery nodejs spider


wombat : Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
author: felipecsl
created: 2011-11-14 05:03:24
crawler dsl ruby scraper


crawler : A node.js crawler support custom special crawl rules.
author: mxfli
created: 2012-01-11 15:12:36
crawler javascript


fscrawler : Elasticsearch File System Crawler (FS Crawler)
author: dadoonet
created: 2012-06-08 17:23:03
crawler elasticsearch java tika


Marsvin : Structural Crawler framework written in PHP
author: krolow
created: 2012-09-17 18:20:02
crawler framework parser php


gocrawl : Polite, slim and concurrent web crawler.
author: PuerkitoBio
created: 2012-09-19 02:08:15
crawler robots-txt go


scrapula : Scrapula is the licentious scraping library
author: j-a-m-l
created: 2012-10-10 11:49:42
crawler ruby scraper


scrapy-kaskus-crawler : Scraping kaskus.co.id, biggest indonesian forum
author: clasense4
created: 2012-11-30 07:51:08
crawler python scrapy


scrapy-blog-crawler : Crawl a blog url, and find all url from it, then save to mysql.
author: clasense4
created: 2012-12-20 10:21:11
crawler python scrapy


Html2Article : Html网页正文提取
author: stanzhai
created: 2013-01-01 08:38:36
article content crawler html spider topic c#


SpiderWho : A very fast whois crawler
author: lanrat
created: 2013-01-10 23:01:02
crawler whois python


scrapy-bhinneka-crawler : Scraping bhinneka.com, just for fun
author: clasense4
created: 2013-01-19 16:44:51
crawler python scrapy


ev-watchdog : Utilitário para monitorar e alertar sobre livros no Estante Virtual (http://estantevirtual.com.br/)
author: lsdr
created: 2013-02-13 00:36:31
crawler heroku ruby


xeHentai : Doujinshi downloader 绅士漫画下载
author: fffonion
created: 2013-03-23 11:18:34
crawler json-rpc python


webmagic : A scalable web crawler framework for Java.
author: code4craft
created: 2013-04-23 12:57:36
crawler framework java scraping


4chan-downloader : Python3 script to continuously download all images/webms of a 4chan thread - without installation
author: Exceen
created: 2013-05-09 14:52:04
4chan 4chan-downloader crawler download download-manager downloader python python3


gopa-abandoned : GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）
author: medcl
created: 2013-07-08 08:56:23
crawler golang lightweight spider go


pycrawler : Crawler in Python-3.6.x
author: vinitkumar
created: 2013-07-08 13:40:33
crawler python python35 python36 utils


opensearchserver : Open-source Enterprise Grade Search Engine Software
author: jaeksoft
created: 2013-07-18 16:03:33
crawler custom-search enterprise indexing java lucene ocr opensearchserver search search-engine synonyms webcrawler webcrawling


skyscanner-crawler : Crawls http://www.skyscanner.net and lists possible combined destinations in given month.
author: andre-wojtowicz
created: 2013-09-01 21:22:29
airline crawler flight selenium trip-planning python


fess : Fess is very powerful and easily deployable Enterprise Search Server.
author: codelibs
created: 2013-09-13 04:50:05
crawler elasticsearch enterprise-search enterprise-software full-text-search fulltext-search java lastaflute lucene search search-engine


google-group-crawler : Get (almost) original messages from google group archives. Your data is yours.
author: icy
created: 2013-09-23 01:48:47
bash cookie crawler google ownership wget shell


scrapy-rakitan : Scraping Rakitan.com
author: clasense4
created: 2013-10-03 21:31:47
crawler python scrapy


prerender-java : java framework for prerender
author: greengerong
created: 2013-10-15 14:48:26
angular1 crawler java prerender prerendered-page seo


blog.cz-backup-robot :
author: jakubboucek
created: 2013-11-06 08:01:17
crawler php


crawler-benchmark : A Reference Framework for the Automated Exploration of Web Applications. Provides some general web features to let you test crawlers in a well defined environment.
author: WebMole
created: 2013-11-11 22:50:07
crawler crawler-benchmark flask site tool css


cordis-projects-crawler : Retrieves data of all projects on CORDIS
author: juzraai
created: 2013-11-23 15:14:13
cordis cordis-parser crawler open-data kotlin


GoogleScraper : A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
author: NikolaiT
created: 2013-12-06 15:34:20
crawler python scraping search-engine search-engine-optimization search-engines html


fetchbot : A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
author: PuerkitoBio
created: 2014-01-06 12:45:18
crawler robots-txt go


arachnid : Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
author: zrashwani
created: 2014-01-06 17:32:41
crawler php scraping seo


proxy_manager : Ruby proxy manager. Gem for easy usage proxy in parser/web bots.
author: kirillplatonov
created: 2014-01-18 11:19:56
crawler parser proxy ruby


html-query : A fluent and functional approach to querying HTML
author: h12w
created: 2014-01-20 06:55:09
crawler dom go golang golang-package html parser


pyspider : A Powerful Spider(Web Crawler) System in Python.
author: binux
created: 2014-02-21 19:18:47
crawler python


scrapher : A web scraper for PHP to easily extract data from web pages
author: Laurentvw
created: 2014-03-09 15:13:41
crawl-page crawler extract parser php scraper scraping selector webscraping


immobilienscout24-tracker : A php based web crawler to track Immobilienscout24.de website for new entries.
author: arkadiusjonczek
created: 2014-03-28 19:22:49
crawler php tracker


go_spider : [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
author: hu17889
created: 2014-05-19 03:23:26
crawler go pipeline schedule spider


5d6d_archiver_crawler : 5d6d discuz论坛压缩档爬虫
author: ycrao
created: 2014-09-13 03:24:53
5d6d autohotkey bbs crawler discuz


hacker-news-digest : :newspaper: A responsive interface of Hacker News with summaries and illustrations.
author: polyrabbit
created: 2014-09-17 13:34:44
article content crawler data-extraction extract-summaries hacker-news hacker-news-digest hacker-news-reader html machine-learning news-aggregator python rss spider topic


minicrawler : Multiplexing web client supporting HTTP/2 and WHATWG URL compliant parser written in C
author: testomato
created: 2014-10-14 14:26:11
agpl c cookie crawler http2 icu multiplexing nghttp2 parser ssl whatwg


spidercrawl : A ruby gem for web crawling and scraping.
author: belsonheng
created: 2014-10-21 06:19:31
crawler rubygems ruby


node-cesspoll : :poop: Turd Miner Node Module
author: dasantonym
created: 2014-10-24 17:29:59
crawler news poopetry potty-humour javascript


ff-moulinette : A dead simple character sheet crawler for Final Fantasy XIV characters.
author: giant-teapot
created: 2014-10-26 12:01:39
crawler ffxiv python


memex-explorer : Viewers for statistics and dashboarding of Domain Search Engine data
author: nasa-jpl-memex
created: 2014-10-27 16:00:40
ache anaconda apache crawler dashboard domain-discovery memex-explorer miniconda nutch tika python


hQuery.php : An extremely fast web scraper that parses megabytes of HTML in a blink of an eye. PHP5.3+, no dependencies.
author: duzun
created: 2014-10-29 20:07:13
crawler hquery html parser php psr-0 psr-4 scraper selectors


glassycrawl :
author: tomakant
created: 2014-11-05 17:47:30
crawler css database employer glassdoor glassdoor-reviews html vectorizer javascript


grucrawler : Simple Ruby crawler
author: slava-vishnyakov
created: 2014-11-18 21:06:39
crawler ruby


chef : Cube 2: Sauerbraten spy bot: collects IP-name combinations from extinfo and provides a web interface to search them.
author: sauerbraten
created: 2014-12-13 05:31:14
crawler go irc-bot sauerbraten


scrapple : A framework for creating semi-automatic web content extractors
author: AlexMathew
created: 2014-12-13 11:32:02
beautifulsoup crawler css-selector extractor lxml python scrapers scraping scrapy selector selector-expression tutorial web-scraper web-scraping xpath-expression


childcrawler : Easily a mini crawler as child.
author: sott0n
created: 2014-12-18 13:10:50
crawler python


httpseed : Cartographer: A new type of seed for the Bitcoin network
author: bitcoinj
created: 2014-12-19 02:04:49
bitcoin crawler kotlin


leboncoin-crawler : Crawler for leboncoin.fr
author: rfussien
created: 2015-01-08 16:48:12
crawler leboncoin php html


Sitemap-Generator-Crawler : Script that generates a sitemap by crawling a given URL
author: knyzorg
created: 2015-01-11 19:37:56
crawler php seo sitemap sitemap-generator xml


movie-showtimes : Web Service & Android Application to look up Vietnam movie showtimes
author: trungdq88
created: 2015-01-21 01:40:52
crawler java movie-showtimes theater


json-web-crawler : Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.
author: Knovour
created: 2015-01-24 16:52:44
crawler javascript jquery json web-crawler


grab-site : The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
author: ludios
created: 2015-02-05 05:01:19
archiving crawl crawler spider warc python


arachnod : High performance crawler for Nodejs
author: risyasin
created: 2015-02-09 18:50:35
cheerio crawler javascript nodejs redis scraper spider


tsec : 台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler
author: Asoul
created: 2015-02-12 14:31:54
crawler stock taiwan taiwan-stock-exchange taiwan-stock-market


crawlcheck : Extensible web crawler
author: eghuro
created: 2015-02-18 15:16:08
configuration crawler http plugin python robots-txt sitemap


TiebaManager : 百度贴吧吧务管理工具，自动扫描帖子并处理违规帖
author: xfgryujk
created: 2015-02-20 18:04:48
crawler tieba c++


tsrtc : 台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler
author: Asoul
created: 2015-02-24 10:32:10
crawler stock taiwan taiwan-stock-exchange taiwan-stock-market javascript


Laravel-Crawler-Detect : A Laravel wrapper for CrawlerDetect - the web crawler detection library
author: JayBizzle
created: 2015-03-18 20:59:36
bot crawler detect laravel php spider


Crawler : Swiftea - Crawler
author: Swiftea
created: 2015-03-21 20:57:09
crawler python


Crawler-Detect : 🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
author: JayBizzle
created: 2015-03-23 20:05:37
bots crawler detect php spider user-agent


webb : Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!
author: hardikvasa
created: 2015-03-29 04:32:18
crawl-pages crawler python-library python


baidu-hi-crawler : Crawler for Baidu Hi, which is a blog space which will be closed on May 2015.
author: sqybi
created: 2015-04-08 02:15:02
c-sharp crawler spider c#


TumblTwo : TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.
author: johanneszab
created: 2015-04-08 15:00:21
crawler downloader photos ripper tumblr tumblr-blog tumblr-downloader videos c#


vcrawler-engine : database and sites api + WPF client
author: v0vc
created: 2015-04-13 09:40:14
crawler gui youtube youtube-dl c#


php-crawler : A php crawler that finds emails on the internets
author: hedii
created: 2015-04-15 12:14:14
crawler laravel php php-crawler vue vuejs webcrawler webscraper webscraping


domfind : A Python DNS crawler to find identical domain names under different TLDs.
author: diogo-fernan
created: 2015-04-24 10:18:38
crawler cybersecurity dns phishing python soa tld


Lcrawl : 一只优雅的正方教务系统爬虫。
author: lndj
created: 2015-04-28 03:54:09
crawler exam grade php zhengfang


wots : Wizard of the Search
author: refaim
created: 2015-05-08 01:06:25
crawler cross-platform desktop gui magic-the-gathering mtg pyqt5 python russian search


seenreq : Generate an object for testing if a request is sent, request is Mikeal's request.
author: mike442144
created: 2015-05-10 14:44:26
crawler duplicates-removed post request spider url javascript


sitemap-generator-cli : Creates an XML-Sitemap by crawling a given site.
author: lgraubner
created: 2015-05-17 16:15:30
cli crawler google seo sitemap xml-sitemap javascript


node-w3c-validator-cli : Crawls a given site and checks for W3C validity.
author: lgraubner
created: 2015-05-25 19:14:15
cli crawler w3c w3c-validator javascript


fluent-plugin-github-activities : Provides ability to watch public activities on GitHub.
author: clear-code
created: 2015-05-28 01:13:49
crawler fluentd fluentd-plugin github-api ruby


SeleniumDemo : Selenium automation test framework
author: tobecrazy
created: 2015-06-22 04:13:22
container crawler docker docker-compose jenkins maven pip python selenium selenium-grid selenium-webdriver snapshot html


TeamFortressOutpostApi : :repeat: An API wrapper for the TF2 Outpost platform. A platform to find great deals for your Team Fortress 2, Counter-Strike: Global Offensive and Dota 2 items with zero hassle.
author: igeligel
created: 2015-06-30 23:04:52
bot bot-framework crawler steam steam-api steambot teamfortress2 c#


pholcus : [Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
author: henrylee2cn
created: 2015-07-03 15:40:41
crawler cross-platform-crawler distributed-crawler fastest-crawler golang high-concurrency-crawler multi-interface spider go


CrawlerDetector : Rapid web crawler detector
author: Macsch15
created: 2015-07-07 01:29:49
bot bot-detector crawler crawler-detector php


CrawlerTutorial : 爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例
author: leVirve
created: 2015-07-18 13:12:34
api api-wrapper crawler multiprocessing parse ptt search spider tutorial python


stock-newspaper-crawler : [UNMAINTAINED]Crawl 4 kinds of finance newspaper corpus (from CCSTOCK.CN).
author: ysh329
created: 2015-07-22 01:59:14
corpus crawled-data crawler database stock-newspaper-crawler python


express-bot : Crawler(robots) decision middleware for Express
author: fkei
created: 2015-07-22 13:26:20
bot crawler express-bot expressjs robot javascript


Crawler-DKXT-DH-2015 : Crawler thống kê danh sách thí sinh đăng kí xét tuyển DH 2015 vào các trường.
author: duyetdev
created: 2015-08-03 17:26:01
crawler duyetdev hcm mongodb nodejs scraping uit university van-duyet-le javascript


fii : API para recuperar informações sobre FII
author: riquellopes
created: 2015-08-08 19:00:27
crawler investiment mongodb nodejs html


woid : Simple news aggregator displaying top stories in real time
author: vitorfs
created: 2015-08-12 17:10:40
crawler django news python


multitor : Make multiple Tor instances in Node.Js
author: warlock
created: 2015-08-13 09:45:43
crawler onion port proxy socket tor javascript


rovers : Rovers is a service to retrieve repository URLs from multiple repository hosting providers.
author: src-d
created: 2015-08-17 15:59:59
bitbucket cgit crawler github html


ngMeta : Dynamic meta tags in your AngularJS single page application
author: vinaygopinath
created: 2015-08-21 21:37:42
angularjs crawler meta-tags opengraph seo ui-router javascript


robotparser-scala : robotparser-scala implements a parser for the robots.txt file format in Scala.
author: bizreach
created: 2015-08-31 05:41:15
crawler parser scala sitemap


youdao : 有道词典网页爬虫
author: hellflame
created: 2015-09-05 13:31:50
crawler youdao python


cffie : CFFie query SBB/CFF/FFS connections from https://cff.ch
author: maidmaid
created: 2015-09-26 08:50:34
crawler php sbb-cff-ffs symfony symfony-console


goose-parser : Universal scrapping tool, which allows you to extract data using multiple environments
author: redco
created: 2015-10-07 08:38:07
browser crawler goose jsdom parser parsing phantomjs scraper scraping javascript


Chemrtron : A document viewer; fuzzy match incremental search.
author: cho45
created: 2015-10-08 13:09:15
crawler document-viewer electron increment javascript


imooc-crawler : [Obsolete] imooc web crawler in Node.js（使用 Node.js 编写的慕课网爬虫）
author: monkeym4ster
created: 2015-10-10 06:30:16
crawler imooc node-js nodejs spider javascript


Scan-T : a new crawler based on python with more function including Network fingerprint search
author: nanshihui
created: 2015-10-14 10:37:41
crawler netfingerprint python sybersecurity c


autodata-crawler : Content (car data and images) crawler for http://auto-data.net
author: bekce
created: 2015-10-26 16:24:43
autodata-crawler car-data car-images crawler shell


crawler : An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
author: spatie
created: 2015-11-02 16:22:09
concurrency crawler guzzle php


forum-search : craw, store and index data for later search
author: at15
created: 2015-11-13 13:10:34
crawler hadoop java


Cookies help us deliver our services. By using our services, you agree to our use of cookies Learn more