DEV Community
•
2026-08-17 14:00
Rate Limiting a Crawler Across Node Cluster Workers
Code: Megapixel99/webCrawler
This project is a small search engine I wrote to understand how one works: a crawler, an inverted index in MongoDB, and a BM25 ranker, all written by hand instead of reaching for Elasticsearch.
The first version was not something you should point at the open web. It honored noindex and nofollow meta tags, but it ignored robots.txt entirely and had no rate limiting,...