Vibe Coding Discovercrawlberg
View on GitHubHigh-performance web crawling engine with bindings for 11 languages
Rust web-crawling engine that turns sites into clean Markdown and structured metadata, with 14 language bindings plus an MCP server and agent skills for feeding AI agents and RAG pipelines. SSRF-safe, robots-aware, optional headless-browser rendering.
Use Cases
Convert web pages to clean Markdown for LLM and RAG ingestionCrawl whole sites into structured datasets for training or retrievalGive AI agents live web access through the bundled MCP serverExtract structured metadata, JSON-LD, Open Graph and linksBatch-scrape hundreds of URLs concurrentlyRender JavaScript-heavy SPAs via headless browser fallbackScrape documentation sites into Markdown knowledge basesRun crawls from CLI, REST API, Docker or 14 language SDKs
Built With
- Language
- Rust
- Frameworks
- Tokio · PyO3 · napi-rs · wasm-bindgen · maturin · Rustler · Magnus · ext-php-rs · cbindgen · headless Chrome · Docker
Tags
web-crawling · web-scraping · markdown · mcp · mcp-server · rust · multi-language-bindings · ffi · headless-browser · rag-data · json-ld · llm-context · cli · wasm · sitemap · robots-txt