Scrapers
unclecode / crawl4ai
Open-source, LLM-friendly web crawler and scraper that outputs clean markdown.
Crawl4AI turns any website into structured, AI-ready data with async crawling and extraction strategies.
pip install -U crawl4ai && crawl4ai-setup
Setup guide
1. Install
pip install -U crawl4ai then crawl4ai-setup.
2. Crawl
Use AsyncWebCrawler().arun(url=...) to get markdown.