Krawler

Modern web crawling and scraping library enables building efficient web crawlers with features like intuitive DSL, concurrent crawling, flexible data extraction, robots.txt compliance, and smart caching.