crawler controls
Spider settings
Understand the core crawler settings, including user agent, crawl politeness, and scope controls.
User agent
The user agent tells servers what kind of crawler is requesting the page. Choose the setting that best matches the type of crawl you want to run and the site behavior you need to observe.
Crawl politeness
Politeness settings help you control crawl pace. Slower crawls are gentler on servers and useful for larger sites or sensitive infrastructure.
Scope and limits
Scope controls help you avoid collecting pages that do not matter to the job. Use them to keep projects smaller, cleaner, and easier to analyze.