JavaScript Rendering

Modern websites rely heavily on JavaScript to render content. SPAs, React apps, dynamically loaded product listings — if your crawler can't execute JavaScript, it's missing the real page.

How it works

Crawltable uses a built-in headless browser to render every page. When a URL is crawled:

1. The HTML is fetched and parsed
2. JavaScript is executed in a sandboxed browser environment
3. The page is given time to fully render (network idle detection)
4. The final DOM is captured for extraction

This means you see exactly what a user (and Googlebot) sees — not just the raw HTML source.

Render timing

Crawltable uses network idle detection to determine when a page has finished rendering. It waits until:

• No new network requests have been made for 500ms
• The DOM has stabilized (no new mutations)

For pages with lazy-loaded content or infinite scroll, you can increase the render timeout in your crawl configuration.

Performance considerations

JavaScript rendering is significantly more resource-intensive than static HTML crawling. Each page requires a browser instance, memory, and CPU time.

On a modern machine:

• Static crawling: 50–100+ pages/second
• JS rendering: 5–15 pages/second (depending on page complexity)

To optimize large crawls, use selective rendering — only render pages that actually need JavaScript, and crawl the rest statically.

Troubleshooting

If pages aren't rendering correctly:

• Increase the render timeout — some pages take longer to fully load
• Check for bot detection — some sites block headless browsers; Crawltable includes stealth options to mitigate this
• Review the rendered HTML — in the page detail view, compare the raw HTML vs. rendered HTML to see what JavaScript is adding