Incremental Crawling

TLDR: Incremental crawling refetches only the pages likely to have changed since the last run, instead of recrawling an entire site every time. It keeps data fresh while cutting request volume dramatically.

Incremental crawling treats a crawl as an ongoing process rather than a one time event. The first run establishes a baseline. Every run after that focuses on what has changed: new pages, updated pages, and pages that have disappeared. On a site where only two percent of content changes between runs, a full recrawl spends 98 percent of its budget confirming that nothing happened.

Change Detection Methods:

1. Conditional requests: Send If-Modified-Since or If-None-Match and let the server answer with 304 Not Modified, which costs almost no bandwidth.
2. Content hashing: Hash the meaningful content of each page and compare against the stored hash from the previous crawl.
3. Sitemap lastmod: Read the lastmod field from the site’s XML sitemap and recrawl only entries with a newer timestamp.
4. Feed monitoring: Poll RSS or JSON feeds that announce new content, which is far cheaper than crawling to discover it.
5. Structural signals: Watch listing pages and pagination for new entries rather than revisiting every detail page.

Conditional Requests:

GET /products/4417 HTTP/1.1
Host: example.com
If-None-Match: "a3f8c21b9"
If-Modified-Since: Tue, 15 Sep 2026 09:14:00 GMT

HTTP/1.1 304 Not Modified
ETag: "a3f8c21b9"

A 304 response carries no body. The crawler learns the page is unchanged for the cost of headers alone, typically a few hundred bytes against a page that might be 200 KB. On a catalog of 100,000 mostly static pages, this is the difference between a 20 GB recrawl and a 30 MB one.

The limitation is that support is inconsistent. Many sites do not send ETags, and some send values that change on every request regardless of content, which makes the mechanism useless on those targets.

Content Hashing:

When a server will not help, the crawler can determine change for itself by hashing content after fetching. This does not save bandwidth, since the page must be retrieved either way, but it does prevent unnecessary reprocessing, storage writes, and downstream pipeline work.

The difficulty is deciding what to hash. Hashing raw HTML produces a different value on nearly every fetch, because timestamps, session tokens, rotating advertisements, and view counters all change independently of the content anyone cares about. Effective hashing targets extracted fields rather than markup: the product title, price, and description rather than the page that contained them.

Recrawl Frequency:

Fixed intervals are a starting point, not an endpoint. Adaptive scheduling performs better: track how often each individual URL actually changes, then shorten the interval for volatile pages and extend it for stable ones. A page that has been identical for 30 consecutive crawls does not need checking as often as one that changes daily.

Detecting Deletions:

Incremental crawling handles new and updated pages naturally, since both surface through normal discovery. Removed pages are harder, because absence produces no signal.

A page may return 404 or 410, which is unambiguous. It may also return 200 with a “no longer available” message, redirect to a category page, or simply stop being linked while remaining reachable by direct URL. Only the first case is detectable without explicitly revisiting.

The common approach is a periodic full sweep, less frequent than the incremental runs, that visits every known URL and marks anything missing or unreachable. Monthly sweeps against daily increments is a reasonable default for most catalogs.

State Requirements:

Incremental crawling requires persistent state that a one off crawl does not. At minimum the crawler must store, per URL, the last crawl timestamp, the last ETag or Last-Modified value, a content hash, an observed change frequency, and a consecutive failure count.

This state is what distinguishes an incremental crawler from a scheduled full crawler. Without it, every run starts from zero and the efficiency gain disappears entirely.

Bright Data’s Scraper APIs and Datasets maintain this state internally and deliver refreshed records on a schedule, which removes the need to build and operate change tracking infrastructure.