Crawl Depth

TLDR: Crawl depth is the number of link hops between a seed URL and a given page. A depth limit stops a crawler from traveling further than a set number of clicks from where it started.

Crawl depth measures link distance, not directory nesting. A seed URL sits at depth 0. Every page linked directly from it is depth 1. Pages linked from those are depth 2, and so on. A page buried at /a/b/c/d/product may be depth 1 if the homepage links to it directly, while a page at /product may be depth 6 if it is only reachable through a long navigation chain.

Why Depth Limits Exist:

1. Termination: Without a limit, a crawler following links on a large or dynamically generated site may never finish.
2. Cost control: Page count grows geometrically with depth, so an unbounded crawl can consume an entire budget unexpectedly.
3. Relevance: Pages far from the seed are frequently less relevant to the crawl’s purpose.
4. Trap avoidance: Calendars, faceted filters, and session URLs generate infinite link chains that only a depth limit or crawl scope rule will stop.

How Page Count Grows:

The figures assume 50 new links per page and no duplicates, so real crawls grow more slowly once deduplication collapses repeated navigation links. The pattern still holds: each additional depth level can multiply the crawl by one or two orders of magnitude. Raising a limit from 3 to 4 is not an incremental change.

Choosing a Depth Limit:

The correct limit depends on how far target content sits from the seed, which is a property of site structure rather than a universal number.

Well-structured documentation sites typically expose every page within two or three hops of the table of contents. E-commerce catalogs usually reach product pages at depth 2 from a category listing, or depth 4 to 5 from the homepage. News archives can require deep pagination traversal to reach older content.

The practical method is to run a shallow test crawl, record the depth at which target pages first appear, then set the production limit one level beyond. Choosing a number without measuring usually means either missing content or paying for pages nobody wanted.

Depth and Seed Strategy:

Depth and seed selection are the same problem viewed from two directions. Better seeds reduce the depth required.

Seeding a homepage and setting depth 5 may reach the same product pages as seeding category URLs and setting depth 2, but the first configuration crawls vastly more irrelevant pages to get there. Where a sitemap is available, seeding from it can reduce required depth to 0 or 1, since every target URL is supplied directly.

Depth in Configuration:

{
  "seeds": ["https://example.com/category/laptops"],
  "max_depth": 2,
  "depth_rules": {
    "/product/*": 3,
    "/reviews/*": 1
  }
}

A global limit of 2 applies by default, product paths are permitted one extra hop to capture variants, and review paths are capped tightly. Per-pattern depth rules avoid the common tradeoff where raising the global limit to reach one section pulls in large volumes of unwanted pages elsewhere.

Depth vs. Breadth:

Depth limits interact with traversal order. A breadth-first crawler completes each depth level before advancing, so a crawl stopped early still has complete coverage of shallow levels. A depth-first crawler descends immediately and may reach the limit in one branch while leaving others untouched. For most data collection work, breadth-first with a depth cap gives more predictable coverage, as covered in breadth-first vs. depth-first crawling.

Bright Data’s Crawl API supports global and per-pattern depth limits, and reports the depth distribution of crawled pages so limits can be tuned against actual site structure.