Boilerplate Removal

TLDR: Boilerplate removal strips the repeated elements that surround a page’s main content, such as navigation, footers, advertisements, and related article panels, leaving only the content unique to that page.

Boilerplate removal, also called content extraction or main content detection, addresses a structural problem with web pages: most of what a page contains is not what the page is about. A news article of 800 words typically ships inside a document containing a navigation menu, a subscription prompt, a sidebar of trending stories, a comment section, and a footer sitemap. The article may represent under 10 percent of the visible text.

What Counts as Boilerplate:

  1. Navigation: Headers, menus, breadcrumbs, and category links that appear identically on every page.
  2. Footers: Sitemaps, legal links, contact details, and social icons.
  3. Advertisements: Display units, sponsored blocks, and promotional banners.
  4. Related content: Trending panels, you might also like modules, and tag clouds.
  5. Interface chrome: Cookie notices, newsletter prompts, paywall overlays, and share buttons.
  6. User generated sections: Comments, which are sometimes boilerplate and sometimes the most valuable part of the page, depending entirely on the use case.

Detection Methods:

Method How it decides Strength Weakness
Semantic tags Trusts article and main elements Accurate when present Many sites do not use them
Text density Favors blocks with high text to markup ratio Works without semantic markup Fails on short or list heavy pages
Link density Rejects blocks that are mostly links Reliably removes navigation Can discard reference sections
Cross page comparison Removes content repeated site wide Very accurate Needs multiple pages from the site
Site specific rules Hand written selectors per site Highest precision Breaks on redesign, does not scale
Model based Classifies blocks by learned features Generalizes across sites Slower, higher cost per page

Text Density in Practice:

The most widely used heuristic rests on a simple observation. Navigation elements pack many links into little text. Article paragraphs contain long runs of text with few links.

Scoring each block by its ratio of text characters to markup characters, then penalizing blocks whose text is mostly inside links, separates the two reliably on a large share of pages. Readability, the algorithm behind Firefox Reader View and many extraction libraries, is built on this principle.

It fails in predictable ways. Pages that are genuinely lists, such as product category pages or search results, look exactly like navigation by these measures and get stripped entirely. Short pages provide too little signal. Pages with heavy inline linking, including reference material, can score as boilerplate despite being the main content.

Extraction with a Fallback Chain:

from bs4 import BeautifulSoup
from readability import Document

STRIP = ["nav", "header", "footer", "aside",
         "script", "style", "form", "iframe"]

def main_content(html):
    soup = BeautifulSoup(html, "lxml")

    for tag in soup.find_all(STRIP):
        tag.decompose()

    # Prefer explicit semantic containers
    for selector in ["article", "main", "[role=main]"]:
        node = soup.select_one(selector)
        if node and len(node.get_text(strip=True)) > 200:
            return node

    # Fall back to density based extraction
    return BeautifulSoup(Document(html).summary(), "lxml")

Semantic containers are trusted first because a site that marks its article element is telling you where the content is. The length check guards against empty wrappers. Density based extraction covers everything else.

The Cost of Getting It Wrong:

Both error directions are damaging, in different ways.

Removing too much deletes real content. An over aggressive link density rule can strip a reference list, a specifications table, or the entire body of a list based page. The failure is silent, since the output still looks like a clean extraction.

Removing too little leaves navigation and footers in the output. In a retrieval system this is worse than it sounds: every document from the same site now shares large blocks of identical text, which pollutes embeddings and causes near duplicate chunks to crowd out genuine matches.

For document chunking specifically, residual boilerplate is corrosive. A chunk consisting mostly of a cookie banner is retrievable, meaningless, and occupies a slot that a real answer could have filled.

Boilerplate Removal and Token Cost:

The economic case is straightforward. A page carrying 800 words of article inside 6,000 words of surrounding furniture costs roughly eight times more to process than the article alone, and the extra tokens contribute nothing.

Across a corpus of millions of pages, boilerplate removal is frequently the single largest cost reduction available in an AI pipeline, and it improves output quality at the same time.

Bright Data’s Crawl API performs content isolation before returning results, so pages arrive as clean Markdown without a separate extraction stage.

CONFIADO POR 50,000+ CLIENTES EN TODO EL MUNDO

Ready to get started?