HTML to Markdown

TLDR: HTML to Markdown conversion strips presentation markup while preserving document structure. It typically reduces page size by 90 percent or more, which is why it has become the standard input format for feeding web content to language models.

HTML to Markdown conversion rewrites a web page into a lightweight text format that keeps headings, lists, links, and emphasis while discarding styling, scripts, and layout containers. The practice existed for years as a documentation convenience. It became infrastructure when language models made token count a direct cost.

Why Conversion Matters for LLMs:

  1. Token economics: Markup consumes tokens that carry no meaning. Class names, inline styles, and nested containers are pure overhead in a prompt.
  2. Context limits: A page that exceeds the model’s context window as HTML often fits comfortably as Markdown.
  3. Signal clarity: Models attend to content more reliably when it is not buried in attributes and nesting.
  4. Structure retention: Unlike plain text extraction, Markdown keeps heading hierarchy and list grouping, which carry real meaning.

The reduction is substantial. A typical article page of 180 KB of HTML converts to roughly 8 to 12 KB of Markdown. At scale, that difference determines whether a retrieval pipeline is affordable.

What Maps Cleanly:

HTML element Markdown output
Heading levels 1 to 6 One to six hash characters
Paragraph Blank line separated text
Strong and bold Double asterisks
Emphasis and italic Single asterisks
Unordered list Hyphen prefixed lines
Ordered list Numbered lines
Anchor Bracketed text with parenthesized URL
Image Exclamation mark, alt text, URL
Inline code Single backticks
Preformatted block Triple backtick fence
Blockquote Greater than prefix

What Converts Badly:

Complex tables. Markdown tables cannot express merged cells, nested tables, or multi row headers. Converters either flatten the structure and lose relationships, or emit raw markup that defeats the purpose of converting.

Forms and interactive elements. Inputs, selects, and buttons have no Markdown equivalent and are usually dropped. If the page’s value was in the form, the conversion destroys it.

Layout carrying meaning. Multi column comparisons and side by side panels become sequential text, and the visual relationship that made them legible disappears.

Custom components. Tabs, accordions, and modals flatten into undifferentiated content, often surfacing text that was hidden in the rendered page.

Conversion in Practice:

import httpx
from markdownify import markdownify
from readability import Document

def page_to_markdown(url):
    html = httpx.get(url, timeout=20).text

    # Isolate main content before converting
    article = Document(html).summary()

    return markdownify(
        article,
        heading_style="ATX",
        strip=["script", "style", "nav", "footer", "form"]
    )

The content isolation step is what separates a usable result from a poor one. Converting a full page yields Markdown containing the entire navigation menu, cookie banner, and footer sitemap. Extracting the main content first, through boilerplate removal, produces output that is both smaller and more useful.

Conversion Order:

The sequence of operations has a larger effect on output quality than the choice of converter library.

  1. Render if needed: Apply JavaScript rendering when content is client generated, since converting an empty shell produces an empty document.
  2. Remove boilerplate: Strip navigation, footers, advertisements, and related content panels.
  3. Resolve URLs: Convert relative links to absolute, or every link in the output will be broken once it leaves the page context.
  4. Convert: Transform the cleaned fragment into Markdown.
  5. Post process: Collapse repeated blank lines and strip residual artifacts.

Markdown and Retrieval Pipelines:

Markdown has become the default intermediate format in retrieval augmented generation for a practical reason beyond size. Its heading structure provides natural boundaries for document chunking, so a document can be split at section boundaries rather than at arbitrary character counts. Chunks that align with headings retain coherent meaning, which measurably improves retrieval quality.

Bright Data’s Crawl API and Search and Extract return Markdown directly, with boilerplate already removed and links resolved, which removes the conversion and cleanup stage from the pipeline entirely.

CONFIADO POR 50,000+ CLIENTES EN TODO EL MUNDO

Ready to get started?