URL Normalization

TLDR: URL normalization converts different URL strings that point to the same resource into one canonical form, so a crawler does not fetch the same page many times under different addresses.

URL normalization, also called URL canonicalization, is the process of standardizing URLs before they enter a crawler’s queue. The same page can be reachable through dozens of URL variants that differ in case, trailing slash, parameter order, or protocol. Without normalization, each variant is treated as a distinct page, which inflates crawl volume, duplicates stored data, and can prevent a crawl from ever terminating.

Standard Normalization Rules:

1. Lowercase scheme and host: HTTP://Example.COM/Page becomes http://example.com/Page. Paths stay case sensitive because many servers treat them that way.
2. Remove default ports: http://example.com:80/page becomes http://example.com/page, and the same applies to port 443 on HTTPS.
3. Resolve dot segments: /a/b/../c becomes /a/c.
4. Decode unreserved characters: %7Euser becomes ~user, since percent encoding of unreserved characters is optional.
5. Sort query parameters: ?b=2&a=1 becomes ?a=1&b=2, making parameter order irrelevant.
6. Strip fragments: Everything after # is client side and never sent to the server, so it is removed.

Before and After:

Rows two and three start as different strings and converge on the same canonical form, which is the entire point. Row five assumes tracking parameters are stripped, which is a configurable choice rather than a universal rule.

Rules That Can Break Things:

Some normalization steps look safe and are not. Each of these changes meaning on at least some sites.

Removing trailing slashes. On most servers /path and /path/ return the same content, but some return a redirect and some return genuinely different pages. Normalizing them together can silently merge two distinct resources.

Stripping all query parameters. Safe for tracking parameters such as utm_source or fbclid, destructive for parameters that select content, such as ?id=4417 or ?page=3.

Forcing HTTPS. Usually correct on modern sites, but it produces dead URLs on hosts that never implemented TLS.

Lowercasing the path. Breaks on case sensitive servers, where /Products and /products can be two different pages or one page and one 404.

The safe default is to normalize only what the URL specification guarantees is equivalent, then add site specific rules after observing how the target actually behaves.

Implementation:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "fbclid", "gclid"}
DEFAULT_PORTS = {"http": 80, "https": 443}

def normalize(url):
    parts = urlsplit(url)

    scheme = parts.scheme.lower()
    host = parts.hostname.lower() if parts.hostname else ""

    if parts.port and parts.port != DEFAULT_PORTS.get(scheme):
        host = f"{host}:{parts.port}"

    params = [(k, v) for k, v in parse_qsl(parts.query)
              if k.lower() not in TRACKING]
    query = urlencode(sorted(params))

    path = parts.path or "/"

    return urlunsplit((scheme, host, path, query, ""))

The fragment is dropped by passing an empty string as the final component. Parameters are sorted so ordering cannot create false duplicates, and known tracking keys are removed.

Normalization and Deduplication:

Normalization is what makes crawler deduplication work. A URL frontier checks each discovered URL against the set of URLs already seen, and that check is only meaningful if both sides were normalized the same way. An inconsistent normalizer produces a frontier that keeps accepting variants of pages it has already fetched.

The effect compounds on large sites. A product page reachable through four category paths, two sort orders, and three tracking sources is twenty four distinct URLs before normalization and one after. On a catalog of 50,000 products, that is the difference between a manageable crawl and one that exhausts its budget on duplicates.

Normalization vs. Canonical Tags:

The two mechanisms solve related problems from opposite ends. URL normalization is performed by the crawler using general rules applied before fetching. A rel="canonical" tag is declared by the site inside the HTML and is only visible after fetching the page.

Canonical tags are more authoritative, since the site owner knows which URL is preferred, but they arrive too late to save the request. Production crawlers use both: normalization to avoid fetching obvious duplicates, and canonical tags to collapse duplicates that only become apparent once the page is retrieved.

Bright Data’s Crawl API normalizes discovered URLs automatically and respects canonical declarations, so repeated variants of the same page do not consume crawl budget.