Web Content Extraction Quality for RAG Pipelines

Most RAG projects that fail in production fail for a reason that has nothing to do with retrieval logic or model capability. They fail because noisy, structurally incoherent web content enters the pipeline before any of the downstream systems ever get a chance to work. The standard mental model of a RAG pipeline runs six steps: load data, chunk it, embed it, store it, retrieve it, generate from it. Engineering attention clusters almost entirely on steps two through six, where the tooling is mature, the documentation is thorough, and the failure modes are well understood. Step one, getting clean content off the web in the first place, is where most teams burn weeks of engineering time before they write a single line of actual RAG logic, and it is the step least likely to get named when something goes wrong.

Published research and production post-mortems describe a consistent set of RAG failure modes: stale data, bad chunks, the wrong embedding model for the query style, missing reranking, hallucinated citations, chunk boundary artifacts. The first two on that list trace directly back to how the content was pulled off the web in the first place, not to anything that happened after. When an answer comes back wrong, teams point at the model, or they point at retrieval recall and start tuning similarity thresholds. Extraction is invisible in the failure signal for a structural reason: by the time a bad answer reaches a user, the noisy input that caused it is long gone, buried several transformations upstream of the output anyone actually inspects. That attribution error is predictable given how the pipeline is built, and naming it early matters because everything that follows in this piece is a correction to that error, not an argument for any particular product.

What a raw HTML page delivers to your embedding model

A typical raw HTML page is mostly boilerplate. Analysis puts the figure at 60 to 80 percent of a page's raw HTML: navigation bars, cookie banners, ad copy, footer links, and script tags that carry no relationship to the content a RAG pipeline actually needs. When that boilerplate gets embedded alongside the real content, it pollutes the vector space with navigation text, privacy policy language, and social media profile links sitting right next to the material a user actually wants answered.

The consequence appears at query time. If navigation menus get embedded along with the article body, a user who asks "how do I navigate the API" can pull up literal nav menus instead of the documentation the question was actually about. The retriever is doing its job correctly. It is returning text that is semantically close to the query. Navigate" sits near both a nav bar and a how-to guide in vector space, and the embedding model has no way to know which one the pipeline meant to index. Cookie consent banners add vectors about privacy policy to a knowledge base that has nothing to do with privacy. Footer links add vectors about social media profiles. Neither contamination is recoverable at retrieval time, because nothing downstream knows it is there to filter out.

Structural incoherence adds to the content that boilerplate already contaminates. Dynamic JavaScript, inconsistent HTML across pages on the same site, and outright adversarial obfuscation, hidden links placed through CSS, transparent tracking images, degrade what any parser can reliably pull from a page. The scale of what gets lost to formats parsers can't handle is documented directly: MMORE, a pipeline built by researchers at a set of research universities (arXiv 2509.11937, September 2025), found that fewer than one percent of publicly available documents, including PDFs, slide decks, spreadsheets, and audiovisual content, are represented in the corpora used to train popular ML systems, because those documents stay locked behind brittle, heterogeneous formats that frustrate automated parsing at scale. That figure describes documents that never make it into a corpus. Boilerplate describes documents that make it in but drag unrelated noise along with them. Both failures happen before a single embedding gets computed.

How bad extraction degrades each downstream stage in sequence

Extraction noise does not stay contained at the point where it enters the pipeline. It propagates through chunking, through embedding, through retrieval, and into generation, compounding at every step along the way.

Chunking sits between extraction and retrieval, and if the chunks are bad, the right answer can exist somewhere in the database while the retriever simply cannot find it. If you apply naive character splitting to raw HTML, it breaks sentences across tag boundaries, so chunks are semantically incomplete before they ever reach the embedding model. This is where chunk boundary artifacts come from: an answer spans two chunks, neither chunk alone carries enough context to answer the question, the retriever dutifully returns both, and the generator has no way to stitch them back into a coherent response. Dense technical documents do better with smaller, overlapping chunks. Policy documents do better with semantic chunking that respects section boundaries. Both strategies need the input text to already be clean and structurally coherent. Neither strategy can fix text that arrived broken.

Embedding degradation follows the same logic. An embedding model produces a vector for whatever text it is given, and boilerplate text does not get zero-weighted just because it is useless. It occupies representation space that should belong to actual content. A separate, named failure mode, using an embedding model mismatched to the query style, becomes far harder to diagnose and fix when the training signal feeding the whole system is already contaminated with navigation text and cookie language.

Retrieval degradation compounds the contamination that embedding already introduced into the vector space. Vector similarity measures proximity, not correctness, so a retriever can hand back text that is highly relevant to the query's phrasing and still wrong, because it's outdated or pulled from the wrong context. Stale data ranks among the most commonly named production failure modes: documents that never get re-indexed cause the model to cite information that's no longer true. That failure belongs to extraction infrastructure, not to the retrieval algorithm sitting on top of it. Reranking with cross-encoders and fusing results across dense and sparse retrieval through methods like Reciprocal Rank Fusion both improve precision meaningfully, but neither can recover information that was never cleanly extracted to begin with. They can only rank what exists.

Generation absorbs everything that came before it, so it makes the damage visible. Hallucinated citations, where the model fabricates source references that do not exist, are a named failure mode, and they erode user trust faster than almost anything else a RAG system can do wrong. The standard fix, forcing retrieval-only answers and returning real source URLs alongside them, works only if the pipeline has reliable, traceable source content to point back to. Generation is where the cascade becomes visible to a user, but it is rarely where the cascade started. Each stage amplifies whatever quality the stage before it delivered. Clean extraction does not guarantee a good RAG system on its own, but noisy extraction guarantees a bad one.

Why fixing retrieval or prompting misses the problem

The instinct to fix a RAG quality problem at the retrieval layer or the prompting layer makes sense on its face, but it is structurally mislocated. Both layers operate on content that extraction has already shaped by the time either one sees it.

Better prompts and larger context windows can change how a model reasons over the content it retrieves. They do nothing to change what that content actually contains. Research testing 18 frontier models found that performance degrades at every length increment tested, not just as a model approaches its hard limit, and models advertised with one to two million token context windows showed severe degradation already at 100,000 tokens. A separate paper, "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval," found coding performance on HumanEval dropping by as much as 50 percent at 30,000 tokens for a model built with a 128,000-token window. Feeding a model more context, or noisier context, does not buy more reasoning headroom. It can actively degrade the reasoning the model would otherwise be capable of.

Retrieval tuning and better prompting are both real improvements, and worth doing. Neither one raises the ceiling that extraction quality sets beneath them. The strongest version of the counterargument says retrieval improvements are cheaper to build than extraction improvements, and that's often true on a per-sprint basis. Smarter extraction reduces the volume of wasted tokens flowing through every later stage, and that saving compounds at scale across every query a system handles. Most engineering teams never see that saving because they have never actually measured what fraction of their ingested content is boilerplate to begin with.

Research published in 2026 gives this a formal shape. One framing defines the extraction task precisely: given a webpage's raw HTML and a natural language user query, an extraction model pulls the content related to that query out of the HTML, conditioned on the query the same way retrieval is, rather than applied as a one-time boilerplate-removal pass uniform across every page. Retrieval and extraction are the same problem viewed from two different points in the pipeline, and treating them as separate engineering concerns lets the retrieval-layer fix fall short.

What clean extraction for RAG requires, technically

Clean extraction for production RAG is a set of distinct, composable capabilities that together guarantee the content reaching the embedding model is faithful to the source, structurally coherent, and current.

Rendering pages the way a browser does

A large share of the public web sits behind Cloudflare, cookie consent overlays, or JavaScript that has to execute before any real content appears in the page. If you call requests.get(url) against a significant share of production targets, you get back a challenge page, a redirect loop, or an empty response. Cloudflare alone sits in front of a substantial portion of all websites, and its TLS fingerprinting reliably flags and blocks the Python requests library at the network level, before the request ever reaches your application code. A request made through that library looks nothing like a real browser to a modern anti-bot system, so it gets flagged. If an initial fetch comes back thin, single-page applications and React-based sites need a rendering pipeline that kicks in automatically, not a fixed rule applied to every URL regardless of what it actually needs.

Isolating the content that matters, before it reaches the embedder

Navigation, ads, cookie banners, footers, and sidebars need to be stripped before content reaches the embedding model, not filtered out afterward once the damage to the vector space is already done. Pulling specific fields out of a page, product names, prices, article titles, publication dates, API parameters, as structured metadata alongside the main content gives the retrieval layer more to work with than plain text alone ever can.

Markdown as the structural contract

Clean Markdown preserves headers, lists, and code blocks without carrying the surrounding noise that HTML tags bring with them. Chunking strategies can use headers as natural, reliable boundaries, but only if that structure survived the extraction step intact. Chunking should happen against Markdown, not against HTML, because Markdown keeps the structural cues a chunking algorithm needs and HTML simply does not preserve them in a form chunkers can use. Prepending the page title and the source URL to each chunk before it gets embedded gives the model useful context about where the content came from, and that discipline has to start at extraction time, because it cannot be reconstructed afterward from a chunk that has already lost its provenance.

Covering more than HTML text

The public web is not only HTML. MMORE handles more than fifteen file types, including tables, images, audio, and video, and processes all of them into one unified format, reporting 40 percent higher layout accuracy on scanned PDFs than Docling on its own benchmarks. A pipeline that only parses HTML text is leaving a substantial share of the available knowledge on the table before indexing even starts.

Keeping the index current

Web content changes constantly, and a pipeline indexed once and never revisited ends up serving facts that stopped being true months earlier. Tracking content changes between crawl snapshots supports a practical refresh strategy: crawl on a schedule, compare each page against its last snapshot, re-embed only what changed, and retire the vectors tied to content that no longer exists. That is an infrastructure design decision that belongs in the crawl layer itself, not a parameter to tune inside the retrieval stack.

Treating metadata as real output

Page titles, URLs, publish dates, and structured fields make a knowledge base filterable in ways plain text cannot support on its own. Structured extraction that captures this metadata enables filtering at retrieval time that a pipeline built on plain text has no way to offer, regardless of how well-tuned its similarity search is.

Where scraping tools and SERP APIs fall short

The tools most teams reach for first, traditional scraping libraries and SERP APIs, were not built to meet any of the requirements laid out above, and their shortfalls are structural rather than something a configuration change can patch.

Scrapy and BeautifulSoup both rely on static HTML parsing built around pre-defined DOM selectors, which makes them highly exposed to ordinary structural changes in a site's layout. Selector rot is a recognized maintenance burden, and it recurs every time a target site redesigns its markup, regardless of how carefully the original selectors were written. Neither library has any semantic reasoning built in, so neither can adapt to dynamically generated content without a person stepping in to rewrite the extraction rules by hand. The infrastructure overhead behind running either one in production adds up fast: browser memory leaks, proxy rotation, CAPTCHA handling, all engineering cost paid before a single line of actual RAG logic gets written. Tools like ScrapingBee return raw HTML by default, so the pipeline has to strip navigation, ads, and markup after the fact. That parsing step burns both tokens and engineering time, and a site layout change breaks it again, forcing the cost to be paid repeatedly.

SERP APIs fail against a different part of the specification. They return short metadata, titles, URLs, brief descriptions, and that's rarely enough on its own to support deep production RAG synthesis. Getting anything substantive out of a result means fetching each target URL separately, executing its JavaScript, stripping its boilerplate HTML, and extracting structured text from what's left, an entire hidden scraping layer that the SERP API itself never provides. These tools were built to serve human search behavior: ranked links and preview snippets that a person scans and clicks through. They were not built to serve AI reasoning, which needs full document content, structured output, a traceable citation trail, and some way to control the token budget being consumed.

The temptation to build the cleaning layer in-house is understandable, but teams that have taken that path report the overhead as substantial and open-ended, never finished, because the web itself keeps changing underneath whatever rules the cleaning layer was built to enforce.