how to detect spam content during document ingestion for RAG in LangChain4j 1.14
Last week Jonathan Sanchez and I was knee-deep in a document ingestion pipeline for an internal knowledge base tool, something our team was building so engineers could ask questions against our product documentation and internal runbooks. The RAG setup was pretty standard: pull documents from a few sources, chunk them, embed them, push to a vector store. The ingestion part felt easy at first, then we started noticing weird answers. Confident, well-structured answers that were just... wrong. We traced it back to the source documents and found chunks that were basically spam: SEO keyword stuffing from scraped web pages, boilerplate legal footers repeated across hundreds of chunks, a few documents that were just lists of product names with zero context. All of it had made it into the vector store. All of it was quietly poisoning our retrieval. So I started thinking seriously about spam detection at ingestion time. Not after the fact, not as a post-retrieval filter but before anything gets embedded. Why Ingestion Time Is the Right Moment There's a temptation to filter at retrieval time, maybe score the chunks you get back and throw away anything that looks low quality. I think that's the wrong instinct, though. By the time you're retrieving, you've already paid the cost: embedding computation, storage, index bloat and you're paying it repeatedly, every time a query runs. Catching spam before it gets embedded means your vector store stays clean, your embeddings stay meaningful, and your retrieval quality doesn't quietly degrade as more documents pile in over time. It's also just easier to reason about. A document is either good enough to ingest or it isn't. Simple gate. What Spam Actually Looks Like in RAG Pipelines Before writing any code, I spent time actually looking at the garbage that was getting through. A few patterns showed up constantly: Pure keyword lists with no sentence structure. Stuff like "buy cheap flights cheap airline tickets discount airfare best price flights online". You know it when you see it. Boilerplate repeated verbatim. Cookie consent text, privacy policy footers, "this email is intended for the recipient only" disclaimers. Individually meaningless, and they cluster together in embedding space in genuinely weird ways. Very short fragments. Under 50 tokens or so. Sometimes these are fine (a heading, a label), but usually they're noise. High link density from web scraping. A chunk that's 60% URLs and anchor text isn't a document; it's a site map. Encoding artifacts. Things like "’" instead of an apostrophe, or strings of replacement characters. Usually means the source document had an encoding issue somewhere upstream. Not great raw material for a knowledge base. Once I had a clearer picture of the problem, I could think about how to detect it programmatically inside the LangChain4j ingestion flow. How LangChain4j 1.14 Structures Ingestion The ingestion pipeline in LangChain4j 1.14 is built around the EmbeddingStoreIngestor . You load documents, optionally transform or split them, and then push them to your embedding store. The relevant interfaces are DocumentTransformer and DocumentSplitter , and the good news is you can plug custom logic in at both levels. A DocumentTransformer receives a Document and returns a Document (or throws, or returns something that signals the document should be dropped). This is exactly where spam detection belongs. You can chain multiple transformers together, which is what I ended up doing. Here's the basic shape of the ingestor setup: EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder() .documentTransformer(new SpamFilterTransformer()) .documentSplitter(DocumentSplitters.recursive(500, 50)) .embeddingModel(embeddingModel) .embeddingStore(embeddingStore) .build(); Clean and readable. The transformer runs before splitting, which matters because you want to reject whole documents, not just individual fragments. Building the SpamFilterTransformer I ended up writing a SpamFilterTransformer that chains several heuristic checks. Each check is a small, independent method that scores or flags the document. If any check fires, the transformer returns null and logs the reason. Well, actually returning null from a DocumentTransformer isn't quite right in all versions. In 1.14, the cleaner pattern is to return the document unchanged if it passes, or throw a custom unchecked exception that you catch at the ingestor level. I went with a wrapper approach in the real codebase: the transformer returns a result object, and a thin outer layer decides whether to continue but I'll show the null-return version here since it's easier to follow. public class SpamFilterTransformer implements DocumentTransformer { private static final Logger log = LoggerFactory.getLogger(SpamFilterTransformer.class); private static final int MIN_WORD_COUNT = 20; private static final double MAX_LINK_DENSITY = 0.4; private static final double MAX_KEYWORD_REPETITION_RATIO = 0.35; @Override public Document transform(Document document) { String text = document.text(); if (isTooShort(text)) { log.warn("Dropping document: too short. Source: {}", document.metadata("source")); return null; } if (hasHighLinkDensity(text)) { log.warn("Dropping document: high link density. Source: {}", document.metadata("source")); return null; } if (isKeywordStuffed(text)) { log.warn("Dropping document: keyword stuffing detected. Source: {}", document.metadata("source")); return null; } if (hasEncodingArtifacts(text)) { log.warn("Dropping document: encoding artifacts detected."); return null; } return document; } private boolean isTooShort(String text) { String[] words = text.trim().split("\\s+"); return words.length < MIN_WORD_COUNT; } private boolean hasHighLinkDensity(String text) { long linkCount = Arrays.stream(text.split("\\s+")) .filter(token -> token.startsWith("http://") || token.startsWith("https://")) .count(); double[] words = {text.split("\\s+").length}; return words[0] > 0 && (double) linkCount / words[0] > MAX_LINK_DENSITY; } private boolean isKeywordStuffed(String text) { String[] tokens = text.toLowerCase().split("\\s+"); if (tokens.length == 0) return false; Map<String, Long> freq = Arrays.stream(tokens) .collect(Collectors.groupingBy(t -> t, Collectors.counting())); long maxFreq = freq.values().stream().mapToLong(Long::longValue).max().orElse(0); return (double) maxFreq / tokens.length > MAX_KEYWORD_REPETITION_RATIO; } private boolean hasEncodingArtifacts(String text) { long replacementChars = text.chars() .filter(c -> c == '\uFFFD') .count(); return replacementChars > 5; } } The keyword stuffing check is the one I'm most proud of, honestly. The logic is simple: if any single token makes up more than 35% of all tokens in the document, something is very wrong. That catches both pure spam and those repetitive boilerplate chunks that slip past simpler length-based filters. Adding an LLM-Based Check for Edge Cases Heuristics got us maybe 80% of the way there. The other 20% was trickier: documents that passed all the structural checks but were still semantically useless. Stuff like auto-generated product description pages that used complete sentences but said absolutely nothing ("The XJ-400 is a product in the XJ series, featuring XJ-series features for XJ-series users"). You've seen these pages. For those, I added an optional LLM-based spam classifier that runs on documents which pass the heuristic filter. I didn't want to call an LLM for every single document (too slow, too expensive), so I only trigger it when...