Feeding raw HTML into LLM context windows is one of the most expensive and inefficient mistakes in modern AI engineering.
A standard modern news or blog page easily spans 1.5MB to 4MB of raw DOM payload. When passed straight into an LLM or vector database, 90% of those tokens are spent on tracking scripts, serialized JSON-LD blobs, cookie banners, navigation menus, and inline CSS styles. This not only causes severe context bloat and escalates inference bills, but it also degrades retrieval-augmented generation (RAG) semantic search precision by polluting your vector space with boilerplate noise.
Here is how to design and build an enterprise-grade, self-hosted extraction microservice using FastAPI, Trafilatura, Readability, and an asynchronous Playwright fallback for SPA rendering. The Bottleneck: Commercial APIs vs. Fragile Scrapers
Most teams start with simple libraries like BeautifulSoup or newspaper3k. These quickly break down: Layout Brittleness: Custom CSS selectors degrade when publishers redesign their DOM or randomize class names (e.g., CSS Modules/Tailwind compiles). Client-Side Rendering (CSR): Single-page applications built on React, Next.js, or Vue return empty shells to standard HTTP clients. Commercial SaaS Overkill: Hosted extraction APIs charge upwards of $0.002 to $0.01 per page. If your pipeline indexes 200,000 URLs monthly, you are paying hundreds of dollars for what essentially amounts to an HTTP request and an AST traversal.
To balance latency, compute cost, and reliability, the optimal architecture uses a tiered extraction waterfall: Tier 1 (Fast Path): High-speed async HTTP fetch + trafilatura. Latency: ~150-300ms. Tier 2 (Heuristic Fallback): If Trafilatura fails or yields empty strings, fall back to Mozilla's readability-lxml algorithm. Tier 3 (Headless Browser Fallback): If client-side hydration or dynamic script loading is detected, trigger an isolated headless Playwright worker to render the DOM before extraction. System Architecture
This setup allows 85% of standard web content to pass through the lightweight Python tier without launching Chromium, keeping memory usage stable and infrastructure costs near zero. Implementation: Code & Core Logic
Below is the complete implementation of the dual-engine pipeline using FastAPI, Pydantic v2, and async execution.
When exposing this service in production pipelines (e.g., connecting n8n webhooks or feeding continuous Celery queues), consider these three safeguards:
Browser Resource Constraints Launching a new Chromium instance via async_playwright on every request will instantly lead to Linux OOM crashes. In a production build: Maintain a single long-lived Browser instance. Create and destroy lightweight BrowserContext sessions per request. Enforce --disable-dev-shm-usage and --no-sandbox flags inside your Docker container.
Redis Caching Layer Articles rarely mutate hourly. Adding an upstream Redis cache using a SHA-256 hash of the normalized canonical URL drastically drops response times to sub-10ms for repeated requests:
You can manually wire this FastAPI microservice into your infrastructure following the architectural design and code above.
If you want the complete, production-ready implementation out of the box—including: Pre-configured Docker Compose with Redis cache and persistent volume configurations Playwright connection-pooling to avoid concurrency memory leaks Rate-limiting middleware (slowapi) and health check probes Complete test suites with synthetic fixtures and ready-to-import n8n orchestration workflows
You can grab the complete source repository directly: Instant Access on Whop: Get the Microservice on Whop Direct Download on Gumroad: Download on Gumroad — Use promo code EARLYBIRD at checkout for 20% off.