Quartz 5

Modular Monolith vs Microservices for High-Throughput Ingestion & NLP Pipelines

Properties2
descriptionWhy we avoided premature microservices complexity and chose a Hexagonal Modular Monolith for automated document scraping, OCR, and LLM extraction on single-host deployments.
tagsarchitecture, systems-design, clean-architecture, cloud-engineering, performance

3 min read

When architecting document ingestion and intelligence systems—pipelines that scrape external disclosures, extract OCR text from attached PDFs/images, run schema-validated LLM analysis, and broadcast alerts—the standard industry impulse is to reach prematurely for distributed microservices: distributed task queues (Celery/RabbitMQ/Redis), independent scraping daemons, separate OCR workers, and isolated API gateways.

Here is why we intentionally avoided distributed microservices and engineered a Hexagonal Modular Monolith for single-host production deployments.

1. The Premature Microservices Penalty

In multi-stage document pipelines, distributing components across separate network services before scaling demands it introduces massive operational friction:

[ Premature Distributed Microservices Architecture ]
   - Scraper Service  ──► [ Redis Task Queue ] ──► [ Celery OCR Workers ]
   - OCR Workers      ──► [ Redis Task Queue ] ──► [ LLM Inference Workers ]
   - Scoring Service  ──► [ RabbitMQ Queue ]   ──► [ Alert Dispatcher ]

Operational Pitfalls on Single-Host Deployments:

  1. Excessive Memory Footprint: Spawning multiple Python runtimes, Celery beat schedulers, and queue daemons easily exhausts available host RAM.
  2. Silent Stream Serialization Failures: Passing raw file pointers or large byte streams across message queues requires complex S3/blob indirection. A network timeout during LLM retry resets the buffer and drops the document silently.
  3. Observability Fragmentation: Tracing a single ingested file across four network hops requires distributed tracing infrastructure (OpenTelemetry/Jaeger), turning simple debugging into a distributed log search.

2. The Solution: In-Process Hexagonal Modular Monolith

Instead of physical network boundaries, we enforce strict in-process domain boundaries using Hexagonal / Clean Architecture (Ports & Adapters) inside a unified Python workspace.

┌────────────────────────────────────────────────────────────────────────┐
│                        Application Domain Core                         │
│     (Use Cases: IngestDocument, ParseAttachment, EvaluateSignals)      │
└───────────▲────────────────────────────────────────────────▲───────────┘
            │                                                │
 (Inbound Ports / Drivers)                        (Outbound Ports / Driven)
            │                                                │
┌───────────┴────────────────────────┐     ┌─────────────────┴──────────────────┐
│ Inbound Adapters                   │     │ Outbound Adapters                  │
│ - Async Cron Scheduler             │     │ - Relational & Analytical DB       │
│ - CLI & Ad-hoc Replay Handlers     │     │ - Multi-format OCR & PDF Extractors│
│ - REST API Webhooks                │     │ - LLM Clients & Structured Parsers │
│                                    │     │ - Alert & Notification Gateways    │
└────────────────────────────────────┘     └────────────────────────────────────┘

Why this design wins:

  1. Single-Process Efficiency: One Python runtime orchestrates async scheduling, file parsing, and database transactions with a lean memory footprint (< 400 MB).
  2. Zero Serialization Latency: Data passes between domain use cases as typed, immutable Pydantic DTOs directly in memory, avoiding JSON/binary serialization overhead across Redis.
  3. True Extraction Readiness: Because domain use cases interact only with abstract Port interfaces, any heavy stage (e.g. GPU OCR worker or batch LLM scoring) can be extracted into an independent microservice later without modifying a single line of business logic.

3. High-Throughput Scraper Optimization: Connection Pooling & Session Reuse

When scraping high-frequency web portals protected by anti-bot verification or rate-limiters, instantiating a new HTTP client per worker leads to TCP socket exhaustion and anti-bot trigger loops.

In our modular ingestion adapter:

  • We maintain a centralized Persistent Session Pool.
  • TLS fingerprints, cookie jars, and keep-alive TCP connections are preserved across recurring poll intervals.
  • This cuts request round-trip latency by over 70% and completely prevents transient 403/429 connection dropouts.

4. Architectural Invariants & Key Takeaways

  1. Monolith First, Microservices When Proven: Keep boundaries in-process until compute bottlenecks or organizational separation physically necessitate network isolation.
  2. Hexagonal Architecture Guarantees Clean Separation: Application use cases must depend strictly on inbound/outbound Port contracts—never on external database drivers, scraping packages, or specific LLM SDKs.
  3. Simplicity is the Ultimate Reliability: A modular monolith running on a well-tuned host with structured observability will consistently outperform an over-engineered, unmonitored microservice cluster.

Graph View

Backlinks