Problem
A plain HTTP proxy can forward a request, but it can't get past
a Cloudflare interstitial challenge — that requires an actual
browser to execute JavaScript and produce a
cf_clearance cookie. Doing that inline, per
request, on every hit would be slow and wasteful. The proxy
needed to detect a challenge, delegate solving it to something
heavier, cache the result, and keep serving other traffic while
it waits — without the fast path (Go, low overhead per
connection) and the slow path (a real browser) becoming
tangled into one process.
Architecture: two processes, one queue
The proxy is Go: it does MITM interception by generating TLS certificates per host on the fly, round-robins across an upstream proxy pool with a per-proxy circuit breaker, rate limits by client IP, and caches cacheable responses in Redis. The solver is TypeScript: it manages a pool of headless Chrome instances with tab reuse, drives them with Puppeteer to clear the challenge, and extracts the resulting cookie.
Redis Streams was the deliberate choice over a plain list or
pub/sub: consumer groups (XREADGROUP /
XACK) mean the solver side scales to multiple
worker instances without a coordination layer, and unacked jobs
are retried rather than silently dropped if a worker dies
mid-solve.
Not solving the same challenge twice
The failure mode that mattered most here: many requests to the same domain arriving while a challenge is already being solved. Without deduplication, that's N browser launches for one outcome.
SET NX solving:{domain} — the first request to hit an unsolved domain wins the lock; the lock value is the job ID, so a stale lock can be identified rather than just trusted.
Concurrent requests for the same domain read the lock's remaining TTL and return a dynamic Retry-After instead of guessing a fixed backoff.
The solver publishes the cookie once; every waiting request picks it up on retry.
Resilience on both sides
The proxy's upstream pool runs each proxy through a circuit
breaker (CLOSED → OPEN → HALF_OPEN) so a degraded
upstream is temporarily removed from rotation instead of
eating every request's timeout budget. Failed requests retry
with exponential backoff against an alternate proxy, and a
Cloudflare challenge solved with a since-expired cookie is
retried before falling back to re-solving from scratch — the
costly path is the last resort, not the default.
On the solver side, a job that fails repeatedly doesn't retry
forever: after a configured number of attempts it moves to a
queue:dead dead-letter stream for inspection,
keeping a systematically broken domain from silently consuming
the entire browser pool.
What I'd point to
The interesting decision isn't "use a browser to solve Cloudflare" — it's keeping the fast, cheap Go proxy path completely decoupled from the slow, expensive browser-solving path, with a queue and a domain-level lock as the only contract between them. Either side can be scaled, restarted, or replaced independently because neither one calls the other directly.