2026-05-22 · 6 min read
Most scrapers default to "proxy everything always, just in case." That bills like it works — paid proxy pools charge per GB. If half your scrape targets serve you fine over the open internet, you're burning money.
Every scrape target carries a proxy_mode field with three values: direct (never use a paid proxy), auto (try direct first, fail over on HTTP block signals), always (go straight to the pool).
Block-signal detection: HTTP 403/429/503, Cloudflare interstitial markers (__cf_chl_, cf-mitigated headers), empty bodies on hosts where empty is unusual, specific captcha provider domains. A match flips the per-host state to using_proxy for a 24-hour window.
Paid fallback ships with Decodo integration — 10 SOCKS5/HTTP gateway endpoints, sticky EU rotation, basic-auth credentials per project. The pool is the same Redis-backed structure that consumes ProtonVPN tunnels.
Scraper-monitor admin page tracks per-host: total requests in last 24h, direct hits vs proxy hits, bytes transferred, median latency direct vs proxy, per-state event log.
Start free trial → · All features →