Link: https://old.reddit.com/r/DataHoarder/comments/1vq0ruq/i_built_trawl_a_selfhosted_scraping_engine_that/

This is an automated archive made by the Lemmit Bot.

The original was posted on /r/datahoarder by /u/Germond_ on 2026-08-16 16:00:05+00:00.

I'm the developer behind TRAWL, a self-hosted web scraping engine built to get through JavaScript challenges, WAF walls and CAPTCHAs.

When a site returns a challenge page instead of the actual content, TRAWL detects it and moves through four tiers:

plain HTTP -> cached session -> Camoufox browser -> optional residential proxy

It currently handles Cloudflare, Akamai Bot Manager, Imperva/Incapsula, Turnstile, reCAPTCHA, hCaptcha and GeeTest.

You can use it through its native scraping API, the FlareSolverr-compatible API, or as an HTTP/HTTPS proxy. The proxy mode means existing apps can send traffic through TRAWL without needing their own browser automation.

Some practical uses are:

• Website, price, stock and content monitoring
• Monitoring protected pages with changedetection.io
• Downloading from protected sites with JDownloader
• Crawlers, search engines and data aggregation
• Feeding current web content into RAG or other data pipelines
• Automated testing of protected or JavaScript-heavy pages
• Prowlarr, Jackett and other FlareSolverr-compatible apps

TRAWL started as a faster and more capable alternative to FlareSolverr and Byparr, but it has grown into a general-purpose scraping and challenge proxy engine.

I ran all three on the same machine and network. These were the results:

Previously solved domains can return in under 500ms by reusing a cached session. You can find the full benchmark and feature comparison here: https://trawl.germondai.com/#compare

TRAWL also supports custom headers, request bodies, authenticated proxies, WebSockets, binary responses, streaming and Range requests. Everything runs locally in Docker. Redis and upstream proxies are optional, and no paid CAPTCHA-solving service is required.

There are also one-click community apps available for TrueNAS and Unraid:

• TrueNAS: https://apps.truenas.com/catalog/trawl_community/
• Unraid: https://ca.unraid.net/apps/trawl-1o4q23p06utr4h

GitHub: https://github.com/germondai/trawl

Docs: https://docs.trawl.germondai.com/

Website: https://trawl.germondai.com/

Feedback, bug reports and contributions are welcome.