Hello! I have 2 services that are only being accessed locally but require HTTPS so I am using Caddy & DuckDNS w/ DNS-01 to reverse proxy them. In DuckDNS, both subdomains are pointing at the local IP address of Caddy, and my Caddyfile has:

service1.duckdns.org:443 {
        tls {
                dns duckdns {
                        api_token API-TOKEN
                }
        }
        reverse_proxy 192.168.0.51:8080
	}

service2.duckdns.org:443 {
        tls {
                dns duckdns {
                        api_token API-TOKEN
                }
        }
        reverse_proxy 192.168.0.20:8080
	}

The issue is that I can access https://service1.duckdns.org/ (within my home network), but I can't access https://service2.duckdns.org/ (502 Bad Gateway). Caddy is running in a LXC in Proxmox, and Service1 is on a different VM on the same Proxmox machine, while Service2 is on a different machine on my network. I don't see any entries in my router's firewall logs or PiHole that indicates something is blocking it. I enabled logging in Caddy for both services and I am only able to see activity for Service1, so I guess that means something is blocking Service2 from reaching Caddy entirely?

I'm able to access service2 directly from the IP address but I'm a novice in networking so I'm trying to understand and learn what could be causing this. Any help would be appreciated!

Edit: service2 is ActualBudget and I had configured it to use the certs using the instructions here (https://actualbudget.org/docs/config/https/) when I didn't need to (since Caddy handles all of this already.. i think)

I'd say there's likely something wrong with your service. Maybe it refuses serving on a domain it's not configured for? Or there's something wrong with the headers or something like that. Or the internal network. Anyway, the 502 error is sent by the Caddy server. That means everything from your browser to the Caddy server should be fine. You can reach it. There's something wrong with the service or the parts in-between the reverse proxy and the application.

Take out the entry for service1 first to simplify the problem.

But from what I can see it looks right. Does service 2 have a self signed ssl certificate? If yes you'll need a TLS setting in that caddy block to ignore validation.

Also do an nslookup on the service 2 host name. Make sure it's going to the right IP

Have you tried NGINX proxy manager? It really simplifies the whole process.

Seems weird that you're not seeing any logs for service2 in caddy, especially if it's caddy serving the 502. I'd expect at least a message from caddy complaining about being unable to reach the upstream service in that case.

Verify that the 502 is coming from caddy and not potentially some other gateway that's part of service2.

tcpdump -nlpiany port 8080

Will show if caddy actually tries to connect the back end.

OK, I noticed I had service1.duckdns.org --> Caddy IP configured in my local DNS for pihole, but didn't have an entry for service2. I added service2.duckdns.org --> Caddy IP. Not exactly sure how that works but now I can see errors in the logs for it in Caddy, and it's still sending me a 502.

I'm not really sure where to start troubleshooting based on the log entry:

{"level":"error","ts":1791273870.8226678,"logger":"http.log.access.log1","msg":"handled request","request":
{"remote_ip":"192.168.0.183","remote_port":"53434","client_ip":"192.168.0.183","proto":"HTTP/2.0","method":"GET","host":"service2.duckdns.org",
"uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"],
"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],
"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Dnt":["1"],"Priority":["u=0, i"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Gpc":["1"]},
"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"service2.duckdns.org"}},"bytes_read":0,"user_id":"","duration":0.001651465,"size":0,"status":502,
"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

I don't think it's self-signed.. I used Caddy to generate the certs and copied them over into the config for the service. The service2 that I'm unable to access is ActualBudget and I followed their instructions from their docs (https://actualbudget.org/docs/config/https)

Yea, no, that log entry doesn't really shed any light on the issue.

However, your comment regarding having configured ActualBudget to serve the certificate might be hinting at what's wrong.

Since your chain is client > caddy > actual you should be following the guide by actualbudget regarding operating behind a reverse proxy, so actual should not be serving the certificate.

You'll likely want/need to configure actualBudget to trust your proxy as well https://actualbudget.org/docs/config/#trustedproxies

This is the right approach, I’ve had several services require extra bits in the caddy config such as:

reverse_proxy 192.168.50.6:8006 { transport http { tls_insecure_skip_verify } }

I would do a search for “your service & caddy”, someone probably has ran into the same issue.

Missing some settings then

https://actualbudget.org/docs/config/reverse-proxies/

FWIW, here is my ActualBudget Caddy config.

budget.example.com {
    reverse_proxy actual-server:5006 {
        header_down -Access-Control-Allow-Origin
        header_down Cross-Origin-Opener-Policy same-origin
        header_down Cross-Origin-Embedder-Policy credentialless
    }
}

I’ve never used ActualBudget, but their docs have a slightly different caddy example config:

budget.example.org {
    encode gzip zstd
    reverse_proxy actual_server:5006
}```

Yep, just sharing what worked for me

Lmao sorry for the confusion, I wasn’t paying attention and thought you were OP, wasn’t trying to correct you!

Thanks for the explanation! I removed the entries for the certs in Actual and now it seems to be working. Solved a problem and learned something new!

Sweet, just keep in mind that the connection between caddy and actual is just a plain HTTP one.
Since your connection is in your local network that's fine, but if you were to change that then be mindful that you are placing an implicit trust on the networking between caddy and actual.