Hi, so i have this problem, where after upgrading from openvidu 3.4.1, to 3.8.0 after few hours I can’t access my VM, because of nf_conntrack: table full, dropping packet error.
After some digging, I found out that caddy is creating lots of internal connections, propably health checks, and in nf_conntrack it looks like that:
sudo awk -F’src=’ ‘{print $2}’ /proc/net/nf_conntrack | awk ‘{print $1}’ | sort | uniq -c | sort -rn | head -20
3363 127.0.0.1
829 10.10.99.15
829 10.10.99.14
67 10.10.99.133
1 10.10.30.110
and the number of connections are getting bigger, to the point where nf_conntrack table is getting full, and then I have to restart vm for it to work again. The stragest thing is, that at the moment when i was digging, there was no one, that would create room, or any connection to this instance of openvidu. I’ve also tried to debug this issue with claude, and this was its response:
“caddy’s active health-checks (interval: 5s) against 127.0.0.1:7780, 127.0.0.1:8080 and 127.0.0.1:9100, combined with the layer4 TCP-proxy hops (:443 → :5656 → :7880) introduced for the single-port architecture, open new loopback TCP connections that are never closed or returned to a reusable pool. Because both endpoints of each connection are the same live caddy process, TCP keepalive never fires (the peer always ACKs), so the connections sit ESTABLISHED indefinitely — bounded only by the kernel’s nf_conntrack_tcp_timeout_established (5 days by default) on one side, and by caddy’s RLIMIT_NOFILE on the other.”
I dont really know if it actualy is the root cause for any of this, but I’m here to provide any additional info if necessary to resolve this issue.
P.S These are the containers names, that I have running:
egress
dashboard
redis
openvidu
ingress
caddy
mongo
operator
And no, I don’t use openvidu-meet, as this is for an custom application, where I don’t need it