Good morning,
we are experiencing problems when trying to connect more than a few users to a meeting, using Kurento and OpenVidu.
Starting the session and adding up to 6-7 users seems to work correctly, but any more users that try to connect are not seeing all other participants (just some of them).
If they get out and back in a different order is always the last ones to connect that have problems (and they may or may not be seen by other participants).
About our servers, we have one dockerized instance of Kurento and one of OpenVidu server that serve multiple environments: on one server we have the production and pre-production environments, while on another we have a number of environments that are created for testing purposes, starting from different branches of the repository. They are removed after testing, but sometimes there are quite a few (like 10 or 15) and that time the problem occurs quicker. If we remove all environments but one or two, it seems to work much better (note that the environments are usually just running containers, but no one is actually using them most of the time).
The problem actually occurs only on the testing server, which is a much smaller one in terms hardware capacity, besides having the containers running.
We could observe the following (on both servers):
* As users connect, CPU usage naturally increases, but doesn't seem to drain the CPU. Memory also seems to be fine
* As users connect, there is a lot of ports being open from the kurento-media process. They increase exponentially on all environments, not just the one being used for the session, reaching a few thousand ports, in listening state (between tcp and udp both IPv4 and IPv6), with just few users. We tested creating two meeting sessions on two separate environments on the same machine, reaching about 6000 open ports with ten users each.
* As users disconnect, the situation gets back to normal, and the open ports get all closed
Both servers use the same turn, which is external, but in other deployments of the same platform, which have the turn running on the same server, we can observe the same as above.
OpenVidu server logs the following errors:
ERROR:Request has timed out in Request: method:onIceCandidate params:{"endpointName":"con_BI0GW7DN03","candidate":"candidate:4193657879 1 udp 25108479 <address of the turn server> 57810 typ relay raddr 161.74.34.173 rport 58571 generation 0 ufrag 8sZQ network-id 3 network-cost 50","sdpMid":"0","sdpMLineIndex":0} request:{"jsonrpc":"2.0","method":"onIceCandidate","params":{"endpointName":"con_BI0GW7DN03","candidate":"candidate:4193657879 1 udp 25108479 <address of the turn server> 57810 typ relay raddr 161.74.34.173 rport 58571 generation 0 ufrag 8sZQ network-id 3 network-cost 50","sdpMid":"0","sdpMLineIndex":0},"id":134}
Error sending ICE candidate: {"request":"{\"jsonrpc\":\"2.0\",\"method\":\"onIceCandidate\",\"params\":{\"endpointName\":\"con_BI0GW7DN03\",\"candidate\":\"candidate:4193657879 1 udp 25108479 <address of the turn server> 57810 typ relay raddr 161.74.34.173 rport 58571 generation 0 ufrag 8sZQ network-id 3 network-cost 50\",\"sdpMid\":\"0\",\"sdpMLineIndex\":0},\"id\":134}","requestTime":1679059986480}
main-es2015.a3e0e714cb59bb96cf1f.js:1 ERROR DATA:"java.lang.NullPointerException\n\tat io.openvidu.server.kurento.core.KurentoParticipant.receiveMedia(KurentoParticipant.java:279)\n\tat io.openvidu.server.kurento.core.KurentoSessionManager.subscribe(KurentoSessionManager.java:675)\n\tat io.openvidu.server.rpc.RpcHandler.receiveVideoFrom(RpcHandler.java:390)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:142)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"
main-es2015.a3e0e714cb59bb96cf1f.js:1 Error for 'Subscriber' (str_CUS_QA4k_con_VlfzNTJ9m1) while trying to subscribe: Error: Error on receiveVideoFrom : {"code":-1,"message":"java.lang.NullPointerException:null","data":"java.lang.NullPointerException\n\tat io.openvidu.server.kurento.core.KurentoParticipant.receiveMedia(KurentoParticipant.java:279)\n\tat io.openvidu.server.kurento.core.KurentoSessionManager.subscribe(KurentoSessionManager.java:675)\n\tat io.openvidu.server.rpc.RpcHandler.receiveVideoFrom(RpcHandler.java:390)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:142)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","requestTime":1679060003679}
Error on receiveVideoFrom : {"code":-1,"message":"java.lang.NullPointerException:null","data":"java.lang.NullPointerException\n\tat io.openvidu.server.kurento.core.KurentoParticipant.receiveMedia(KurentoParticipant.java:279)\n\tat io.openvidu.server.kurento.core.KurentoSessionManager.subscribe(KurentoSessionManager.java:675)\n\tat io.openvidu.server.rpc.RpcHandler.receiveVideoFrom(RpcHandler.java:390)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:142)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","requestTime":1679060003679}
at main-es2015.a3e0e714cb59bb96cf1f.js:1:3321748
at main-es2015.a3e0e714cb59bb96cf1f.js:1:3986189
at Object.p [as callback] (main-es2015.a3e0e714cb59bb96cf1f.js:1:1542037)
at w (main-es2015.a3e0e714cb59bb96cf1f.js:1:1543406)
at u.decode (main-es2015.a3e0e714cb59bb96cf1f.js:1:1543221)
at WebSocket.f (main-es2015.a3e0e714cb59bb96cf1f.js:1:1539284)
at l.invokeTask (polyfills-es2015.fc910774638739ab503b.js:1:7920)
at Object.onInvokeTask (main-es2015.a3e0e714cb59bb96cf1f.js:1:3453679)
at l.invokeTask (polyfills-es2015.fc910774638739ab503b.js:1:7841)
at i.runTask (polyfills-es2015.fc910774638739ab503b.js:1:3329)
ERROR:java.lang.NullPointerException:null in Request: method:receiveVideoFrom params:{"sender":"str_CUS_QA4k_con_VlfzNTJ9m1","sdpOffer":"v=0\r\no=- 3113115685717345373 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:RuPQ\r\na=ice-pwd:9bO4ZpEH9Tey2ucKsmUsZEaQ\r\na=ice-options:trickle\r\na=fingerprint:sha-256 74:41:2F:10:48:82:A4:F6:C7:39:4B:AA:96:2F:D2:14:71:9B:BF:C4:08:83:31:F1:75:1B:8D:18:7C:31:47:3A\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:126 telephone-event/8000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 35 36 37 38 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 45 46 47 48 112 113 114 115 116 49\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:RuPQ\r\na=ice-pwd:9bO4ZpEH9Tey2ucKsmUsZEaQ\r\na=ice-options:trickle\r\na=fingerprint:sha-256 74:41:2F:10:48:82:A4:F6:C7:39:4B:AA:96:2F:D2:14:71:9B:BF:C4:08:83:31:F1:75:1B:8D:18:7C:31:47:3A\r\na=setup:actpass\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:13 urn:3gpp:video-orientation\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 VP9/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 profile-id=2\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:35 VP9/90000\r\na=rtcp-fb:35 goog-remb\r\na=rtcp-fb:35 transport-cc\r\na=rtcp-fb:35 ccm fir\r\na=rtcp-fb:35 nack\r\na=rtcp-fb:35 nack pli\r\na=fmtp:35 profile-id=1\r\na=rtpmap:36 rtx/90000\r\na=fmtp:36 apt=35\r\na=rtpmap:37 VP9/90000\r\na=rtcp-fb:37 goog-remb\r\na=rtcp-fb:37 transport-cc\r\na=rtcp-fb:37 ccm fir\r\na=rtcp-fb:37 nack\r\na=rtcp-fb:37 nack pli\r\na=fmtp:37 profile-id=3\r\na=rtpmap:38 rtx/90000\r\na=fmtp:38 apt=37\r\na=rtpmap:102 H264/90000\r\na=rtcp-fb:102 goog-remb\r\na=rtcp-fb:102 transport-cc\r\na=rtcp-fb:102 ccm fir\r\na=rtcp-fb:102 nack\r\na=rtcp-fb:102 nack pli\r\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\r\na=rtpmap:103 rtx/90000\r\na=fmtp:103 apt=102\r\na=rtpmap:104 H264/90000\r\na=rtcp-fb:104 goog-remb\r\na=rtcp-fb:104 transport-cc\r\na=rtcp-fb:104 ccm fir\r\na=rtcp-fb:104 nack\r\na=rtcp-fb:104 nack pli\r\na=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\r\na=rtpmap:105 rtx/90000\r\na=fmtp:105 apt=104\r\na=rtpmap:106 H264/90000\r\na=rtcp-fb:106 goog-remb\r\na=rtcp-fb:106 transport-cc\r\na=rtcp-fb:106 ccm fir\r\na=rtcp-fb:106 nack\r\na=rtcp-fb:106 nack pli\r\na=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=rtpmap:107 rtx/90000\r\na=fmtp:107 apt=106\r\na=rtpmap:108 H264/90000\r\na=rtcp-fb:108 goog-remb\r\na=rtcp-fb:108 transport-cc\r\na=rtcp-fb:108 ccm fir\r\na=rtcp-fb:108 nack\r\na=rtcp-fb:108 nack pli\r\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\r\na=rtpmap:109 rtx/90000\r\na=fmtp:109 apt=108\r\na=rtpmap:127 H264/90000\r\na=rtcp-fb:127 goog-remb\r\na=rtcp-fb:127 transport-cc\r\na=rtcp-fb:127 ccm fir\r\na=rtcp-fb:127 nack\r\na=rtcp-fb:127 nack pli\r\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f\r\na=rtpmap:125 rtx/90000\r\na=fmtp:125 apt=127\r\na=rtpmap:39 H264/90000\r\na=rtcp-fb:39 goog-remb\r\na=rtcp-fb:39 transport-cc\r\na=rtcp-fb:39 ccm fir\r\na=rtcp-fb:39 nack\r\na=rtcp-fb:39 nack pli\r\na=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f\r\na=rtpmap:40 rtx/90000\r\na=fmtp:40 apt=39\r\na=rtpmap:41 H264/90000\r\na=rtcp-fb:41 goog-remb\r\na=rtcp-fb:41 transport-cc\r\na=rtcp-fb:41 ccm fir\r\na=rtcp-fb:41 nack\r\na=rtcp-fb:41 nack pli\r\na=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f\r\na=rtpmap:42 rtx/90000\r\na=fmtp:42 apt=41\r\na=rtpmap:43 H264/90000\r\na=rtcp-fb:43 goog-remb\r\na=rtcp-fb:43 transport-cc\r\na=rtcp-fb:43 ccm fir\r\na=rtcp-fb:43 nack\r\na=rtcp-fb:43 nack pli\r\na=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f\r\na=rtpmap:44 rtx/90000\r\na=fmtp:44 apt=43\r\na=rtpmap:45 AV1/90000\r\na=rtcp-fb:45 goog-remb\r\na=rtcp-fb:45 transport-cc\r\na=rtcp-fb:45 ccm fir\r\na=rtcp-fb:45 nack\r\na=rtcp-fb:45 nack pli\r\na=rtpmap:46 rtx/90000\r\na=fmtp:46 apt=45\r\na=rtpmap:47 AV1/90000\r\na=rtcp-fb:47 goog-remb\r\na=rtcp-fb:47 transport-cc\r\na=rtcp-fb:47 ccm fir\r\na=rtcp-fb:47 nack\r\na=rtcp-fb:47 nack pli\r\na=fmtp:47 profile=1\r\na=rtpmap:48 rtx/90000\r\na=fmtp:48 apt=47\r\na=rtpmap:112 H264/90000\r\na=rtcp-fb:112 goog-remb\r\na=rtcp-fb:112 transport-cc\r\na=rtcp-fb:112 ccm fir\r\na=rtcp-fb:112 nack\r\na=rtcp-fb:112 nack pli\r\na=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f\r\na=rtpmap:113 rtx/90000\r\na=fmtp:113 apt=112\r\na=rtpmap:114 red/90000\r\na=rtpmap:115 rtx/90000\r\na=fmtp:115 apt=114\r\na=rtpmap:116 ulpfec/90000\r\na=rtpmap:49 flexfec-03/90000\r\na=rtcp-fb:49 goog-remb\r\na=rtcp-fb:49 transport-cc\r\na=fmtp:49 repair-window=10000000\r\n"} request:undefined
This errors keep repeating while in the meeting, and may vary from one to another depending on whether the current user was the latest to enter or another one was.
Kurento does not log any errors, apart from some warnings that may be relevant, like:
2023-03-17T14:41:08,963517 1 0x00007f65fd681700 warning rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 36822, expected 36853)
2023-03-17T14:41:08,726349 1 0x00007f656f7fe700 warning kmsutils kmsutils.c:504 gap_detection_probe() <kmswebrtcendpoint532_kmsagnosticbin2-732:sink> GAP of 10 ms at PTS=0:02:08.362922344 (packet loss?); will request a new keyframe
2023-03-17T14:41:08,726658 1 0x00007f656f7fe700 warning kmsutils kmsutils.c:478 discont_detection_probe() <kmswebrtcendpoint532_kmsagnosticbin2-732:sink> DISCONTINUITY at non-keyframe; will drop until keyframe
2023-03-17T14:41:07,766864 1 0x00007f65fd681700 warning rtpjitterbuffer rtpjitterbuffer.c:785 rtp_jitter_buffer_insert() duplicate packet 36585 found
2023-03-17T14:39:03,592880 1 0x00007f656f7fe700 warning rtpsynchronizer kmsrtpsynchronizer.c:553 kms_rtp_synchronizer_process_rtp_buffer_mapped() <KmsRtpSynchronizer@0x7f6520114360> [Sorted mode] Fix PTS not increasing monotonically, SSRC: 2733565625, seq: 9284, rtp_ts: 94803623, ext_ts: 94803623, last: 0:00:03.708081492, current: 0:00:03.706622253, fixed = last: 0:00:03.708081492
2023-03-17T14:39:03,592990 1 0x00007f656f7fe700 warning kmsutils kmsutils.c:1455 kms_utils_depayloader_adjust_pts_out() <rtpvp8depay179> Fix PTS not strictly increasing, last: 0:00:03.708081492, current: 0:00:03.708081492, fixed = last + 1: 0:00:03.709081492
2023-03-17T14:39:03,139567 1 0x00007f65fd681700 warning rtpsource rtpsource.c:1176 update_receiver_stats() probation: seqnr 9248 != expected 9252
Following is a partial output of netstat -tualpn | grep kurento | sort
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 <public ip of the server>:40385 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:40602 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:43812 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:44054 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:48800 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:49228 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:54863 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 <public ip of the server>:56495 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:41218 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:44272 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:44357 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:46499 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:46826 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:52743 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:54216 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.18.0.1:55691 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.27.0.1:40383 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.27.0.1:44831 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.27.0.1:45094 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.27.0.1:46314 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 172.27.0.1:50480 0.0.0.0:* LISTEN 31555/kurento-media
....
tcp 0 0 173.5.65.1:49372 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.65.1:50977 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:42765 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:48461 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:49634 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:50924 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:52138 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:54394 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:54885 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.71.1:56572 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:43234 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:44284 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:45704 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:46299 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:46820 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:52036 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:55123 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.85.1:55180 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:41298 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:41347 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:41562 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:41679 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:43034 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:44586 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:46069 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.88.1:56648 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:41417 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:43227 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:45077 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:46396 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:46943 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:48961 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:51446 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 173.5.93.1:54465 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:42048 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:43186 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:43429 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:44244 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:51664 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:54040 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:54184 0.0.0.0:* LISTEN 31555/kurento-media
tcp 0 0 192.168.160.1:55680 0.0.0.0:* LISTEN 31555/kurento-media
tcp6 0 0 127.0.0.1:8888 127.0.0.1:54598 ESTABLISHED 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::40243 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::40348 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::42433 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::45361 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::48379 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::52697 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::54903 :::* LISTEN 31555/kurento-media
tcp6 0 0 2b01:4f9:5b:150a::55270 :::* LISTEN 31555/kurento-media
tcp6 0 0 :::8888 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::42:10ff:fee:42171 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::42:10ff:fee:44830 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::42:10ff:fee:44894 :::* LISTEN 31555/kurento-media
.. parallel to ipv4 above ..
tcp6 0 0 fe80::aaa1:59ff:f:49328 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::aaa1:59ff:f:49379 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::aaa1:59ff:f:49872 :::* LISTEN 31555/kurento-media
tcp6 0 0 fe80::aaa1:59ff:f:56537 :::* LISTEN 31555/kurento-media
udp 0 0 <public ip of the server>:42374 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:42540 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:45746 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:45926 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:47126 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:49143 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:52336 0.0.0.0:* 31555/kurento-media
udp 0 0 <public ip of the server>:55280 0.0.0.0:* 31555/kurento-media
udp 0 0 172.18.0.1:41038 0.0.0.0:* 31555/kurento-media
udp 0 0 172.18.0.1:42532 0.0.0.0:* 31555/kurento-media
.. parallel to tcp above ..
udp 0 0 173.5.93.1:51283 0.0.0.0:* 31555/kurento-media
udp 0 0 173.5.93.1:53971 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:40947 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:41023 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:46016 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:46765 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:48715 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:50784 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:55326 0.0.0.0:* 31555/kurento-media
udp 0 0 192.168.160.1:56644 0.0.0.0:* 31555/kurento-media
udp6 0 0 2b01:4f9:5b:150a::40134 :::* 31555/kurento-media
udp6 0 0 2b01:4f9:5b:150a::41128 :::* 31555/kurento-media
.. parallel to ipv4 above ..
udp6 0 0 fe80::aaa1:59ff:f:53964 :::* 31555/kurento-media
udp6 0 0 fe80::aaa1:59ff:f:54538 :::* 31555/kurento-media
udp6 0 0 fe80::aaa1:59ff:f:55416 :::* 31555/kurento-media
We could notice a slight improvement (as in the problem occurs after a little more more users connected) by cleaning up many unused docker images and containers, thus freeing some disk space, but even before there still was plenty free.
Does anyone have ideas, at least about where to start looking, whether in the application code or server configurations? Maybe bandwitdth issues?
It is a pretty worrying issue as it may probably occur in production also, given the right conditions, and we really are at a loss about how to approach it.
Many thanks in advance!