No more available ports

Censor_K0 · March 21, 2023, 7:51am

OpenVidu Enterprise 2.26.0 ERROR Logs:

openvidu-server_1  | [ERROR] 2023-03-21 07:00:31,889 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:31,889 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry
openvidu-server_1  | [ERROR] 2023-03-21 07:00:32,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:32,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry
openvidu-server_1  | [ERROR] 2023-03-21 07:00:32,889 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:32,889 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry
openvidu-server_1  | [ERROR] 2023-03-21 07:00:33,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:33,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry
openvidu-server_1  | [ERROR] 2023-03-21 07:00:33,889 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:33,890 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry
openvidu-server_1  | [ERROR] 2023-03-21 07:00:34,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retry error. OpenVidu Server [org.kurento.client.KurentoClient@277edb2d] could not connect to Media Node media_192.168.122.31 with IP 192.168.122.31 in 3 seconds
openvidu-server_1  | [INFO] 2023-03-21 07:00:34,389 [Timer-3] io.openvidu.server.kurento.kms.KmsManager - Retrying reconnection to Media Node media_192.168.122.31 with IP 192.168.122.31. Infinite retry

Retrieve all Media Nodes info from RESTAPI shows all nodes are connected!!!

{
    "numberOfElements":1,
    "content":[
        {
            "id":"media_192.168.122.31",
            "object":"mediaNode",
            "ip":"192.168.122.31",
            "uri":"ws://192.168.122.31:8888/mediasoup",
            "connected":true,
            "connectionTime":1679198466017,
            "mediaServer":"mediasoup",
            "load":0.28,
            "environmentId":null,
            "status":"running",
            "launchingTime":1679198466006
        }
    ]
}

Error logs from mediasoup-controller:

[Nest] 1   - 03/21/2023, 7:39:21 AM   [JsonRpc] [onRequest] Method 'create' failed, error: no more available ports [transport:udp, ip:'0.0.0.0', numAttempt:10000] [method:router.createWebRtcTransport] +22ms
[Nest] 1   - 03/21/2023, 7:39:21 AM   [JsonRpc] [onRequest] Method 'create' failed, error: no more available ports [transport:udp, ip:'0.0.0.0', numAttempt:10000] [method:router.createWebRtcTransport] +4ms

List mediasoup connected numbers:

root@xxx:~# netstat -anp | grep mediasoup|wc -l
26522

1.Deploying OpenVidu Enterprise on premises.
2.mediasoup-controller and media-node-controller both are healthy.

In OpenVidu Enterprise 2.22.0, there are no issues with ports being fully occupied. Could this be due to ports not being released by mediasoup when leaving the channel?

cruizba · March 21, 2023, 10:37am

Is your previous deployment removed? Maybe those ports are attached by some other process.

I would check it with ss -tulpn. Maybe you have something running in the background.

Censor_K0 · March 21, 2023, 11:00am

After using ss -tulpn to inspect, I found that besides systemd-resolve, sshd, and docker-proxy, all other occupied ports are taken up by mediasoup.

The system only runs two Docker instances, openvidu/mediasoup-controller:2.26.0 and openvidu/media-node-controller:2.26.0. There is nothing else.

The system is Ubuntu 20.04, openvidu has been upgraded from version 2.22.0 to 2.26.0 one by one. Two days after running, there is a port shortage. The only difference from the deployment of 2.22.0 is that OPENVIDU_PRO_ELASTICSEARCH=false has been set.

root@xxx:~# ss -tulpn
Netid       State        Recv-Q       Send-Q              Local Address:Port              Peer Address:Port      Process                                              
udp         UNCONN       0            0                      172.17.0.1:14698                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=50))        
udp         UNCONN       0            0                         0.0.0.0:14700                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=264))       
udp         UNCONN       0            0                  192.168.122.31:14724                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=47))        
udp         UNCONN       0            0                      172.17.0.1:14777                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=97))        
udp         UNCONN       0            0                         0.0.0.0:14778                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=213))       
udp         UNCONN       0            0                  192.168.122.31:14838                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=100))       
udp         UNCONN       0            0                      172.17.0.1:14897                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=240))       
udp         UNCONN       0            0                  192.168.122.31:14899                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=176))       
udp         UNCONN       0            0                      172.17.0.1:14927                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=131))       
udp         UNCONN       0            0                      172.17.0.1:14957                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=433))       
udp         UNCONN       0            0                  192.168.122.31:14958                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=341))       
udp         UNCONN       0            0                  192.168.122.31:15017                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=213))       
udp         UNCONN       0            0                         0.0.0.0:15028                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=357))       
udp         UNCONN       0            0                  192.168.122.31:15029                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=160))       
udp         UNCONN       0            0                         0.0.0.0:15074                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=350))       
udp         UNCONN       0            0                  192.168.122.31:15076                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=318))       
udp         UNCONN       0            0                  192.168.122.31:15101                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=216))       
udp         UNCONN       0            0                         0.0.0.0:15103                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=192))       
udp         UNCONN       0            0                  192.168.122.31:15112                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=122))       
udp         UNCONN       0            0                  192.168.122.31:15134                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=343))       
udp         UNCONN       0            0                      172.17.0.1:15153                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=283))       
udp         UNCONN       0            0                         0.0.0.0:15178                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=128))       
udp         UNCONN       0            0                  192.168.122.31:15199                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=156))       
udp         UNCONN       0            0                         0.0.0.0:15240                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=41))        
udp         UNCONN       0            0                  192.168.122.31:15251                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=53))        
udp         UNCONN       0            0                  192.168.122.31:15260                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=80))        
udp         UNCONN       0            0                  192.168.122.31:15266                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=268))       
udp         UNCONN       0            0                         0.0.0.0:15292                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=426))       
udp         UNCONN       0            0                  192.168.122.31:15293                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=146))       
udp         UNCONN       0            0                         0.0.0.0:15295                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=273))       
udp         UNCONN       0            0                  192.168.122.31:15301                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=228))       
udp         UNCONN       0            0                  192.168.122.31:15319                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=300))       
udp         UNCONN       0            0                      172.17.0.1:15362                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=44))        
udp         UNCONN       0            0                         0.0.0.0:15379                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=135))       
udp         UNCONN       0            0                         0.0.0.0:15487                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=208))       
udp         UNCONN       0            0                         0.0.0.0:15557                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=266))       
udp         UNCONN       0            0                      172.17.0.1:15558                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=201))       
udp         UNCONN       0            0                  192.168.122.31:15567                  0.0.0.0:*          users:(("mediasoup-worke",pid=421128,fd=294))       
udp         UNCONN       0            0                         0.0.0.0:15578                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=110))       
udp         UNCONN       0            0                         0.0.0.0:15633                  0.0.0.0:*          users:(("mediasoup-worke",pid=421095,fd=69))

root@xxx:~# ps -ef | grep mediasoup
xxx  420811  420792  7 08:05 ?        00:12:28 dist/bin/mediasoup-controller
xxx  421095  420811  7 08:09 ?        00:12:13 /mediasoup-worker --logLevel=debug --logTag=bwe --logTag=info --logTag=rtcp --logTag=rtp --logTag=score --logTag=simulcast --logTag=srtp --rtcMinPort=11000 --rtcMaxPort=20999
xxx  421128  420811  7 08:10 ?        00:12:03 /mediasoup-worker --logLevel=debug --logTag=bwe --logTag=info --logTag=rtcp --logTag=rtp --logTag=score --logTag=simulcast --logTag=srtp --rtcMinPort=11000 --rtcMaxPort=20999
root      431242  418002  0 10:54 pts/0    00:00:00 grep --color=auto mediasoup

cruizba · March 21, 2023, 1:05pm

Those range port is unusual from what I am used to see. mediasoup-controller uses ports 40000-65535 by default, did you the ports configuration somewhere in your deployment installation?

I’ve tested with one of our environments and if the user leaves the session, the connection is destroyed. Does this error appears only with 0.0.0.0 ?:

[Nest] 1   - 03/21/2023, 7:39:21 AM   [JsonRpc] [onRequest] Method 'create' failed, error: no more available ports [transport:udp, ip:'0.0.0.0', numAttempt:10000] [method:router.createWebRtcTransport]

Or it also appears with other network interfaces?

Censor_K0 · March 21, 2023, 1:34pm

Yes, we have customized the media port number range to be 11000-20999. Because we need both the external network and the internal network to be able to receive media packets. Therefore, we have configured the listening address as 0.0.0.0. We haven’t tried other network interfaces, as our system requires all network interfaces to be able to accept media packets.

cruizba · March 21, 2023, 1:43pm

Is some kind of exception happening in your server? It’s quite strange that those ports are attached if there is no connection… Are those connections appears as active in /inspector from OpenVidu PRO?

How much people connect to your server? Does this happen on high demand?

cruizba · March 21, 2023, 1:59pm

What I did to test it is:

In a clean environment with no users, I execute watch -n1 | 'ss -tlpn | grep mediasoup' in the media node.
I connect using OpenVidu Call to a session.

When new users enter into that session, you will see new ports in the output of the previous command. When users leave, those ports should be removed from the output.

It could be that you are running out of ports because of many users, or something unexpected is happening in your environment which is keeping your ports occupied.

Censor_K0 · March 22, 2023, 3:24am

We haven’t installed openvidu/openvidu-call:2.26.0, so there is no /inspector to view. However, we have noticed through the REST API’s /openvidu/api/sessions that 50% of sessions have not been released, only one connection in the session. some of which date back to yesterday and others to several days ago. Therefore, we suspect that there may be an issue with session timeout release, leading to an accumulation of too many timed-out sessions after prolonged operation.

Is there an automatic timeout release mechanism for sessions if users cannot leave the session due to various network issues? If there is an issue with the automatic timeout release mechanism for sessions, it could be the cause of the current problem.

Censor_K0 · March 24, 2023, 3:11am

@cruizba @micael.gallego @pabloFuente

In these days, we have tried to regularly clean up unclosed exceptional sessions through REST API, but we still cannot avoid the problem of insufficient ports, and the error message is as follows:

[Nest] 1   - 03/24/2023, 2:26:24 AM   [MediaObject] [constructor] (WebRtcEndpoint_7b664c5f-1015-4867-9d1d-e9f082ca5c7e) MediaObject created
[Nest] 1   - 03/24/2023, 2:26:24 AM   [MediaObject] [constructor] (PassThrough_3beeeba9-f8da-49cf-bf22-9be7b849a90f) MediaObject created
[Nest] 1   - 03/24/2023, 2:26:24 AM   [JsonRpc] [onRequest] Method 'create' failed, error: no more available ports [transport:udp, ip:'0.0.0.0', numAttempt:10000] [method:router.createWebRtcTransport] +20ms

But we have found a solution. We don’t know how OpenVidu will handle this problem though.

cruizba · March 24, 2023, 12:26pm

This is not a solution, it’s a mediasoup feature to expose only one port, but this can not be applied to OpenVidu right now.

For what you are saying, it looks like you are right about ports not being closed. Can you send me some OpenVidu Pro server logs?

I will dm you so you can send it to me.

Censor_K0 · March 28, 2023, 4:02pm

I have sent the relevant logs to you via PM.

Censor_K0 · April 6, 2023, 5:53am

@cruizba
How to make mediasoup-work only listen to 0.0.0.0 instead of other IP addresses? We found that mediasoup-work is listening on multiple addresses, and since 0.0.0.0 can receive all incoming requests to the host, why do we need to listen to all IP addresses separately? This would waste a lot of ports, such as docker’s 172.17.0.1, the LAN address 192.168.12.31, and the public IP 11.11.11.11 of the host, which do not need to be listened to again. Can OpenVidu achieve only listening to 0.0.0.0?

tcp        0      0 0.0.0.0:16475           0.0.0.0:*               LISTEN      288945/mediasoup-work
tcp        0      0 172.17.0.1:19355        0.0.0.0:*               LISTEN      288946/mediasoup-work
tcp        0      0 192.168.12.31:12379     0.0.0.0:*               LISTEN      288946/mediasoup-work
tcp        0      0 11.11.11.11:19763       0.0.0.0:*               LISTEN      288945/mediasoup-work

cruizba · April 6, 2023, 1:11pm

There is a reason for that.

The reason why OpenVidu listens on 0.0.0.0 instead of specific IP addresses is because OpenVidu is designed to be deployed on multiple types of networks. By listening on 0.0.0.0, OpenVidu ensures that it can function across any network or topology.

Considering the relay and TURN candidates, it’s important to keep in mind that TURN can be located on both the master node and the media node. Given the possible locations of TURN, listening on 0.0.0.0 is the most reliable way to make sure OpenVidu operates correctly in any network environment.

In summary, the choice to listen on 0.0.0.0 is made to guarantee OpenVidu’s adaptability and seamless functioning across various network configurations.

Censor_K0 · April 6, 2023, 1:23pm

How can I make mediasoup-work only listen on 0.0.0.0 instead of listening on other local IPs?

cruizba · April 6, 2023, 1:38pm

First, you will need to examine the mediasoup configuration in your deployment. To do this, access your media node via SSH and execute the following command as the root user:

docker inspect kms | grep -oP '"WEBRTC_LISTENIPS[^"]*=\S*"

This command will display the announced IPs and listening IPs currently in use.

For instance, you might see output like this:

"WEBRTC_LISTENIPS_0_ANNOUNCEDIP=<PUBLIC_IP>"
"WEBRTC_LISTENIPS_0_IP=0.0.0.0"
"WEBRTC_LISTENIPS_1_ANNOUNCEDIP=<PRIVATE_IP_1>"
"WEBRTC_LISTENIPS_1_IP=<PRIVATE_IP_1>"
"WEBRTC_LISTENIPS_2_ANNOUNCEDIP=<PRIVATE_IP_2>"
"WEBRTC_LISTENIPS_2_IP=<PRIVATE_IP_2>"

Although it’s not possible to remove parameters, you can override them. For example, to override the WEBRTC_LISTENIPS_0_IP , you should add the following environment variable to your master node in the /opt/openvidu/.env file:

KMS_DOCKER_ENV_WEBRTC_LISTENIPS_0_IP=1.2.3.4

Afterward, restart OpenVidu using the following command:

./openvidu restart

This action will restart your entire cluster and reconfigure mediasoup with the updated listening IP.

Topic		Replies	Views
Openvidu server cannot connect KMS after restart Issues with deployment v2	1	1064	December 21, 2020
2.14 > 2.15 Upgrade Issue Issues with deployment v2	5	508	July 10, 2020
Ping request timeouts Issues developing apps v2	9	1571	January 27, 2020
Openvidu outbound request How to implement? v2	6	380	December 20, 2021
Openvidu server problems Issues with deployment v2	3	531	April 22, 2021

No more available ports

OpenVidu Enterprise 2.26.0 ERROR Logs:

Retrieve all Media Nodes info from RESTAPI shows all nodes are connected!!!

Error logs from mediasoup-controller:

List mediasoup connected numbers:

Related topics