Request timeout when participants more than 10

Issues developing apps
1

We are developing app with openvidu 2.15. The problem we met is when the eleventh participants joined the room, the room session crash. Error messages as below:

^[[36mopenvidu-server_1 |^[[0m org.kurento.jsonrpc.JsonRpcException: [KurentoClient] Timeout of 10000 milliseconds waiting from response to request {“id”:844,“method”:“subscribe”,“params”:{“type”:“Error”,“object”:“9d1828ae-ee94-4892-b194-6ff4e94240d5_kurento.MediaPipeline/602647a2-4122-4f09-bb78-7845b7d46fd1_kurento.WebRtcEndpoint”,“sessionId”:“e2399ab2-4ddc-45f8-92df-120692ad1ab4”},“jsonrpc”:“2.0”}
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.internalSendRequestWebSocket(AbstractJsonRpcClientWebSocket.java:399)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket$1.internalSendRequest(AbstractJsonRpcClientWebSocket.java:141)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.internal.JsonRpcRequestSenderHelper.sendRequest(JsonRpcRequestSenderHelper.java:75)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.internal.JsonRpcRequestSenderHelper.sendRequest(JsonRpcRequestSenderHelper.java:69)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.client.JsonRpcClient.sendRequest(JsonRpcClient.java:112)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.sendRequest(RomClientJsonRpcClient.java:228)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.subscribe(RomClientJsonRpcClient.java:130)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.subscribe(RomClientJsonRpcClient.java:122)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.client.RomManager.subscribe(RomManager.java:190)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.client.RemoteObject.addEventListener(RemoteObject.java:252)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.client.RemoteObjectInvocationHandler.subscribeEventListener(RemoteObjectInvocationHandler.java:219)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.client.RemoteObjectInvocationHandler.internalInvoke(RemoteObjectInvocationHandler.java:133)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.client.internal.client.DefaultInvocationHandler.invoke(DefaultInvocationHandler.java:39)
^[[36mopenvidu-server_1 |^[[0m at com.sun.proxy.$Proxy109.addErrorListener(Unknown Source)
^[[36mopenvidu-server_1 |^[[0m at io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig.addEndpointListeners(KurentoParticipantEndpointConfig.java:203)
^[[36mopenvidu-server_1 |^[[0m at io.openvidu.server.kurento.core.KurentoParticipant.receiveMediaFrom(KurentoParticipant.java:262)
^[[36mopenvidu-server_1 |^[[0m at io.openvidu.server.kurento.core.KurentoSessionManager.subscribe(KurentoSessionManager.java:533)
^[[36mopenvidu-server_1 |^[[0m at io.openvidu.server.rpc.RpcHandler.receiveVideoFrom(RpcHandler.java:360)
^[[36mopenvidu-server_1 |^[[0m at io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:130)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)
^[[36mopenvidu-server_1 |^[[0m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
^[[36mopenvidu-server_1 |^[[0m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
^[[36mopenvidu-server_1 |^[[0m at java.lang.Thread.run(Thread.java:748)
^[[36mopenvidu-server_1 |^[[0m Caused by: java.util.concurrent.TimeoutException: null
^[[36mopenvidu-server_1 |^[[0m at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411)
^[[36mopenvidu-server_1 |^[[0m at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84)
^[[36mopenvidu-server_1 |^[[0m at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.internalSendRequestWebSocket(AbstractJsonRpcClientWebSocket.java:377)
^[[36mopenvidu-server_1 |^[[0m … 23 common frames omitted

Is there limitation to number of participants in one room?

2

The limitation is in both your server hardware and browsers. If your server CPU is 100%, you won’t be able to fit more streams. If you are displaying lots of videos in a browser tab, it can crash.
Keep an eye on your server’s CPU (of course running a more powerful server will allow more users in your sessions) and check that your application can handle all the videos and elements i’s supposed to display.

3

Our server has 12 X 2.5Ghz CPU, and only 20% CPU usage during the test. It should not caused by CPU of server.

4

During the test, we only enable video and audio stream of one participants. So the loading in browser is not high too. However, when we increase the participants to 12 or 13, all connections crash. All participants disconnect from the session.

5

So check your CPU usage in your server. What is the size of your machine?

6

When there are 10 participants in the session (1 with video+ audio, others no video and no audio), CPU usage as below:

D3TK2RNJZAT~MUET88)0CDJ
D3TK2RNJZAT~MUET88)0CDJ955×441 40.8 KB

And then 5 more participants join to session without video and audio, CPU usage as below:
NIX_24@ZCLN6NILU82EDR
NIX_24@ZCLN6NILU82EDR955×441 42.9 KB

Then after few seconds, all participants disconnect to the session, error message in server log as below:

openvidu-server_1 | [ERROR] 2020-08-26 08:44:50,976 [main] io.openvidu.server.config.OpenviduConfig - .env file not found at /.env
openvidu-server_1 | [ERROR] 2020-08-26 08:44:52,041 [main] io.openvidu.server.config.OpenviduConfig - .env file not found at /.env
openvidu-server_1 | [ERROR] 2020-08-26 08:50:35,532 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcNotificationService - No rpc session found for private id 9n830t7u905tcri3ulfcehrmnu, unable to send notification participantEvicted: {“connectionId”:“con_ZZc6O2f5GK”,“reason”:“networkDisconnect”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:35,533 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcNotificationService - No rpc session found for private id 9n830t7u905tcri3ulfcehrmnu, unable to send notification participantEvicted: {“connectionId”:“con_ZZc6O2f5GK”,“reason”:“networkDisconnect”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:35,534 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcNotificationService - No session found for private id 9n830t7u905tcri3ulfcehrmnu, unable to cleanup
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,706 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:165,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_FzHkuQA4YD”,“candidate”:“candidate:3350505436 1 udp 2122260223 192.168.29.102 55593 typ host generation 0 ufrag k4+n network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,708 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:166,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_FzHkuQA4YD”,“candidate”:“candidate:3350505436 1 udp 2122260223 192.168.29.102 55594 typ host generation 0 ufrag k4+n network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,708 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:167,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_FzHkuQA4YD”,“candidate”:“candidate:188323919 1 udp 1686052607 113.89.213.136 37634 typ srflx raddr 192.168.29.102 rport 55594 generation 0 ufrag k4+n network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,708 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:168,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_FzHkuQA4YD”,“candidate”:“candidate:188323919 1 udp 1686052607 113.89.213.136 37633 typ srflx raddr 192.168.29.102 rport 55593 generation 0 ufrag k4+n network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,709 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:169,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_Bz68HUgNKr”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag wZgt network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,709 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:170,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_Bz68HUgNKr”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag wZgt network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,709 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:171,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_A1anJac68y”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag rhdo network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,709 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:172,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_A1anJac68y”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag rhdo network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,711 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:173,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_RGccOSOhi1”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag jAdT network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,711 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:174,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_RGccOSOhi1”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag jAdT network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,712 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:175,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_TvKX203TWg”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag BEBo network-id 1”,“sdpMid”:“0”,“sdpMLineIndex”:0},“jsonrpc”:“2.0”}
openvidu-server_1 | [ERROR] 2020-08-26 08:50:51,712 [SessionHandler-umc3ck4g99qsucfa357jojr0r1-e36-t0] org.kurento.jsonrpc.internal.JsonRpcHandlerManager - Exception while processing request {“id”:176,“method”:“onIceCandidate”,“params”:{“endpointName”:“con_TvKX203TWg”,“candidate”:“candidate:2302106412 1 tcp 1518280447 192.168.29.102 9 typ host tcptype active generation 0 ufrag BEBo network-id 1”,“sdpMid”:“1”,“sdpMLineIndex”:1},“jsonrpc”:“2.0”}

7

You can take a look to the number of publishers and subscribers (= number of streams) supported by different servers according to their number of cores: https://docs.openvidu.io/en/stable/troubleshooting/#9-which-is-the-current-status-of-openvidu-regarding-performance-scalability-and-fault-tolerance

If your sessions are N:N, then the number of streams per session is NxN. A single 15 user session would have 15*15= 225 streams. If some users are not publishing, then adjustments in these number should be made.

8

I did some test as below:
I was able to setup three 10 X 10 sessions in the same server which only consume 50% percent CPU and all sessions run smoothly.
However I can’t setup single 15 X15 session. the connection will crash.
For total of streams, 3X10 X10 scenario has more streams than 15 X 15 one。So I think it should not be issue of total number of streams per server.
It seems has some constrains to total number of streams in a single session.

9

That seems strange. I did a quick test with two computers and 12 participants and it seems to work in our deployment at demos.openvidu.io.

Can you do the load test using our demos server? The basic auth credentials are:
user: OPENVIDUAPP
pass: MY_SECRET

10

Hi， cruizba, what is the server domain and port? https://demos.openvidu.io:4443?

11

We found that there is a setting of max sessions

LT3O79`PWJ$0H}47$%{1I
LT3O79`PWJ$0H}47$%{1I811×433 28.3 KB

https://ove.readthedocs.io/en/stable/ove-apps/packages/ove-app-webrtc/README.html

Do you think it will cause the limitation? How can I change the setting?

12

Hi @Micky_Huang1, this application is not developed by us, it uses OpenVidu, but it manages an internal state per session. This is probably managed by the application you’re using.

Yes. maybe this maxSessions parameter is related with the problem you’re facing with the number of participants.

13

Hi, @cruizba I have some new findings for the case:

  1. I try to join the participants one by one to the session and everything is OK after I join 15th participants.
  2. When I start to join the 16th participant, there are errors in the server log , which told me that the first joined participant is disconnect.
    H6D8(B9@YX{WC(8TI{54W4
    H6D8(B9@YX{WC(8TI{54W41366×736 67.1 KB
  3. Also there are errors in the console log of the first joined participant that also tell that the websocket is not connected
    @FG(}P4SHU`SE9P3@HE%J(X
    @FG(}P4SHU`SE9P3@HE%J(X1012×536 43.6 KB
  4. Then the user was kicked out of the session
    The disconnected websocket is between browser and openvidu server. Do you think it is caused by the nginx connection limitations?
14

Hi @Micky_Huang1, that’s probably a problem in the browser side. Are you rendering 15 videos in all the browsers? This consumes a lot of CPU in each browsers and that could be the problem of the limitation.