Hi,
We’re facing a weird situation, and I’m wondering if someone can help troubleshooting it.
We have an Openvidu Pro deployment on AWS, with 1 Media Node (t2.small) and 1 Openvidu Server (t3.large). The idea was to scale up/down by adding more media nodes as needed.
On our test sessions with only 8 participants, only 6 participants of them could get in, and 2 were left out as the Openvidu Server CPU utilization, went up to 94%, but the Media Node stayed at around 8% CPU utilization.
We were assuming the load should be all on the Media node, not on the Openvidu Server, so we don’t understand why that’s happening.
This happened with the latest version 2.16.
On an old deployment, with Openvidu 2.15 (not pro), on a single T2.large server, the server was able to handle many more connections.
Is this a known issue with 2.16, or if not, what could cause the Openvidu server to take most of the load without it getting to the media server?
Any hints?
Thanks!
Pedro
FYI: This are our settings:
Hello Pedro,
IMHO the OpenVidu PRO Server instance is too small because of OpenVidu Server is living with ElasticSearch and Kibana (with OpenVidu CE , the ELK aren’t available). This is the main reason why the instance should be bigger.
Another thing to bear in mind is that the sessions can’t be splitted by multiple Media Nodes (it’s in our road map).
If you create a single session and add participant into it, this participant will try to connect to the session in the existing media node. Once athis session is initialized inside a Media Node, it will remain there until it is closed which this means that Media Nodes may become saturated if sessions continue to be created and load keeps growing. Check limitations of OpenVidu scalability.
This is the main reason why you can’t connect more than 6 users in a session. As your Media Mode instance is a quite bit tiny, it will be overload in a blink of an eye.
I will recommend read the following documentation OpenVidu Docs
Scalability features will be released in the following releases. Stay tuned.
Hi CSantosM,
Thanks for the quick reply! I think you are right, we just started taking things from the Openvidu Server (we removed recordings, which were being sent to S3 anyway) and it’s now a bit better, at least that server CPU utilization isn’t going that high anymore, but we still can’t get any more video sessions into the server.
Perhaps we can also take away the ElastiSearch and Kibana to a different instancce. We’ll try with other instance types to see if we can get better results.
Thanks!
Pedro
1 Like
Hi CSantos,
Just as an update, we upgraded the servers to two C5.2xlarge instances, and now the load is much better, never goes above 5% on either server. (but these servers are too expensive, this was just a test).
However, we’re still facing another issue: We’re getting a limit of 6 simultaneous video sessions, where on old server we can go beyond it without a problem. Are you familiar with any configuration which could restrict the number of participants to 6?
Thanks!
Best regards,
Pedro
OpenVidu doesn’t have any sessions restrictions configuration.
Check the openvidu server logs to figure it out what is happening please.
1 Like
If you are using COMPOSED recording take into account that that type of recording consumes a lot of computational resources of the master node.
We are releasing just now 2.17 version. In that version, COMPOSED recording is performed on media nodes, so your master node should have less CPU consumption.
Hi CSantos,
how many people joined a session for simultaneous session?
what is score limit on nginx.conf on docker?
You can check the nginx config file by executing:
docker exec openvidu_nginx_1 cat /etc/nginx/conf.d/default.conf
docker exec openvidu_nginx_1 cat /etc/nginx/nginx.conf
This is a 2.17.0 /etc/nginx/nginx.conf configuration:
AFAIK the unique limitation is 10240 concurrent nginx workers. But not “virtual” limits are setup. 10240 seems razonable for a default value. You can check what this parameter represents here: https://nginx.org/en/docs/ngx_core_module.html#worker_connections
This parameter can be changed in /opt/openvidu/.env in property WORKER_CONNECTIONS.
