Poor performance on enterprise version

[Jean_Robert_Alves]

Hello, we have an openvidu enterprise license and we cant get it to a minimum acceptable performance.

Even in a call with only two devices (my mac + my smartphone) it lags.

We followed your guides and deployed it with docker on 3 Google cloud engine instances. 1 master and 2 media nodes.
we are using the latest version as we deployed it this week.

our instances on GCP size is e2-standard-4 (4vcpu 16gb RAM) and SSD 100gb.
Default network config with public ips for each machine.
Maximum outbound network bandwidth: 8Gbps
VM to Public IP: 7Gbps

Yesterday we had a call trying to use it with 50 persons, but it was impossible. Many of them keep disconnecting, many of them tried to enable camera but it just shows a black screen, many freezing, etc.

We saw that our master node was at 80% cpu and media node only 40%, and the biggest use was for elasticsearch/kibana, so i disabled it today.

Today we had a test with 20 person and it was also very bad. Even with a very low cpu use now, we had many cameras freezing and many black screens.
In chrome console i can see this errors:
main.fa54572f3a51d9ba.js:1 [SessionComponent] ERROR: NO_STREAM_PLAYING_EVENT StreamManager of Stream str_CAM_HDmP_con_WaVLDTxpge (Subscriber) did not trigger “streamPlaying” event in 4000 ms

I tried to open all the server ports on firewall to see if it was blocking something, but we couldnt see any improvement.

Our team give up and i tried to make a new call just with two devices (my mac and smartphone) and even in this 2 person call i have a very bad performance.

I just cant see any problem on the hardware level.
As you can see we have resources.
Below you can see our general hardware metrics on this test with 2 devices only.
The remote camera was always freezing in both devices.

We just cant seem to get it to work properly and are getting a very bad exprience on this product.
Can you look at it and give us some insights?

[CSantosM]

Hi @Jean_Robert_Alves,

I understand that you’re experiencing performance issues with your OpenVidu implementation on Google Cloud Platform, and it seems that you’ve followed some recommended configuration guidelines. However, it’s important to note that the processing capability and performance of video conferences in web browsers can be influenced by various factors.

One of those factors is the rendering and processing of videos in the browser. The more video elements the browser displays, the lower its own performance becomes. Decoding and rendering real-time video is a highly CPU and GPU-intensive task, and since resources are not infinite, there comes a point where the browser lacks sufficient resources, making it impossible to render and process the videos. This results in a less smooth viewing experience and can cause videos to appear choppy or frozen.

Here are some alternatives you can implement to address this issue:

1. Avoid rendering videos for all participants in your application.
2. Set a limit on the number of participants per session, which translates to a limit on the number of video elements that need to be rendered.
3. Reduce the frame rate and resolution of the rendered videos, which can alleviate processing load and enhance the overall performance of the video conference.

[Jean_Robert_Alves]

Hmm, but theres no way that this is the normal performance of openvidu… as I said, with 20 person many of them keep disconnecting ramdonly, and even in a test with only 2 deviced it freezes the image…
I understand that theres no magic for a very large session, but we are looking to trade google meet and zoom for our openvidu instalation, but our team make those calls with no problem on these other platforms, and i need help to understand why our openvidu instalation isnt with the same quality even in much smaller calls.

We are using the default webapp from openvidu, and in the original post you can see the infrsstructure metrics. Can you see any problem on that or maybe have some tips i could try change?

[Jean_Robert_Alves]

about those 3 tips… do you have examples on how to limit number of displayed/rendered cameras?

lets say we have calls with 100 users but want to limit video rendering for only 10.

[Jean_Robert_Alves]

just to give a followup and give information for newcomers like me, we are testing with this option on our server .env:
OPENVIDU_WEBRTC_SIMULCAST=true
(Simulcast - OpenVidu Docs)

and it seems to give better results already in my test with mac + smartphone.
Will have to wait for a bigger test to see if this solved our problem 100%.

[cruizba]

I can understand a 50 people call having problems, even 20 if the same tab is rendering all the videos. But with 2 people?

Can you show me your port rules from GCE? Are your machines located far frome the browser?
Does GCE have special instances for better network performance?

[CSantosM]

I understand your perspective, and indeed, there’s no magic solution to handle massive sessions where all participants are publishing their audio and video through a web browser.

If this is a feature essential for your business, enabling SIMULCAST in mediasoup can be very beneficial as it will reduce the resource demand on the browser.

Our team is actively working to provide support for massive sessions without these kinds of issues in the medium term. However, in the meantime, you have the flexibility to manage the layout logic of your application and decide how many participant videos to display, allowing the browser to handle it efficiently.

If you’re using OpenVidu Components, I recommend checking out our tutorial that demonstrates how to customize the layout behavior.
https://docs.openvidu.io/en/stable/components/openvidu-custom-layout/

[Jean_Robert_Alves]

We tried a call with 10 user today, all of them with camera and audio enabled and it was almost perfect.

No one was freezing and the quality was good.
Cluster use was normal to

The improvement Simulcast option did is crazy. Maybe you should let this more visible on the deployment docs, like a Warning, because the difference on the first impression you can do to your new customers with this is huge.

Now we have just some imrovements to do in the webcapp. It seems that some users has their camera still in use after left the call.
My android phone is one of them and after leave one session, i just cant enter any other session. It says: “DEVICE ALREADY IN USE. NotReadableError: Could not start video source.”

It only works if i open on anonymous tab.

Another problem is that sometime we see like ghost users/sessions. We all leave the call but the session stays on the cluster for too long,
Anyone has tips for this problems?

Anyway, the simulcast option seems to resolve the performance issues that i had, so thankyou all.

[Jean_Robert_Alves]

Today we tested again trying to put more than 30 users in a call.
I see that after 30 it became impossible to join, even with our media node cpu use below 20%.
Some users had delay of some seconds…while their quality of audio/video was great, we can see that they were like 5 seconds behind in video/audio just because of their reactions and answers.
We also had some users that were in the same physical room, with camera enabled but for the other users this ones appear as camera disabled, etc.

In this test, our media node was with 8vcpu and 32gb RAM and used only 20% of it, so i think it must be a problem to solve on the client side application.

Do you have any tips on what to do or how to debug it?
Maybe any way to make our mobile users focus on some streams of the main user who are speaking and stop others less active, if the large number of streams could be the problem here. I saw that with 30 users we had like 27 subscribers for each user.

Tested right now again and it crashed already with like 20 tabs on my Mac M2

we are using the default openvidu webapp so 30 users should work fine, am I right?

[cruizba]

Hi @Jean_Robert_Alves

Sorry for the late response

OpenVidu as it is in version 2.29.0 offers scale at the level of number of sessions. But not at the actual number of participants (we are researching to improve this bottleneck). Usually a unique session start having problems with ~20/30 users at the same time, because all the videos are being rendered at the same time.

The number of participants inside a session is bottle necked in the browser side.

So, the unique solution I see to your problem, is maybe reduce the quality of the media/framerate of the web cameras being publishing, or not publish the webcam of those users which have the camera disabled.

Thanks for the suggestion on Simulcast enabled by default, we will consider it to have it as default.