High use CPU - up 100%

Issues developing apps
1

i am doing some load tests and i had some problems with CPU consumption. The kurento-media-server is consuming 100% of cpu.
I’m testing 1: N with approximately 200 subscribers. I’m using openvidu default configurations. Only change default ports(4443-8888).
The publisher is using ManyCam to load a virtiualCam on browser (Chrome or Firefox)
Server:
AMD Ryzen 5 3600 Hexa-Core
64GB ram

Here the part of KMS log:

117:16:08.243382080 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad: Upstream provided caps: audio/x-opus, rate=(int)48000, channels=(int)2, channel-mapping-family=(int)0, stream-count=(int)1, coupled-count=(int)1, streamheader=(buffer)< 4f707573486561640102000080bb0000000000, 4f707573546167731f0000004753747265616d657220656e636f64656420766f72626973636f6d6d656e740000000001 >
117:16:08.243476528 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad: Downstream wanted caps: audio/x-opus, channels=(int)[ 1, 2 ], channel-mapping-family=(int)0
117:16:08.243760842 1 0x7fa8800af230 INFO kmsutils kmsutils.c:494:kms_utils_pad_monitor_gaps:<‘’:sink_video_default> Add probe: Detect stream gaps
117:16:08.243900094 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad: Upstream provided caps: ANY
117:16:08.243916154 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad: Downstream wanted caps: video/x-vp8
117:16:08.244669801 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad: Upstream provided caps: video/x-vp8, framerate=(fraction)23/1, height=(int)360, width=(int)480, profile=(string)0
117:16:08.244697653 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad: Downstream wanted caps: video/x-vp8
117:16:08.245130226 1 0x7fa8800af230 INFO agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad: Upstream provided caps: video/x-vp8, framerate=(fraction)23/1, height=(int)360, width=(int)480, profile=(string)0

Here, the JS of publisher:

var publisher = OV.initPublisher(“publisher”,
{
audioSource: undefined, // The source of audio. If undefined default microphone
videoSource: videoSource, // The source of video. If undefined default webcam
publishAudio: true, // Whether you want to start publishing with your audio unmuted or not
publishVideo: true, // Whether you want to start publishing with your video enabled or not
resolution: ‘640x480’, // The resolution of your video
//frameRate: 30, // The frame rate of your video
insertMode: ‘APPEND’, // How the video is inserted in the target element ‘video-container’
mirror: false // Whether to mirror your local video or not
});

Is it a problem in the conversion?
Thks a lot!
Doug

2

Hello Doug,

I suppose you know the load test we do some time ago and we published here:

As you can see, KMS can consume a lot of CPU.

But maybe you experienced some bug we recently fixed about hight CPU consumption when tens of participants are disconnected at the same time:

We have published a beta version of KMS to test with. Can you please test with this KMS version?

Also, can you share with us your load testing results?

Regards

3

Hi Micael! Thks for feedback!
Oh yes, i read both topics, but as the topology is 1: N and the processing was increasing as new students entered, i thought there would be no relationship.
I will do the update and tomorrow I can perform a new test, and return here with the result.
Thanks again!

1 Like
4

Hi Macael! Sorry man, i made a mistake in checking consumption.
Im using top and forgot that it sum the percentages of all my cores. I realized this when install Htop. Excuse me again.
So, for future reference:

Processor: AMD Ryzen 5 3600 Hexa-Core (dedicated server)
Memory: 64Gb

Test 1:N: 1-230 (only one session)
Memory usage: ~3GB
Cpu usage: ~12%

As soon as possible i’m will try a massive test to see my server limit.

ps.: i’m updated the kurento suggested by j1elo and the performance stayed the same.

Thks again!
Doug

2 Likes
5

Hi Micael! New test! Now up to 500 (1:N).
I did a test with 3 sessions and another with only 1 session.
In both, the connection limit was approximately 520. After that, the api returns error 500 (GET api/sessions) and no user can connect. Without doing anything, once the number of connections drops, other users are able to connect. Perhaps, a memory limitation on the JVM? I’ll look at the logs carefully to see if I can find anything.

Processor: AMD Ryzen 5 3600 Hexa-Core (dedicated server)
Memory: 64Gb

Test 2(1:N): 1-520 (only one session)
Memory usage: ~4GB
Cpu usage: ~40%

Test 3(1:N): 3 sessions and 520 connections
Memory usage: ~4GB
Cpu usage: ~40%

Thks Micael!
Doug

6

Hi again my friend! I found the problem. There is a limitation on workers in nginx when try to access the API. Lets you can see:

[alert] 71#71: 1024 worker_connections are not enough
[alert] 71#71: *6918 1024 worker_connections are not enough while connecting to upstream

There is a way to change this config file to increase this number?
Thks!
Doug

7

Hello @doug,

As every user is connected with a websocket connection, then nginx has to manage all of them.

We don’t know how to increase the limits. Possibly it can be increased tweaking the nginx config.

Please share with us if you find it.

8

Hi Micael! Yep its simple. But the nginx.conf is inside of container image and its doesnt have a volume to map on docker-compose.yml. Im try to keep the files original as possible to not lose hard modifications when update openvidu. Is it possible take this options in a next release? Until that i will made my own modifications.
Thks again!

9

I think it is better if you create your own Nginx container based on the provided one.

If you fix the problem, we can include the fix in the next version.

Regards

10

Hi Micael! I made a copy of nginx.conf from container and create a volume in docker-compose.yml to map this file in host machine. This is more simple to edit if i need increase this number again.
The default value in file is 1024. But, nginx used as a reverse proxy consume 2 connections per time. Running like a charm.
Thks a lot!
Doug

1 Like
11

@doug Any chance you can share how you went about doing this? I need to add a redirect and have no idea how to go about doing this with docker.

12

Hi @Wesley_Swanepoel!
What do you mean with “redirect”?
To increase a number of connections to your OV media server is in my last post.

13

Hi @doug. No I required a simple redirect from Nginx from www to non-www and didn’t know how to accomplish this using docker-compose but it will be sorted in the next release so no need for your example. I had an SSL issue as raised here.
Thanks
Wes

14

Great @Wesley_Swanepoel!!