Openvidu recording out of memory

OpenVidu v2 Issues with deployment
1

We deploy openvidu server on a server with 48GB RAM, however we got error “OOM(out of memory) killer” issue, and which cause the service corrupt. The out of memory error is caused by ffmpeg process during recording video. You can see in the below logs,

##header fields
Dec 5 11:52:10 cps-ms01 kernel: [12071.088015] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name

9E8FZ@G7Q})9)}LQC))~P5
9E8FZ@G7Q})9)}LQC))~P51111×580 25.8 KB

The memory usage seems grows indefinitely and at last eat up all memory resources then cause OOM ERROR. any idea about this? Thanks.

2

Do you have a way to replicate this error? We’ve never faced such issue.

Best Regards,
Carlos

3

Don’t know exactly how to replicate, When the issue happen, we are doing below things:

  1. we created 15 room sessions
  2. there are total 10 participants. They are able to join every room session, but max number of participants of each session is 5
  3. We join and left in these room sessions during the period
  4. when the session create, will start recording and OPENVIDU_RECORDING_AUTOSTOP_TIMEOUT is set to 0
  5. The event last for about 3 hours, then trigger the OOM error.
4

BTW, is it possible to limit the memory usage of recording module? Since it is started by openvidu server, can’t set the limitation in yml file

5

Hi @cruizba , I can repeat the issue in our testbed. The repeat steps as below:

0IY4_$3AA29B__ZAG70D(S
0IY4_$3AA29B__ZAG70D(S1121×659 30.2 KB

the test sever with 10 GB RAM

  1. We total create 6 room sessions and total 4 participants
  2. 1 participants stay in room 1
  3. other participants change from one room to another in 10 minute interval
  4. then after 30 minutes, the OOM issue happen.
    5 single recording process will use memory over 1 GB and keep grows
6

I found that it is very easy to get the issue in our testbed, just create one room session and only one participants in the room. The memory usage keep increase from 500M to 1.5 G in only 5 minutes. You can see the docker stats, the CPU usage very high. However, not easy to repeat the issue in production site. Do you think it is related to CPU performance?

R}M_N}16)FG}0H9%)LAR1{N
R}M_N}16)FG}0H9%)LAR1{N1219×246 15.5 KB

7

Hi, @cruizba, I think the recording module should have memory leak issue. The issue should cause by:

  1. when use composed recording, it will consume much CPU resources
  2. Recording process receive streams from KMS and then pass it to ffmpeg
  3. If the CPUs are not enough, ffmpeg not able to process the streams in time, the streaming data may be stored in Memory buffer of the machine.
  4. However the buffer seem doesn’t have limitation, and at last, it makes the machine corrupt
8

Hi @Micky_Huang1

We’ve never experienced or received any report of Out of Memory errors, but we will take a look. The COMPOSED recording use intensively the CPU. If the problem is related with ffmpeg buffer, maybe more CPUs instead of RAM can solve the issue.

I’ll try to replicate and see the problem when available.

Regards,
Carlos