# Unexpected restart **URL:** \ **Category:** Issues with deployment\ **Tags:** v2\ **Created:** [January 5, 2022, 10:20pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689 "2022-01-05T22:20:08Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 5, 2022, 10:20pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/1 "2022-01-05T22:20:08Z")
Hello, We are experiencing unexpected restart of openvidu container from time to time while presumably there are enough resource on this physical server. It starts with java complaining about insufficient memory and then **java.lang.OutOfMemoryError** and crash. [log](https://drive.google.com/file/d/1I0Qv5pgXu5QXRzhdr8FWoxbfGpQCKYNx/view?usp=sharing) As specified in the log file, it is an Intel E5-2697 v2 with 125 GB of ram and Ubuntu 20.04. We already changed java heap size (-Xms/-Xmx) in openvidu java command. Is there any tweak or tuning we can make in the server to prevent this or at least less frequent? ---
**Author:** ![pabloFuente](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/pablofuente/32/2536_2.png) [@pabloFuente](https://openvidu.discourse.group/u/pabloFuente)\ **Post date:** [January 6, 2022, 6:04pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/2 "2022-01-06T18:04:39Z")
Seeing your log, I’m afraid that the JVM is simply telling that it cannot allocate 16KB of new RAM. How many GBs of memory does your server have? ---
**Author:** ![micael.gallego](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/micael.gallego/32/2_2.png) [@micael.gallego](https://openvidu.discourse.group/u/micael.gallego)\ **Post date:** [January 7, 2022, 3:37pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/3 "2022-01-07T15:37:44Z")
Please make sure you configure the memory available to the OpenVidu Server java process using the JAVA\_OPTIONS property in the .env file: > Currently there are a limitation and Java process is not using all the memory available in the machine. Best regards ---
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 10, 2022, 10:41am UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/4 "2022-01-10T10:41:55Z")
We run openvidu server java with these **-Xms2048m -Xmx4096m** args but no tangible changes felt and server still crashes while at least half of RAM is available. **top** output has been logged [here](https://drive.google.com/file/d/1-sgPYg4WHnh7gAComU--1aXtjUbnkAdy/view?usp=sharing). You can see it yourself in the top log that when crash took place somewhere around 09:51 to 09:53 no resource shortage observed. ---
**Author:** ![cruizba](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/cruizba/32/18_2.png) [@cruizba](https://openvidu.discourse.group/u/cruizba)\ **Post date:** [January 10, 2022, 5:04pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/5 "2022-01-10T17:04:00Z")
I’m pretty sure you’re not setting up correctly those parameters. Can you show me the output of: ```auto ps aux | grep openvidu-server ``` Regards ---
**Author:** ![cruizba](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/cruizba/32/18_2.png) [@cruizba](https://openvidu.discourse.group/u/cruizba)\ **Post date:** [January 10, 2022, 5:10pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/6 "2022-01-10T17:10:43Z")
Also you don’t need to use those specific values, adapt to your environment. With 125GB try to use… ` -Xms10240m -Xmx20480m`. Play whith those parameters, until you reach your desired peak performance. ---
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 15, 2022, 5:56am UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/7 "2022-01-15T05:56:02Z")
> [@cruizba](#): > > `ps aux | grep openvidu-server` @cruizba Sorry for delay, > root@server6:~# ps aux | grep openvidu-server > root 309623 0.0 0.0 8164 2460 pts/0 S+ 05:51 0:00 grep --color=auto openvidu-server > root 4122463 19.6 1.9 31501944 2513976 ? Sl Jan14 123:27 java -Xms2048m -Xmx4096m -jar openvidu-server.jar It explicitly says in the log that in order to prevent jvm from crashing one of the thing is decreasing heap size! ---
**Author:** ![cruizba](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/cruizba/32/18_2.png) [@cruizba](https://openvidu.discourse.group/u/cruizba)\ **Post date:** [January 17, 2022, 7:58pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/8 "2022-01-17T19:58:27Z")
Sure, but did you try to increment the heap size in the way I’ve recommended? Where is this log you’re talking about? ---
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 17, 2022, 8:52pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/9 "2022-01-17T20:52:05Z")
Yes, We did change java options as you said. > root@server14:~# ps aux |grep openvidu-server > root 753927 0.0 0.0 5196 2412 pts/0 S+ 20:42 0:00 grep --color=auto openvidu-server > root 3126609 36.3 5.8 49346368 7771104 ? Sl 13:25 158:41 java -Xms10240m -Xmx20480m -jar openvidu-server.jar As the time of this very post, server has crashed and restarted 7 hours ago > root@server14:~# docker ps |grep openvidu-server > b8786352f3f6 openvidu/openvidu-server:2.20.0 “/usr/local/bin/entr…” 2 weeks ago Up 7 hours openvidu\_node\_1\_openvidu-server\_1 And in the openvidu container, a file has been created [hs\_err\_pid16.log](https://drive.google.com/file/d/1XI_9apa01HrywQP1atyC-DR4xdJ7CjGv/view?usp=sharing) consisting of all crash related lines. This file is located beside openvidu-server.jar in /. > root@server14:/# ls -lsh > total 126M > 4.0K drwxr-xr-x 1 root root 4.0K Jan 7 19:08 bin > 4.0K drwxr-xr-x 2 root root 4.0K Apr 24 2018 boot > 0 drwxr-xr-x 5 root root 340 Jan 17 13:25 dev > 8.0K drwxr-xr-x 1 root root 4.0K Jan 5 12:32 etc > 4.0K drwxr-xr-x 2 root root 4.0K Apr 24 2018 home > 36M -rw-r–r-- 1 root root 36M Jan 7 12:47 hs\_err\_pid15.log > 36M -rw-r–r-- 1 root root 36M Jan 17 13:25 hs\_err\_pid16.log > 4.0K drwxr-xr-x 1 root root 4.0K May 23 2017 lib > 4.0K drwxr-xr-x 2 root root 4.0K Aug 27 07:18 lib64 > 4.0K drwxr-xr-x 2 root root 4.0K Aug 27 07:16 media > 4.0K drwxr-xr-x 2 root root 4.0K Aug 27 07:16 mnt > 45M -rw-rw-r-- 1 root root 45M Sep 22 16:16 openvidu-server.jar > 4.0K drwxr-xr-x 1 root root 4.0K Jan 3 12:37 opt > 0 dr-xr-xr-x 943 root root 0 Jan 17 13:25 proc > 4.0K drwx------ 1 root root 4.0K Jan 5 12:29 root > 4.0K drwxr-xr-x 1 root root 4.0K Jan 3 12:37 run > 4.0K drwxr-xr-x 2 root root 4.0K Aug 27 07:18 sbin > 4.0K drwxr-xr-x 2 root root 4.0K Aug 27 07:16 srv > 0 dr-xr-xr-x 13 root root 0 Jan 15 01:12 sys > 16K drwxrwxrwt 1 root root 12K Jan 17 13:25 tmp > 8.0K drwxr-xr-x 1 root root 4.0K Aug 27 07:16 usr > 8.0K drwxr-xr-x 1 root root 4.0K Aug 27 07:18 var ---
**Author:** ![cruizba](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/cruizba/32/18_2.png) [@cruizba](https://openvidu.discourse.group/u/cruizba)\ **Post date:** [January 17, 2022, 9:38pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/10 "2022-01-17T21:38:00Z")
There’s a lot of `AbstractJsonRpcClientWebSocket` blocked threads. Did your KMS container restarted also?. Do you have peaks of demand or it just happens randomly? ---
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 18, 2022, 9:07am UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/11 "2022-01-18T09:07:56Z")
No, KMS did not restart while Openvidu did sporadically without any meaningful relation to load on server. Actually there is 3 other server with same hardware and specifications which are all suffering this very same problem. ---
**Author:** ![cruizba](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/cruizba/32/18_2.png) [@cruizba](https://openvidu.discourse.group/u/cruizba)\ **Post date:** [January 18, 2022, 7:41pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/12 "2022-01-18T19:41:17Z")
Hello @ali_ebrahimi We have not seen such behaviour never, even in our demos infrastructure which actually have a lot of demand. This look like a memory leak due to locked threads, but it is seems related with your application logic. In order to help with your problem, we need to replicate that error and more information about the problem. I would like you for these requirements: 1. Are you using OpenVidu CE or PRO? 2. Do you see any kind of progression when this happen? I mean, the process takes more and more RAM progresively during the period it works correctly? 3. Would it be possible to know how many sessions you have when the error is produced? 4. Can you provide a project which uses OpenVidu to replicate that exact leak? Best regards, Carlos. ---
**Author:** ![ali\_ebrahimi](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/ali_ebrahimi/32/1695_2.png) [@ali\_ebrahimi](https://openvidu.discourse.group/u/ali_ebrahimi)\ **Post date:** [January 25, 2022, 5:54pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/13 "2022-01-25T17:54:28Z")
Hello @cruizba 1- We use the community edition. 2- It seems that server consumes RAM progressively, but still way behind memory ceiling. ![memory-graph](https://global.discourse-cdn.com/free1/uploads/openvidu/original/2X/5/5e826875ab479c4a4391cc167085e0405f807f44.png) 3- Between 25 to 30 sessions, each one had 5 to 15 publishers. ![number-of-sessions](https://global.discourse-cdn.com/free1/uploads/openvidu/original/2X/3/33a78515a890d2459ff5c89aa07a959b36992b33.png) 4-We are ready to provide you with the **ssh credentials** of the mentioned server(s) suffering from the issue. ps: complete resource log which above diagrams are based on can be accessed [here](https://drive.google.com/file/d/1Pbvy9Rf-jcY_V-HiP2GDQ6KDQxVoQnEb/view?usp=sharing) . server restarted in between 10:25:25 to 10:25:27. The second restart at 10:25:54 was intentional as the result of our monitoring system. ---
**Author:** ![micael.gallego](https://yyz2.discourse-cdn.com/free1/user_avatar/openvidu.discourse.group/micael.gallego/32/2_2.png) [@micael.gallego](https://openvidu.discourse.group/u/micael.gallego)\ **Post date:** [January 26, 2022, 6:01pm UTC](https://openvidu.discourse.group/t/unexpected-restart/3689/14 "2022-01-26T18:01:38Z")
It is clear a memory leak is present in OpenVidu. As we never seen this error before, we suspect your application is using OpenVidu in some way we never tested before. Can you provide a minimal version of your app so we can reproduce the issue ourselves? Regards