Memory leak in openvidu server

The memory usage of openvidu server increases continuously. This was verified using res field of htop command. I allocated 8GB of heap space but due to memory leak resident memory was reported as 9GB(htop command). Server logs are as below:-

[INFO] 2022-02-03 11:07:12,260 [SessionHandler-mdv78sqovkhu268rld341r485-e141780-t0] io.openvidu.server.rpc.RpcHandler - New Connection con_Jgs5tWMdFw in Session 19d10c2d2973f0bfbb04c2691e5d6211 with IP 189.203.37.106 and platform Android 29
[ERROR] 2022-02-03 11:07:12,260 [SessionHandler-mdv78sqovkhu268rld341r485-e141780-t0] io.openvidu.server.kurento.core.KurentoSessionManager - PARTICIPANT con_Jgs5tWMdFw: Error joining/creating session 19d10c2d2973f0bfbb04c2691e5d6211
io.openvidu.client.OpenViduException: There is no available Media Node where to initialize session ‘19d10c2d2973f0bfbb04c2691e5d6211’. Code: 204
at io.openvidu.server.kurento.core.KurentoSessionManager.selectMediaNode(KurentoSessionManager.java:1409)
at io.openvidu.server.kurento.core.KurentoSessionManager.joinRoom(KurentoSessionManager.java:133)
at io.openvidu.server.rpc.RpcHandler.joinRoom(RpcHandler.java:296)
at io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:119)
at org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)
at org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] 2022-02-03 11:07:12,276 [http-nio-0.0.0.0-5443-exec-96] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/sessions/23e7969605bc8237f254622505cb3770/connection {session=23e7969605bc8237f254622505cb3770}
[INFO] 2022-02-03 11:07:12,276 [http-nio-0.0.0.0-5443-exec-96] io.openvidu.server.coturn.CoturnCredentialsService - Creating COTURN user
[INFO] 2022-02-03 11:07:12,278 [http-nio-0.0.0.0-5443-exec-124] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/sessions/e14b6e4f19a9156d27732fa03876f841/connection {session=e14b6e4f19a9156d27732fa03876f841}
[INFO] 2022-02-03 11:07:12,278 [http-nio-0.0.0.0-5443-exec-124] io.openvidu.server.coturn.CoturnCredentialsService - Creating COTURN user
[INFO] 2022-02-03 11:07:12,286 [http-nio-0.0.0.0-5443-exec-46] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/sessions/b38a0182101cfdc5dfd38af6a488b202/connection {session=b38a0182101cfdc5dfd38af6a488b202}
[INFO] 2022-02-03 11:07:12,286 [http-nio-0.0.0.0-5443-exec-46] io.openvidu.server.coturn.CoturnCredentialsService - Creating COTURN user
[INFO] 2022-02-03 11:07:12,343 [http-nio-0.0.0.0-5443-exec-198] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/sessions/fe08a75f182b93b42fac9aba35ce2cf0/connection {session=fe08a75f182b93b42fac9aba35ce2cf0}
[INFO] 2022-02-03 11:07:12,343 [http-nio-0.0.0.0-5443-exec-198] io.openvidu.server.coturn.CoturnCredentialsService - Creating COTURN user
[INFO] 2022-02-03 11:07:12,573 [http-nio-0.0.0.0-5443-exec-32] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:12,574 [http-nio-0.0.0.0-5443-exec-36] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:12,577 [http-nio-0.0.0.0-5443-exec-33] io.openvidu.server.coturn.CoturnCredentialsService - Garbage collector cleaning turn log files at path /var/log/
[INFO] 2022-02-03 11:07:12,577 [http-nio-0.0.0.0-5443-exec-33] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:12,606 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: vfui957ahst860la8kma61rs49 - Status: null
[INFO] 2022-02-03 11:07:13,147 [http-nio-0.0.0.0-5443-exec-177] io.openvidu.server.config.HttpHandshakeInterceptor - New HttpSession F60582D4B02F5AA8400CCB872B967F67
[INFO] 2022-02-03 11:07:13,312 [http-nio-0.0.0.0-5443-exec-117] io.openvidu.server.config.HttpHandshakeInterceptor - New HttpSession 383079857DBA6A83DB9CA1C014E4EF7A
[INFO] 2022-02-03 11:07:13,404 [http-nio-0.0.0.0-5443-exec-188] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/sessions/fba22974e60777f77be9f59d7a894a66/connection {session=fba22974e60777f77be9f59d7a894a66}
[INFO] 2022-02-03 11:07:13,404 [http-nio-0.0.0.0-5443-exec-188] io.openvidu.server.coturn.CoturnCredentialsService - Creating COTURN user
[INFO] 2022-02-03 11:07:13,435 [http-nio-0.0.0.0-5443-exec-131] io.openvidu.server.config.HttpHandshakeInterceptor - New HttpSession F8EA81C0FDC222129AF0E83D77275982
[INFO] 2022-02-03 11:07:18,433 [http-nio-0.0.0.0-5443-exec-104] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,434 [http-nio-0.0.0.0-5443-exec-46] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-198] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-167] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-58] io.openvidu.server.config.HttpHandshakeInterceptor - New HttpSession F9CD30D95F0401F7DE9EE7A411EAD47F
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-96] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-146] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-124] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,435 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: kanqp4q4s6039dk62sub48akk - Status: null
[INFO] 2022-02-03 11:07:18,435 [http-nio-0.0.0.0-5443-exec-87] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,437 [http-nio-0.0.0.0-5443-exec-188] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,444 [http-nio-0.0.0.0-5443-exec-176] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,445 [http-nio-0.0.0.0-5443-exec-124] io.openvidu.server.rpc.RpcHandler - After connection established for WebSocket session: gshbb0smju6asv7rmufisvrl2p
[INFO] 2022-02-03 11:07:18,445 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: tgelb7csq9gpocqibso1noulq2 - Status: null
[INFO] 2022-02-03 11:07:18,445 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: r2fbrqlscl39ikh14jf73d1tq8 - Status: null
[INFO] 2022-02-03 11:07:18,445 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: h2a7es2l4nuv0c9qol17un97f7 - Status: null
[INFO] 2022-02-03 11:07:18,445 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: j3ccclgn0tpc858sdksrt09vj4 - Status: null
[INFO] 2022-02-03 11:07:18,446 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: 58gl4h9ge6cas9ru2ppni4i06u - Status: null
[INFO] 2022-02-03 11:07:18,446 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: quthh253iok3v00dnus1762048 - Status: null
[ERROR] 2022-02-03 11:07:18,446 [http-nio-0.0.0.0-5443-exec-51] io.openvidu.server.rpc.RpcHandler - Transport exception for WebSocket session: 1eg9j06s789i7rniv9f3h27la9 - Exception: null
[INFO] 2022-02-03 11:07:18,446 [http-nio-0.0.0.0-5443-exec-56] io.openvidu.server.rpc.RpcHandler - After connection established for WebSocket session: 1l7cckoke0utlbol10pqsahclr
[ERROR] 2022-02-03 11:07:18,447 [http-nio-0.0.0.0-5443-exec-51] io.openvidu.server.rpc.RpcHandler - Transport exception for WebSocket session: qco7tjn1qrcfrbg57s1pv70jad - Exception: null
[INFO] 2022-02-03 11:07:18,447 [http-nio-0.0.0.0-5443-exec-192] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,436 [http-nio-0.0.0.0-5443-exec-153] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[INFO] 2022-02-03 11:07:18,446 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: 812b920ukc3bdqr2sf707ca7un - Status: null
[INFO] 2022-02-03 11:07:18,452 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: fffpnvafkrrgm1v9ts7qvsv9hh - Status: null
[INFO] 2022-02-03 11:07:18,452 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: csq8urt43c5c5458cllppg65e5 - Status: null
[INFO] 2022-02-03 11:07:18,453 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: 4vng0ui1tqaq02504mip8ndaea - Status: null
[INFO] 2022-02-03 11:07:18,453 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: vpti9oq0mqfqdvbg2b6smc91hi - Status: null
[INFO] 2022-02-03 11:07:18,453 [jsonrpcTaskScheduler-1] io.openvidu.server.rpc.RpcHandler - After connection closed for WebSocket session: ogeplv1t2m1fqahrr2diaphpbq - Status: null
[INFO] 2022-02-03 11:07:18,452 [http-nio-0.0.0.0-5443-exec-129] io.openvidu.server.coturn.CoturnCredentialsService - COTURN user created: true
[ERROR] 2022-02-03 11:07:18,452 [http-nio-0.0.0.0-5443-exec-131] org.apache.coyote.http11.Http11NioProtocol - Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapCharBuffer.(HeapCharBuffer.java:61)
at java.base/java.nio.CharBuffer.allocate(CharBuffer.java:348)
at org.apache.tomcat.websocket.WsFrameBase.(WsFrameBase.java:96)
at org.apache.tomcat.websocket.server.WsFrameServer.(WsFrameServer.java:45)
at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.init(WsHttpUpgradeHandler.java:128)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:915)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
[INFO] 2022-02-03 11:07:19,032 [SessionHandler-1l7cckoke0utlbol10pqsahclr-e141784-t0] io.openvidu.server.core.SessionManager - Participant con_WD20SGc4kX of session fd55e04510c35e11b963f1cd6f42bb96 is a final user connecting to this session for the first time
[INFO] 2022-02-03 11:07:19,032 [SessionHandler-1l7cckoke0utlbol10pqsahclr-e141784-t0] io.openvidu.server.rpc.RpcHandler - New Connection con_WD20SGc4kX in Session fd55e04510c35e11b963f1cd6f42bb96 with IP 189.174.17.16 and platform Android 26
[ERROR] 2022-02-03 11:07:19,032 [SessionHandler-1l7cckoke0utlbol10pqsahclr-e141784-t0] io.openvidu.server.kurento.core.KurentoSessionManager - PARTICIPANT con_WD20SGc4kX: Error joining/creating session fd55e04510c35e11b963f1cd6f42bb96
io.openvidu.client.OpenViduException: There is no available Media Node where to initialize session ‘fd55e04510c35e11b963f1cd6f42bb96’. Code: 204
at io.openvidu.server.kurento.core.KurentoSessionManager.selectMediaNode(KurentoSessionManager.java:1409)
at io.openvidu.server.kurento.core.KurentoSessionManager.joinRoom(KurentoSessionManager.java:133)
at io.openvidu.server.rpc.RpcHandler.joinRoom(RpcHandler.java:296)
at io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:119)
at org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)
at org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[ERROR] 2022-02-03 11:07:19,038 [http-nio-0.0.0.0-5443-exec-48] org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:358)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
at io.openvidu.server.utils.CommandExecutor.commonExecCommand(CommandExecutor.java:82)
at io.openvidu.server.utils.CommandExecutor.execCommand(CommandExecutor.java:42)
at io.openvidu.server.coturn.BashCoturnCredentialsService.createUser(BashCoturnCredentialsService.java:83)
at io.openvidu.server.core.TokenGenerator.generateToken(TokenGenerator.java:52)
at io.openvidu.server.core.SessionManager.newToken(SessionManager.java:330)
at io.openvidu.server.rest.SessionRestController.newWebrtcConnection(SessionRestController.java:667)
at io.openvidu.server.rest.SessionRestController.initializeConnection(SessionRestController.java:243)
at jdk.internal.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:113)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
[INFO] 2022-02-03 11:07:19,038 [http-nio-0.0.0.0-5443-exec-76] io.openvidu.server.rpc.RpcHandler - After connection established for WebSocket session: t3j6e129h32gjl157907ds0fqc

Hello @darkdev , I am quite interested in this memory leak. We actually have in demos.openvidu.io an infrastructure up and running with a lot of demos applications which are used by a lot of people internationally and we can’t see such behavior. It looks like another user on this thread has the same memory leak: Unexpected restart

We think this is something related with logic application which uses OpenVidu and makes the server to have a memory leak but we are not able to reproduce the same behaviour.

Would it be possible from your side to have a way to replicate such memory leak? Can you create an example app which makes this happen and send it to us?

Regards.

Hi Cruizba,
We have already bought your ios application.
We will create sample application for both android and ios and send to you.