Enterprise Install Error

I installed Enterprise for Mediasoup for development.
I used CF-OpenVidu-Enterprise-latest.yaml
When I log “openvidu logs -f”, I can see the below error message.
And the target group of loadbalancer repeats create/fail EC2.

Minimun number of Master Nodes = 1
Desired number of Master Nodes = 1
Maximum number of Master Nodes = 2
Minimun number of Media Nodes = 1
Desired number of Media Nodes = 1
Maximum number of Media Nodes = 2

[INFO] 2022-04-22 08:22:32,672 [pool-4-thread-1] org.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://127.0.0.1:4443

[ERROR] 2022-04-22 08:22:32,674 [pool-4-thread-1] io.openvidu.server.webhook.HttpWebhookSender - IOException posting event [mediaNodeStatusChanged] to endpoint http://127.0.0.1:4443/openvidu/replication-manager-webhook?OPENVIDU_SECRET=qwer1234: Connect to 127.0.0.1:4443 [/127.0.0.1] failed: Connection refused (Connection refused)

[ERROR] 2022-04-22 08:22:32,674 [pool-4-thread-1] io.openvidu.server.webhook.HttpWebhookSender - Error sending webhook event: IOException posting event [mediaNodeStatusChanged] to endpoint http://127.0.0.1:4443/openvidu/replication-manager-webhook?OPENVIDU_SECRET=qwer1234: Connect to 127.0.0.1:4443 [/127.0.0.1] failed: Connection refused (Connection refused)

[INFO] 2022-04-22 08:22:32,699 [I/O dispatcher 1] io.openvidu.server.pro.cdr.CDRLoggerElasticSearch - New event of type “cdr” sent to Elasticsearch: {“timestamp”:1650615752622,“id”:“media_10.10.3.83”,“environmentId”:“i-08e59f387603bce78”,“ip”:“10.10.3.83”,“uri”:“ws://10.10.3.83:8888/kurento”,“newStatus”:“launching”,“oldStatus”:null,“clusterId”:“my-openvidu-cluster-id”,“event”:“mediaNodeStatusChanged”,“elastic_type”:“cdr”,“cluster_id”:“my-openvidu-cluster-id”,“master_node_id”:“master_i-03287a87070236b35”}

[INFO] 2022-04-22 08:22:32,724 [http-nio-0.0.0.0-5443-exec-2] org.kurento.jsonrpc.client.JsonRpcClientNettyWebSocket - Connecting native client

[INFO] 2022-04-22 08:22:32,725 [http-nio-0.0.0.0-5443-exec-2] org.kurento.jsonrpc.client.JsonRpcClientNettyWebSocket - Creating new NioEventLoopGroup

[INFO] 2022-04-22 08:22:32,922 [nioEventLoopGroup-2-1] org.kurento.jsonrpc.client.JsonRpcClientNettyWebSocket - Initiating new Netty channel. Will create new handler too!

[WARN] 2022-04-22 08:22:33,032 [http-nio-0.0.0.0-5443-exec-2] org.kurento.jsonrpc.client.JsonRpcClientNettyWebSocket - Trying to close a JsonRpcClientNettyWebSocket with channel == null

[WARN] 2022-04-22 08:22:33,045 [JsonRpcClient-hearbeatExec-e1-t0] org.kurento.jsonrpc.client.JsonRpcClient - Error sending heartbeat to server. Exception: InterruptedException while trying to acquire lock

[WARN] 2022-04-22 08:22:33,046 [JsonRpcClient-hearbeatExec-e1-t0] org.kurento.jsonrpc.client.JsonRpcClient - Stopping heartbeat and closing client: failure during heartbeat mechanism

[ERROR] 2022-04-22 08:22:33,045 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.kurento.kms.KmsManager - OpenVidu Server couldn’t connect to KMS with uri ws://10.10.3.83:8888/kurento

[ERROR] 2022-04-22 08:22:33,048 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.kurento.kms.KmsManager - None of the KMSs in [ws://10.10.3.83:8888/kurento] are within reach of OpenVidu Server

java.lang.Exception

at io.openvidu.server.pro.kurento.kms.MultipleKmsManager.initializeKurentoClients(MultipleKmsManager.java:116)

at io.openvidu.server.pro.infrastructure.InfrastructureManager.initNewKms(InfrastructureManager.java:458)

at io.openvidu.server.pro.rest.InfrastructureRestControllerPro.sharedNewMediaNodeResponse(InfrastructureRestControllerPro.java:345)

at io.openvidu.server.pro.rest.InfrastructureRestControllerPro.newMediaNode(InfrastructureRestControllerPro.java:304)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:113)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:204)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:97)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.base/java.lang.Thread.run(Thread.java:829)

[ERROR] 2022-04-22 08:22:33,072 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.utils.RestUtils - KMS ws://10.10.3.83:8888/kurento could not be added to OpenVidu Server

Make sure that you are deploying Media Nodes and Master nodes in public subnets and in the same VPC

May I ask what parameters did you used apart of those related with autoscaling groups?

Hello Cruizba,

I have attached a screen shot file(PDF) that I did.
I hope its may catch what I was missing.

(Attachment EnterpriseInstallError.pdf is missing)

Attached elements sent via email can not be accesed. Can you please send a capture in the browser client? Or just send me a DM

EDIT: Don’t worry, I was able to recover the pdf by looking at RAW data from the email.

Don’t worry, I was able to recover the pdf by looking at RAW data from the email.

I think I see the problem.

Redeploy again with the same parameters, but don’t define anything at parameter: OpenVidu S3 Bucket (This parameter is giving some problems…): A bucket will be created automatically with proper configuration. I will need to fix it for the next version 2.22.0. After the cloudformation is created, you will see a new S3 bucket.

Also, make sure that Subnets are from this VPC selected.

Regards

I appreciate your kind guide.

I will do it again. Many thanks.

1 Like

After creating an additional MediaNode in the Inspector and connecting the Elastic IP address to the MediaNode, the phenomenon that the MediaNode is no longer shut down is gone. It’s ok now, it seems to be a bug when CloudFormation does yaml.

But the cluster status in OpenSearch is still yellow instead of green.
{“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“no permissions for [cluster:monitor/health] and User [name=arn:aws:iam::xxx:root, backend_roles=[], requestedTenant=null]”}],“type”:“security_exception”,“reason”:“no permissions for [cluster:monitor/health] and User [name=arn:aws:iam::xxx:root, backend_roles=[], requestedTenant=null]”},“status”:403}

After installing enterprise(also pro), I tested some live broadcasts and looks good.
And then next day when I try to access to web site and SSH, there is no response from the server.
although the status of EC2 in AWS console looks good.
Security Group is ok. I use Giga Network.
But these disorders occur frequently.
That is why I try to re-install the enterprise/pro again.
This will be a serious problem to use Openvidu.

What checks and actions should I take whenever these symptoms appear?

Hello @Wann_Lee , you are deploying now a single master node deployment. For this deployment you don’t need to deploy your own OpenSearch, it is deployed in the master node automatically without the need of creating it in AWS. Also, I recommend you to use at the moment Elasticsearch 7.10 instead of OpenSearch. I don’t know why you had those problems on deploying HA.

This will be a serious problem to use Openvidu.

This is not a commont problem. I only experienced this kind of behaviour if the region itself has problems in AWS (Which AWS usually notify). We’ve never seen this kind of behaviour you have. We have lots of customers that use OpenVidu PRO/ENTERPRISE without any problem.

Regards.

I appreciate you kind explanation. I will look into once again.