PRO: inspector is not available error 502

Anto · November 5, 2020, 2:58pm

Hello,

We are using pro version 2.15.1, we cannot access to inspector, and we got error 502. Same if we try to access to pro api for mount a media node.
My guess is that is related with elasticSearch, I send a error we got at the end of this email.

Please this issue is quite urgent.

Error starting openvidu-server_1 openvidu-server_1 openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | openvidu-server_1 | ApplicationContext. To display the conditions report re-run your application with ‘debug’ enabled.
| [ERROR] 2020-11-05 14:26:05,526 [main] org.springframework.boot.SpringApplication - Application run failed
| org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘elasticSearchConfig’: Invocation of init method failed; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;]]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at io.openvidu.server.pro.OpenViduServerPro.main(OpenViduServerPro.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:597)
| Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1897)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1867)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1624)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1596)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1563)
at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:139)
at io.openvidu.server.pro.config.ElasticSearchConfig.createIndex(ElasticSearchConfig.java:115)
at io.openvidu.server.pro.config.ElasticSearchConfig.init(ElasticSearchConfig.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
… 26 common frames omitted
Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:9200], URI [/filebeat-openvidu-2020.11.05?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
| {“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;”},“status”:400}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1611)
… 38 common frames omitted

cruizba · November 5, 2020, 4:13pm

Hello @Anto , I don’t know how important is your Elasticsearch data. There are two ways to solve this:
First option will delete previous data. Second option will try to preserve it, but elasticsearch may take some minutes to start.

First option:

Stop OpenVidu:

cd /opt/openvidu/
./openvidu stop

The next commands are executed as sudo in /opt/openvidu directory

Delete this directory in /opt/openvidu/:

rm -rf elasticsearch

Create again the folder: elasticsearch folder in /opt/openvidu

mkdir elasticsearch

Change permissions to elasticsearch folder

chown -R 1000:1000 elasticsearch

If you have 4GB or more, add this environment variable in your docker-compose.yml in the elasticsearch section:

...
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
...

It should look like this:

elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
        restart: always
        environment:
            - discovery.type=single-node
            - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
        ports:
            - 9200:9200
        volumes:
            - ./elasticsearch:/usr/share/elasticsearch/data

Restart openvidu with:

./openvidu start

Second option:
0. Stop OpenVidu:
Stop OpenVidu:

cd /opt/openvidu/
./openvidu stop

The next commands are executed as sudo in /opt/openvidu directory

If you have 4GB of RAM or more, add this environment variable in your docker-compose.yml in the elasticsearch section:

...
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
...

It should look like this:

elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
        restart: always
        environment:
            - discovery.type=single-node
            - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
        ports:
            - 9200:9200
        volumes:
            - ./elasticsearch:/usr/share/elasticsearch/data

Restart openvidu with:

./openvidu start

Elasticsearch may take some minutes to start because the problem is probably due because of a lot of data saved or low memory resources.

Be sure to check disk space and RAM. Elasticsearch run better in machines with 8 GB of RAM for example.

I hope this helps
Regards,
Carlos

Anto · November 5, 2020, 7:23pm

Hello Carlos,

thank you.

First I tried without remove the elasticsearch folder, but I was not working. So I tried to rm this folder, and now it is working.

Now I have a question: this folder will grow up again, we have to remove this data periodically? The vm has 254GB of disk (the disk was a 11%) and 8GB of Ram.

Thank you really much.
Regards
Antonello

cruizba · November 5, 2020, 10:12pm

We’ve seen some problems regarding Docker container logs and the index used to store OpenVidu Pro events. (It is called openvidu).

All the indexes in ElasticSearch follow a policy that delete files older than the value indicated in the env variable OPENVIDU_PRO_ELASTICSEARCH_MAX_DAYS_DELETE, but it is not working for openvidu index.

In the next version that is coming the next week we will fix this, openvidu documents older than OPENVIDU_PRO_ELASTICSEARCH_MAX_DAYS_DELETE will be deleted and also you will be able to configure ElasticSearch externally if you want.

Regarding docker logs issues, we will configure a max-size of 100MB for each container running. You can do a cronjob which truncate docker logs to avoid this potential problem of lack of space in disk until the next version:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Default value of OPENVIDU_PRO_ELASTICSEARCH_MAX_DAYS_DELETE will be 15 days

Regards,
Carlos

Anto · November 10, 2020, 4:38pm

Hello Carlos,

thank you a lot for your help. I really appreciate.
I have one more question: could be possible recover the data? Or at least make it consultable?

Thank you.
Regards
Antonello

Topic		Replies	Views
Inspector & Kibana not working Issues with deployment	12	1175	June 7, 2021
Openvidu pro inspector 502 bad gateway How to implement?	10	987	April 21, 2021
On Premises Pro Deployment Kibana Issues (openvidu-server_1 \| Waiting for kibana in 'http://127.0.0.1/kibana' 'URL'.) Issues with deployment	11	648	June 15, 2022
Inspector randomly not working/timing out? (OpenVidu Enterprise on AWS) Issues with deployment	4	360	December 9, 2021
No Media Node available where to initalize and Unable to Log In into Kibana Issues with deployment	6	352	September 11, 2020

PRO: inspector is not available error 502

Related Topics