OpenViduPro 2.15.1: Error during single session recording

OpenVidu v2 Issues with deployment
1

We are using OpenVidu Pro 2.15.1 with 2 media nodes.

We are seeing the following error and getting HTTP 500 and HTTP 507 on the java library when trying to start recording. In OpenVidu logs, we are seeing these errors:

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-03 04:57:38,114 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to start in session ses_WXuBNv0TE4
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-03 04:57:38,114 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - Recording start failed for session ses_WXuBNv0TE4: Couldn’t initialize some RecorderEndpoint
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-03 04:57:38,123 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.utils.CustomFileManager - Couldn’t overwrite file /opt/openvidu/recordings/ses_WXuBNv0TE4/.recording.ses_WXuBNv0TE4. Error: /opt/openvidu/recordings/ses_WXuBNv0TE4/.recording.ses_WXuBNv0TE4 (No such file or directory)
e[36mopenvidu-server_1 |e[0m [INFO] 2020-09-03 04:57:38,123 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping individual (video+audio) recording ses_WXuBNv0TE4 of session ses_WXuBNv0TE4. Reason: null

We are also seeing these errors

[36mopenvidu-server_1 | [0m [ERROR] 2020-09-03 04:57:38,169 [AbstractJsonRpcClientWebSocket-reqResEventExec-e5-t5852] io.openvidu.server.utils.CustomFileManager - Couldn’t create file /opt/openvidu/recordings/ses_WXuBNv0TE4/.stream.str_CUS_NBBU_con_P6mRBybe5N. Error: /opt/openvidu/recordings/ses_WXuBNv0TE4/.stream.str_CUS_NBBU_con_P6mRBybe5N (No such file or directory)

Please let us know how we can solve this issue. As it is impacting our transaction in production environment, a quick turnaround is much appreciated.

Thanks
Vinoth

2

Hello @vinoth ,
How did you deploy OpenVidu.

  • Are you using AWS or on premises?
  • Did you follow official deployment instructions?
  • Did this error appear suddenly? Or just in a clean environment?

Regards,
Carlos

3

Hi Carlos

Thanks for the response. This is an on premise deployment and it is been running for a few months now. We are starting to see this issue more and more now. Up until yesterday, we were getting about 10 failure out of 300 recordings with a HTTP 500 error per day. But today we have seen about 6 HTTP 507 errors in the morning and about 31 HTTP 500 messages. Nothing has changed from our end. So not sure where the issue is. Please let us know if you need any other information.

Thanks
Vinoth

4

Hi Carlos,

Today from our application log we have noticed io.openvidu.java.client.OpenViduHttpException: 504 exception 15 times within an hour

2020-09-04 11:13:15,318 ERROR [stderr] (default task-5408) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:24:08,412 ERROR [stderr] (default task-5441) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:25:15,700 ERROR [stderr] (default task-5441) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:31:11,220 ERROR [stderr] (default task-5437) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:35:38,946 ERROR [stderr] (default task-5449) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:37:59,672 ERROR [stderr] (default task-5451) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:39:18,241 ERROR [stderr] (default task-5451) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:45:24,432 ERROR [stderr] (default task-5479) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:46:36,807 ERROR [stderr] (default task-5479) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:52:34,312 ERROR [stderr] (default task-5488) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:53:54,294 ERROR [stderr] (default task-5492) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:55:01,206 ERROR [stderr] (default task-5494) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 11:56:45,460 ERROR [stderr] (default task-5494) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 12:01:24,584 ERROR [stderr] (default task-5494) io.openvidu.java.client.OpenViduHttpException: 504
2020-09-04 12:10:18,814 ERROR [stderr] (default task-5516) io.openvidu.java.client.OpenViduHttpException: 504

It is impacting the transactions in production environment

Thanks
Prabu R

5

Error Log

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:07,734 [http-nio-0.0.0.0-5443-exec-9] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to start in session ses_WBIvPlbydz
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:07,735 [http-nio-0.0.0.0-5443-exec-9] io.openvidu.server.recording.service.RecordingService - Recording start failed for session ses_WBIvPlbydz: Couldn’t initialize some RecorderEndpoint
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:07,737 [http-nio-0.0.0.0-5443-exec-9] io.openvidu.server.utils.CustomFileManager - Couldn’t overwrite file /opt/openvidu/recordings/ses_WBIvPlbydz/.recording.ses_WBIvPlbydz. Error: /opt/openvidu/recordings/ses_WBIvPlbydz/.recording.ses_WBIvPlbydz (No such file or directory)

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:07,744 [AbstractJsonRpcClientWebSocket-reqResEventExec-e5-t1602] io.openvidu.server.utils.CustomFileManager - Couldn’t create file /opt/openvidu/recordings/ses_WBIvPlbydz/.stream.str_CUS_SjOn_con_HeqpILL7hW. Error: /opt/openvidu/recordings/ses_WBIvPlbydz/.stream.str_CUS_SjOn_con_HeqpILL7hW (No such file or directory)

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:12,746 [http-nio-0.0.0.0-5443-exec-9] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to stop in session ses_WBIvPlbydz

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:12,758 [Thread-428] io.openvidu.server.pro.recording.RecordingKmsHttpDownloader - Cannot create a FileOutputStream to file /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_SjOn_con_HeqpILL7hW.webm: /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_SjOn_con_HeqpILL7hW.webm (No such file or directory)
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:12,758 [Thread-428] io.openvidu.server.pro.recording.RecordingKmsHttpDownloader - /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_SjOn_con_HeqpILL7hW.webm (No such file or directory)
e[36mopenvidu-server_1 |e[0m [INFO] 2020-09-04 06:53:12,759 [Thread-429] io.openvidu.server.pro.recording.RecordingKmsHttpDownloader - File http://10.9.208.4:3000/media-node/recordings/ses_WBIvPlbydz/str_CUS_DCaj_con_OJg0qUuN7M.webm is 0 KB in size
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:12,759 [Thread-429] io.openvidu.server.pro.recording.RecordingKmsHttpDownloader - Cannot create a FileOutputStream to file /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_DCaj_con_OJg0qUuN7M.webm: /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_DCaj_con_OJg0qUuN7M.webm (No such file or directory)
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:12,759 [Thread-429] io.openvidu.server.pro.recording.RecordingKmsHttpDownloader - /opt/openvidu/recordings/ses_WBIvPlbydz/str_CUS_DCaj_con_OJg0qUuN7M.webm (No such file or directory)

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:34,903 [http-nio-0.0.0.0-5443-exec-6] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to start in session ses_WBIvPlbydz
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:34,903 [http-nio-0.0.0.0-5443-exec-6] io.openvidu.server.recording.service.RecordingService - Recording start failed for session ses_WBIvPlbydz: Couldn’t initialize some RecorderEndpoint
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:34,904 [http-nio-0.0.0.0-5443-exec-6] io.openvidu.server.utils.CustomFileManager - Couldn’t overwrite file /opt/openvidu/recordings/ses_WBIvPlbydz/.recording.ses_WBIvPlbydz. Error: /opt/openvidu/recordings/ses_WBIvPlbydz/.recording.ses_WBIvPlbydz (No such file or directory)

e[36mopenvidu-server_1 |e[0m [ERROR] 2020-09-04 06:53:34,911 [AbstractJsonRpcClientWebSocket-reqResEventExec-e5-t1617] io.openvidu.server.utils.CustomFileManager - Couldn’t create file /opt/openvidu/recordings/ses_WBIvPlbydz/.stream.str_CUS_SjOn_con_HeqpILL7hW. Error: /opt/openvidu/recordings/ses_WBIvPlbydz/.stream.str_CUS_SjOn_con_HeqpILL7hW (No such file or directory)