Hi all!
Sometimes we can not start an individual recording via the API call. In this example there were 4 participants. The logs of openVidu show that one of the recorder endpoints could not be started and in this case the whole recording will not be started and the start fails. But we can not find more information in the log files WHY the endpoint could not be started. This error is not easy to reproduce and even in the same conference after some time the recording is suddenly working again - but the error happens regularly.
In the KMS log files you can see that 4 recorder endpoints are started but I can not find a reason why one of them is stopped or fails to start.
Any idea how to find the cause of this problem?
Thanks!
openVidu log file:
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,504 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/recordings/start {session=a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d, outputMode=INDIVIDUAL}
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,513 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - New recording id (a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6) and final name (a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6)
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,513 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Starting individual (video+audio) recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6 of session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,513 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Starting single stream recorder for stream str_CAM_RPI5_con_Yp4JZLbSNv in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,555 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Starting single stream recorder for stream str_CAM_IOLn_con_PQQfJAcGr0 in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,597 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Starting single stream recorder for stream str_CAM_HtQB_con_EwMC71UMD5 in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,639 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Starting single stream recorder for stream str_CAM_Z46w_con_KSl61xzNMg in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,640 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording started event for stream str_CAM_IOLn_con_PQQfJAcGr0
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:36,681 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording started event for stream str_CAM_HtQB_con_EwMC71UMD5
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:37,120 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording started event for stream str_CAM_Z46w_con_KSl61xzNMg
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:37,164 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991857163
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:37,294 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991857294
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:37,448 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991857447
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:37,816 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowOutStateChange] -> endpoint: str_CAM_RPI5_con_Yp4JZLbSNv (publisher) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991857816
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:38,649 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15985] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991858649
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:38,649 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15980] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991858649
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:38,649 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowOutStateChange] -> endpoint: str_CAM_RPI5_con_Yp4JZLbSNv (publisher) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991858649
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:38,649 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991858649
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:39,164 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991859164
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:39,295 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991859294
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:39,448 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991859448
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,448 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991861448
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-12-03 10:37:41,681 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to start in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-12-03 10:37:41,681 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - Recording start failed for session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d: Couldn't initialize some RecorderEndpoint
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,681 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - Sealed recording metadata file at /opt/openvidu/recordings/a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6/.recording.a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6 with status [failed]
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,681 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping individual (video+audio) recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6 of session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d. Reason: null
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,681 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping single stream recorder for stream str_CAM_Z46w_con_KSl61xzNMg in recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,682 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping single stream recorder for stream str_CAM_IOLn_con_PQQfJAcGr0 in recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,682 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping single stream recorder for stream str_CAM_HtQB_con_EwMC71UMD5 in recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,683 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording stopped event for stream str_CAM_Z46w_con_KSl61xzNMg
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,683 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording stopped event for stream str_CAM_IOLn_con_PQQfJAcGr0
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,683 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Stopping single stream recorder for stream str_CAM_RPI5_con_Yp4JZLbSNv in recording a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,683 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.recording.service.SingleStreamRecordingService - Recording stopped event for stream str_CAM_HtQB_con_EwMC71UMD5
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:41,817 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowOutStateChange] -> endpoint: str_CAM_RPI5_con_Yp4JZLbSNv (publisher) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991861817
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:42,878 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991862878
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:42,902 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991862902
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:42,902 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: VIDEO | timestamp: 1606991862902
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-12-03 10:37:42,903 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] org.kurento.client.internal.client.RemoteObjectInvocationHandler - Exception while processing event 'RecordingEvent' with params '{timestampMillis=1606991862902, source=e11ed96a-e9b0-4f76-aa10-22c03331a8b2_kurento.MediaPipeline/6560dd0d-bba1-40dd-aac5-aec7d40cbfb3_kurento.RecorderEndpoint, type=Recording, tags=[], timestamp=1606991862}'
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,165 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863164
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,295 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: NOT_FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863295
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,538 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15980] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_PQQfJAcGr0_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863538
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,539 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15995] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_EwMC71UMD5_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863538
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,538 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15985] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowInStateChange] -> endpoint: con_KSl61xzNMg_str_CAM_RPI5_con_Yp4JZLbSNv (subscriber) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863538
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:43,539 [AbstractJsonRpcClientWebSocket-reqResEventExec-e2-t15988] io.openvidu.server.kurento.core.KurentoParticipantEndpointConfig - KMS event [MediaFlowOutStateChange] -> endpoint: str_CAM_RPI5_con_Yp4JZLbSNv (publisher) | state: FLOWING | pad: default | mediaType: AUDIO | timestamp: 1606991863538
e[36mopenvidu-server_1 |e[0m [ERROR] 2020-12-03 10:37:46,684 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.SingleStreamRecordingService - Error waiting for some recorder endpoint to stop in session a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:46,767 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - KMS recording file permissions successfully updated
e[36mopenvidu-server_1 |e[0m [INFO] 2020-12-03 10:37:46,772 [http-nio-0.0.0.0-5443-exec-2] io.openvidu.server.recording.service.RecordingService - Sealed recording metadata file at /opt/openvidu/recordings/a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6/.recording.a15ec615d1fd01d6a5e2fd1eb27fe89b7fa4623d-6 with status [failed]