Speech detection stops working after refreshing page

Hi.

We are developing React web app and faced one issue witch is stable reproducible.

So in according to docs when page is loaded then new connection to openvidu is created and listener is created like:

session.on('publisherStartSpeaking', (event) => {
    console.log('User ' + event.connection.connectionId + ' start speaking');
});

When page is unloaded there is closing session in componentWillUnmount and also by “beforeunload” window listener.

So when we refresh page (via F5 for example) everything works as expected, i.e. connection is closed and the created again, BUT speech detection does not work after refresh (video and audio streaming work as expected).

I looked at console logs and see there:

Remote 'Connection' with 'connectionId' [con_KAIfjHb2qy] is now configured for receiving Streams with options
'Connection' created (remote) with 'connectionId' [con_KAIfjHb2qy]
Subscribing to con_KAIfjHb2qy
'Subscriber' (str_MIC_LxHt_con_KAIfjHb2qy) successfully subscribed
The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu
Subscribed correctly to con_KAIfjHb2qy
IceConnectionState of RTCPeerConnection 2179994c-e743-40c8-83ce-f40e3c97d5e1 (str_MIC_LxHt_con_KAIfjHb2qy) change to "checking"
Remote 'Stream' with id [str_MIC_LxHt_con_KAIfjHb2qy] video is now playing
IceConnectionState of RTCPeerConnection 2179994c-e743-40c8-83ce-f40e3c97d5e1 (str_MIC_LxHt_con_KAIfjHb2qy) change to "connected"
'Publisher' (str_CAM_EPiE_con_KCmzV9HJRO) successfully published to session
IceConnectionState of RTCPeerConnection b6b58803-2dd6-4e9e-aac5-a8e541d5cc12 (publisher of con_KCmzV9HJRO) change to "checking"
IceConnectionState of RTCPeerConnection b6b58803-2dd6-4e9e-aac5-a8e541d5cc12 (publisher of con_KCmzV9HJRO) change to "connected"
Event 'streamDestroyed' triggered by 'Session'
Event 'connectionDestroyed' triggered by 'Session'
'Connection' created (remote) with 'connectionId' [con_MAC96dKdxC]
Event 'connectionCreated' triggered by 'Session'
Remote 'Connection' with 'connectionId' [con_MAC96dKdxC] is now configured for receiving Streams with options:
Event 'streamCreated' triggered by 'Session'
Subscribing to con_MAC96dKdxC
'Subscriber' (str_MIC_OXtA_con_MAC96dKdxC) successfully subscribed
Subscribed correctly to con_MAC96dKdxC
IceConnectionState of RTCPeerConnection 15ae6cf5-e5b4-426b-8ada-b9983c6a9cf3 (str_MIC_OXtA_con_MAC96dKdxC) change to "checking"
IceConnectionState of RTCPeerConnection 15ae6cf5-e5b4-426b-8ada-b9983c6a9cf3 (str_MIC_OXtA_con_MAC96dKdxC) change to "connected"
Remote 'Stream' with id [str_MIC_OXtA_con_MAC96dKdxC] video is now playing
IceConnectionState of RTCPeerConnection 2179994c-e743-40c8-83ce-f40e3c97d5e1 (str_MIC_LxHt_con_KAIfjHb2qy) change to "disconnected". Possible network disconnection

I’m not sure exactly but it seems to me that there is next timeline:
After pressing Refresh button this.session.disconnect(); is called. Then page is loaded again and connection is created BUT old connection is not destroyed yet. It’s destroyed after some time only.

May it cause such behavior that speech detection does not work after that?

Version of OpenVidu which we use is 2.17.0.

Speech detection is actually completely performed in the browser. So an unexpected connection closure from an old Connection due to a browser refresh or a similar operation (you simply cannot trust close up events to be fired in all cases) shouldn’t affect at all to new speech detection listeners. The Connection entity may be still alive in the server, until ping-pong failure ends it after some time. But that should not affect the new Connection built after the refresh by the same final user.

Make sure your Session object is being properly re-constructed and that the event listener is being added to it. Simply treat the new Connection as a completely new user to the Session. If your app runs the same code, starting from the same intitial conditions, the first time and the second time after the refresh, all events should work in the exact same manner.

Thank you for your answer.

It looks like the deal is in “The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. Autoplay policy in Chrome - Chrome Developers” warning. If to click on page before OpenVidu initSession() then everything works. I’ve investigated source code a bit and see that there is hark.js inside and it has resume() method that can help if to do something like:

document.addEventListener('click', async () => {
                harker.resume()
            }
          }, { once: true });

My question is how to get this harker object? Or underlying instance of AudioContext.
Or maybe there is another workaround for this case.

Any solutions on my issue with harker object/AudioContext?
We are using OpenVidu in prototype and going to buy Pro or Enterprise version but it’s not fact that we can do it without proper working in all cases.

You can access the hark instance very easily:

Stream.speechEvent

In this way you could call:

await publisher.stream.speechEvent.resume();

Be aware that the hark instance will only be defined for that Stream object if any of the following events for the Publisher owning the stream has at least one active listener: publisherStartSpeaking or publisherStopSpeaking or streamAudioVolumeChange

thanks for the awesome information.

thanks my issue has been fixed.