How to apply chroma filter correctly?

OpenVidu v2 Issues developing apps
1

Hi,
I am using Openvidu 2.16 with latest kurento server. I am using following method to apply Chroma filter

publisher.stream.applyFilter(
    "ChromaFilter",
    {
        "window": {
            "topRightCornerX": 0,
            "topRightCornerY": 0,
            "width": 50,
            "height": 50
        },
        "backgroundImage": "https://www.maxpixel.net/static/photo/1x/Cool-Blue-Liquid-Lake-Abstract-Background-Clear-316144.jpg"
    });

I am getting following error

Applying filter to stream undefined OpenViduLogger.ts:31:19
Sending request: {method:"applyFilter", params: {"type":"ChromaFilter","options":"{\"window\":{\"topRightCornerX\":0,\"topRightCornerY\":0,\"width\":50,\"height\":50},\"backgroundImage\":\"https://10.21.34.246:8000/static/images/background2.jpeg\"}"}} OpenViduLogger.ts:25:19
Request: method:applyFilter params:{"type":"ChromaFilter","options":"{\"window\":{\"topRightCornerX\":0,\"topRightCornerY\":0,\"width\":50,\"height\":50},\"backgroundImage\":\"https://10.21.34.246:8000/static/images/background2.jpeg\"}"} jsonrpcclient.js:164:19
ERROR:java.lang.RuntimeException:applyFilter in Request: method:applyFilter params:{"type":"ChromaFilter","options":"{\"window\":{\"topRightCornerX\":0,\"topRightCornerY\":0,\"width\":50,\"height\":50},\"backgroundImage\":\"https://10.21.34.246:8000/static/images/background2.jpeg\"}"} request:undefined jsonrpcclient.js:172:27
ERROR DATA:"java.lang.RuntimeException: applyFilter\n\tat io.openvidu.server.rpc.RpcHandler.getStringParam(RpcHandler.java:759)\n\tat io.openvidu.server.rpc.RpcHandler.applyFilter(RpcHandler.java:489)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:152)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n" jsonrpcclient.js:176:31
    send jsonrpcclient.js:176
    dispatchCallback index.js:614
    processResponse index.js:745
    decode index.js:806
    transportMessage index.js:225
    (Async: EventListener.handleEvent)
    registerMessageHandler webSocketWithReconnection.js:145
    addEventListener webSocketWithReconnection.js:147
    setTransport index.js:251
    RpcBuilder index.js:268
    JsonRpcClient jsonrpcclient.js:143
    startWs OpenVidu.ts:769
    connectAux Session.ts:1226
    connectAux Session.ts:1225
    connect Session.ts:197
    connect Session.ts:186
    joinSession helloapp.js:17
    (Async: promise callback)
    joinSession helloapp.js:15
    onsubmit hello:1
Error applying filter for Stream undefined 
Object { code: -1, message: "java.lang.RuntimeException:applyFilter", data: "java.lang.RuntimeException: applyFilter\n\tat io.openvidu.server.rpc.RpcHandler.getStringParam(RpcHandler.java:759)\n\tat io.openvidu.server.rpc.RpcHandler.applyFilter(RpcHandler.java:489)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:152)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n", requestTime: 1610615193508 }
OpenViduLogger.ts:42:18
Uncaught (in promise) 
Object { code: -1, message: "java.lang.RuntimeException:applyFilter", data: "java.lang.RuntimeException: applyFilter\n\tat io.openvidu.server.rpc.RpcHandler.getStringParam(RpcHandler.java:759)\n\tat io.openvidu.server.rpc.RpcHandler.applyFilter(RpcHandler.java:489)\n\tat io.openvidu.server.rpc.RpcHandler.handleRequest(RpcHandler.java:152)\n\tat org.kurento.jsonrpc.internal.JsonRpcHandlerManager.handleRequest(JsonRpcHandlerManager.java:142)\n\tat org.kurento.jsonrpc.internal.server.ProtocolManager$3.run(ProtocolManager.java:218)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n", requestTime: 1610615193508 }
Stream.ts:325:35

I logged the publisher object and found that it indeed has stream value set.

{…}
​
accessAllowed: false
​
accessDenied: false
​
canPlayListener: function canPlayListener()​
ee: Object { _events: {…} }
​
element: <div id="publisher">​
firstVideoElement: Object { targetElement: div#publisher
, video: video, canplayListenerAdded: false, … }
​
isSubscribedToRemote: false
​
lazyLaunchVideoElementCreatedEvent: false
​
openvidu: Object { recorder: false, webrtcStatsInterval: 0, libraryVersion: "2.16.0", … }
​
permissionDialogTimeout: 8
​
properties: Object { audioSource: "7YmYAVqMgf0s/De3Iav3xTymvhF4fmfirT8JdJGAOnY=", insertMode: "APPEND", mirror: true, … }
​
remote: false
​
session: Object { isFirstIonicIosSubscriber: true, countDownForIonicIosSubscribersActive: true, startSpeakingEventsEnabled: false, … }
​
stream: {…}
​​
audioActive: true
​​
connection: Object { disposed: false, connectionId: "con_VSiljPS6fl", creationTime: 1610615359536, … }
​​
ee: Object { _events: {…} }
​​
frameRate: undefined
​​
hasAudio: true
​​
hasVideo: true
​​
isLocalStreamPublished: false
​​
isLocalStreamReadyToPublish: false
​​
isSubscribeToRemote: false
​​
outboundStreamOpts: Object { mediaConstraints: {…}, publisherProperties: {…} }
​​
publishedOnce: false
​​
publisherStartSpeakingEventEnabled: false
​​
publisherStartSpeakingEventEnabledOnce: false
​​
publisherStopSpeakingEventEnabled: false
​​
publisherStopSpeakingEventEnabledOnce: false
​​
session: Object { isFirstIonicIosSubscriber: true, countDownForIonicIosSubscribersActive: true, startSpeakingEventsEnabled: false, … }
​​
streamManager: Object { userHandlerArrowHandler: WeakMap(0), lazyLaunchVideoElementCreatedEvent: false, remote: false, … }
​​
typeOfVideo: "CAMERA"
​​
userHandlerArrowHandler: WeakMap(0)
​​
videoActive: true
​​
volumeChangeEventEnabled: false
​​
volumeChangeEventEnabledOnce: false
​​
<prototype>: Object { constructor: Stream(session, options), on: on(type, handler), once: once(type, handler)
, … }
​
targetElement: <div id="publisher">​
userHandlerArrowHandler: WeakMap(0)
​
videos: Array []
​
<prototype>: Object { constructor: Publisher(targEl, properties, openvidu), publishAudio: publishAudio(value), publishVideo: publishVideo(value)
, … }
helloapp.js:38:29

What could be the problem ?

2

Hello,

Did you install this?

You must install Kurento ChromaFilter library in the same host as Kurento Media Server:
sudo apt-get install kms-chroma

Cheers,
Mihail J

3

I am using latest docker image of kurento media server which I suppose comes with chroma filter installed. I just managed to apply filter on subscriber stream. But the background image appearing is very disappointing. Did I miss something ?
kurento_chroma_filter_openvidu

4

Hello @lkpatel,

Can you describe why the filter wasn’t working for you in the first try? To help others that found this issue on Internet.

ChromaFilter is just a sample filter implemented for Kurento.

It lacks some production features like threshold configuration to adjust it to variances in the background green.

Remember that Chroma Filter is opensource and you can adapt it to your needs.

Regards

5

@micael.gallego I am still struggling to apply filter in publishing steam. It shows stream undefined. May be stream not available at the moment when I am calling it to apply filer due to asynchronous nature of code. From the help doc, I learned that I can apply filter on subscribe stream as well. So tried to apply it on subscriber stream. I am studying about working of chroma filer. Things improve a bit if I reduce published video frame rate. So I need to adopt the filter as per my need. I will definitely update once I found a acceptable solution to my problem. Thanks for your kind suggestion.
Regards

6

From the reply of Juan Navarro in a google group question,


On 21/9/20 16:06, Carlos Navarro wrote:
> it's just to give an idea of the possibilities, not a real filter use

That's exactly the case. The filter showcases integration with OpenCV, but the actual code is just as dummy as it gets: look for a given color, and try to replace it with a given background image.

Real-World (tm) background substitution would have to use much more advanced techniques such as detecting color gradients, sharp edges, changes in lightning, or (more probably) a mixture of all these and more. Even, as of today the best background replacers are using some kind of machine learning to "know" what is the shape of a person and help it with the detection of the person's edge.

Regards,

--
Juan Navarro
Kurento developer
Connect: @j1elo at GitHub & Twitter

I came to know that chroma filter at this stage require a lot of work to be used in real world.
Regards

Ref- https://groups.google.com/g/kurento/c/1VOK0c2O4b4/m/z8cW5MG-BAAJ

7

Well, actually the Chroma filter works fine. But of course a chroma filter will do that: search for a color and replace it with an image. It is not a “background-replacing” filter that will get rid of any kind of background, cutting the user’s silhouette. You need a single-color background for it to work fine. That’s why you get this result:
image

Best regards.

8

@pabloFuente Thanks for clearing my doubts.