Live stream HTML5 video element to canvas not working

How to implement?
1

I use ReactJS to display a live stream (from my webcam) using the HTML5 video element. The OpenVidu media server handles the backend. I would like to use the canvas element to draw the video live stream onto the canvas using the drawImage() method.

I have seen other examples, but in them the video element always has a source. Mine does not have a source - when I inspect the video element all I see is: <video autoplay id="remote-video-zfrstyztfhbojsoc_CAMERA_ZCBRG"/>

This is what I have tried, however the canvas does not work

export default function Canvas({ streamManager }) {
  const videoRef = createRef()
  const canvasRef = createRef()

  useEffect(() => {
    if (streamManager && !!videoRef) {

      //OpenVidu media server attaches the live stream to the video element 
      streamManager.addVideoElement(videoRef.current)

      if (canvasRef.current) {
        let ctx = canvasRef.current.getContext('2d')
        ctx.drawImage(videoRef.current, 0, 0)
      }
    }
  })

  return (
    <>
      //video element displays the live stream 
      <video autoPlay={true} ref={videoRef} />

      // canvas element NOT working, nothing can be seen on screen 
      <canvas autoplay={true} ref={canvasRef} width="250" height="250" />
    </>
  )
}

Thanks in advance

2

Hello Marc,

We are not experts on React, however In openvidu-react-native tutorial we get the reference to RTCView and we will run openvidu-browser method addVideoElement , passing the reference of the RTCView when it is defined:

{this.state.subscribers.map((item, index) => {
    if(!!item){
        return (
            <View key={index}>
                <Text>{this.getNicknameTag(item.stream)}</Text>
                <RTCView zOrder={0}  objectFit="cover" style={styles.remoteView}  ref={(rtcVideo) => {
                    if (!!rtcVideo){
                        item.addVideoElement(rtcVideo);
                    }
                }} />
            </View>
        )
}

You could do the same with your canvas:

<canvas	autoplay={true}	ref={(canvasRef) => {
		if (!!canvasRef) {
			// do something
		}
	}}
	width="250"
	height="250"
/>;

Regards

3

@CSantosM thanks for the info. After further investigation, I found the solution. You could do it like that and with a setInterval(), but the problem with this approach is that when the component unmounts it will NOT clear the setInterval().

  <canvas
    ref={canvasRef => {
      if (canvasRef && videoRef.current) {
        const timer = setInterval(() => {
          const ctx = canvasRef.getContext('2d')
          ctx.drawImage(videoRef.current, 0, 0, 250, 188)
        }, 100)
      }
    }}
    width="250"
    height="188"
  />

The correct solution is provided below.

4

Therefore, the best solution is to extract the canvas logic in a self contained component, and use a setInterval() so that it will draw the video element on the canvas, every 100 ms (or as needed).
I opted for this solution because displaying a lot of live streams on a page using the HTML5 video element uses a considerable amount of CPU. When it comes to canvas, you can set the interval/refresh rate and that lowers the CPU usage significantly. Code as follows:

OVVideo Component

import React, { useEffect, createRef } from 'react'
import Canvas from './Canvas'

export default function OVVideo({ streamManager }) {
  const videoRef = createRef()

  useEffect(() => {
    if (streamManager && !!videoRef) {

      //OpenVidu media server attaches the live stream to the video element 
      streamManager.addVideoElement(videoRef.current)
    }
  })

  return (
    <>
      //video element displays the live stream 
      <video autoPlay={true} ref={videoRef} />

      //extract canvas logic in new component
      <Canvas videoRef={videoRef} />

    </>
  )
}

Canvas component

import React, { createRef, useEffect } from 'react'

export default function Canvas({ videoRef }) {
  const canvasRef = createRef(null)

  useEffect(() => {
    if (canvasRef.current && videoRef.current) {
      const interval = setInterval(() => {
        const ctx = canvasRef.current.getContext('2d')
        ctx.drawImage(videoRef.current, 0, 0, 250, 188)
      }, 100)
      return () => clearInterval(interval)
    }
  })

  return (
    <canvas ref={canvasRef} width="250" height="188" />
  )
}