Character encoding problem to use Send a signal API

Hi!

I use Send a signal to a Session REST API and record chat conversation using custom-layout.
but When using the East Asian character set, the character “???” A phenomenon has occurred.

version info : openvidu server 2.18

  • send signal event(REST API call “POST /openvidu/api/signal”) : sending data “data”:“signal シグナル 시그널 信号”

curl -u OPENVIDUAPP:MY_SECRET -d ‘{“session”:“ses_YnDaGYNcd7”,“to”: [“con_Xnxg19tonh”, “con_TNVdbuQCJF”],“type”:“MY_TYPE”,“data”:“signal シグナル 시그널 信号”}’ -H “Content-Type: application/json;charset=utf-8” -X POST http://127.0.0.1:5443/openvidu/api/signal

  • server log : character “???” A phenomenon has occurred

[INFO] 2021-06-09 00:34:33,932 [http-nio-0.0.0.0-5443-exec-4] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/signal {session=ses_YnDaGYNcd7, to=[con_Xnxg19tonh, con_TNVdbuQCJF], type=MY_TYPE, data=signal ??? ??? ???}

and record chat conversation using custom-layout

  • screen capture : chat conversation recording file character encoding error

  • front app code : using openvidu-browser 2.18

sendMessage(message: string) {
message = message.replace(/ +(?= )/g, ‘’);
if (message !== ‘’ && message !== ’ ') {
const data = {
message: message,
nickname: this.oVSessionService.getWebcamUserName()
};
const sessionAvailable = this.oVSessionService.getConnectedUserSession();
sessionAvailable.signal({
data: JSON.stringify(data),
to: this.to,
type: this.signal
});
}
}

  • server log : character “???” A phenomenon has occurred

[INFO] 2021-06-09 01:13:50,890 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.rpc.RpcHandler - Participant con_GbeaPEcjRQ is calling method ‘signal’ in session roomA
[INFO] 2021-06-09 01:13:50,891 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.cdr.CDRLoggerFile - {“signalSent”:{“sessionId”:“roomA”,“uniqueSessionId”:“roomA_1623201025049”,“timestamp”:1623201230891,“from”:“con_GbeaPEcjRQ”,“to”:[“con_Fcc21sDvIR”,“con_GbeaPEcjRQ”,“RECORDER”],“type”:“chat”,“data”:"{“message”:“signal”,“nickname”:“teacher”}"}}
[INFO] 2021-06-09 01:14:26,749 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.rpc.RpcHandler - Participant con_GbeaPEcjRQ is calling method ‘signal’ in session roomA
[INFO] 2021-06-09 01:14:26,752 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.cdr.CDRLoggerFile - {“signalSent”:{“sessionId”:“roomA”,“uniqueSessionId”:“roomA_1623201025049”,“timestamp”:1623201266752,“from”:“con_GbeaPEcjRQ”,“to”:[“con_Fcc21sDvIR”,“con_GbeaPEcjRQ”,“RECORDER”],“type”:“chat”,“data”:"{“message”:"??? “,“nickname”:“teacher”}”}}
[INFO] 2021-06-09 01:14:37,703 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.rpc.RpcHandler - Participant con_GbeaPEcjRQ is calling method ‘signal’ in session roomA
[INFO] 2021-06-09 01:14:37,704 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.cdr.CDRLoggerFile - {“signalSent”:{“sessionId”:“roomA”,“uniqueSessionId”:“roomA_1623201025049”,“timestamp”:1623201277704,“from”:“con_GbeaPEcjRQ”,“to”:[“con_Fcc21sDvIR”,“con_GbeaPEcjRQ”,“RECORDER”],“type”:“chat”,“data”:"{“message”:"??? “,“nickname”:“teacher”}”}}
[INFO] 2021-06-09 01:14:43,386 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.rpc.RpcHandler - Participant con_GbeaPEcjRQ is calling method ‘signal’ in session roomA
[INFO] 2021-06-09 01:14:43,387 [SessionHandler-oe8p6oh7u0aihahms8ae6njh2q-e4-t0] io.openvidu.server.cdr.CDRLoggerFile - {“signalSent”:{“sessionId”:“roomA”,“uniqueSessionId”:“roomA_1623201025049”,“timestamp”:1623201283387,“from”:“con_GbeaPEcjRQ”,“to”:[“con_Fcc21sDvIR”,“con_GbeaPEcjRQ”,“RECORDER”],“type”:“chat”,“data”:"{“message”:"??",“nickname”:“teacher”}"}}

Is there any solution that with some OpenVidu Server configuration ?

Hi,

In version 2.18.0 we updated all character encodings to UTF-8 on openvidu-server. This encoding should support what I think it is Korean (sorry if I’m mistaken, I am not well versed in Asian languages).
This is OpenVidu Call application 2.18.0 against OpenVidu Server 2.18.0:

As you can see the message travels without a problem end-to-end.
It is true that openvidu-server logs this message:

[INFO] 2021-06-09 10:19:19,261 [https-jsse-nio-0.0.0.0-4443-exec-10] io.openvidu.server.rest.SessionRestController - REST API: POST /openvidu/api/signal {session=institutional-green-donkey, to=[], data=signal ???? ??? ??}

But that’s probably a problem with the logging operation itself, and it does not seem to affect the final message sent to the participants.

I have also been testing the CDR and WebHook with your custom message and I received it no problem:

{"signalSent":{"sessionId":"TestSession","uniqueSessionId":"TestSession_1623234638137","timestamp":1623234665564,"from":null,"to":["con_VtqVLvCXQd"],"type":null,"data":"signal シグナル 시그널 信号"}}

I’ve seen messages pass end-to-end without problems, but problem with the logging operation.

I am also composed recording chat conversations using custom-layout.

  • version : openvidu server 2.18
  • outputMode : COMPOSED

But there is a problem with the recording file.

There is an problem displaying the character.

[81:81:0610/043206.304276:INFO:CONSOLE(172876)] “Received request: {“method”:“sendMessage”,“params”:{“data”:”{“message”:“signal”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}}", source: (172876)
[81:81:0610/043206.304357:INFO:CONSOLE(172888)] “New signal: {“data”:”{“message”:“signal”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}", source: (172888)

[81:81:0610/043211.030850:INFO:CONSOLE(172876)] “Received request: {“method”:“sendMessage”,“params”:{“data”:”{“message”:“シグナル “,“nickname”:“teacher”}”,“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}}”, source: (172876)
[81:81:0610/043211.030885:INFO:CONSOLE(172888)] “New signal: {“data”:”{“message”:“シグナル “,“nickname”:“teacher”}”,“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}”, source: (172888)

[81:81:0610/043216.183982:INFO:CONSOLE(172876)] “Received request: {“method”:“sendMessage”,“params”:{“data”:”{“message”:“시그널”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}}", source: (172876)
[81:81:0610/043216.184021:INFO:CONSOLE(172888)] “New signal: {“data”:”{“message”:“시그널”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}", source: (172888)
[81:81:0610/043216.184054:INFO:CONSOLE(172888)] “Event ‘signal:chat’ triggered by ‘Session’ [object Object]”, source: (172888)

[81:81:0610/043222.327787:INFO:CONSOLE(172876)] “Received request: {“method”:“sendMessage”,“params”:{“data”:”{“message”:“信号”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}}", source: (172876)
[81:81:0610/043222.327848:INFO:CONSOLE(172888)] “New signal: {“data”:”{“message”:“信号”,“nickname”:“teacher”}",“type”:“signal:chat”,“from”:“con_BC95tzqfTt”}", source: (172888)

Is there any solution ?

Ok, we have a theory. Maybe the thing is that the recording module does not have installed the needed fonts. It doesn’t matter that UTF-8 encoding is used if the underlying OS does not have the required fonts.
We have released a patched version of the recording module with the fonts properly installed.
We have also added a new flag to avoid getting that annoying “translate” pop up at the upper right corner.

To test it just add the following configuration property to your openvidu-server .env file:

OPENVIDU_RECORDING_VERSION=2.18.0-all-fonts

Please, test it and let us know if both things are fixed (the lack of korean characters in the recording file and the translate pop up).

Best regards.

I tested it. there is no problem with the record file.

Issue has got solved thank you for help.

Best regards,
pablo Fuente

We are still adjusting the fix. The size of the recording module was too big on the first approach (from 370MB went to 800MB just by installing new fonts. Which is unsustainable).
We have been fine-tuning the installed fonts to reach a compromise. The new recording module has grown up to 417MB with the new selected fonts.

Please, if you can repeat the test to confirm that it still works, that would be much appreciated.

To download the new module just run this command on your nodes before starting OpenVidu:

docker pull openvidu/openvidu-recording:2.18.0-all-fonts

Best regards.

I tested new openvidu-recording version.
there is no problem with the record file.

docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
openvidu/openvidu-recording 2.18.0-all-fonts dd568c766abf 5 days ago 1.08GB

Best regards.