WebRTC Low Latency Streaming Adds WebRTC features into JWPlayer 8

Introduction

The WebRTC is a browser based low latency peering streaming solution with only the requirement of a signal server.

While WebRTC can allow for full 2 way peering, the intended solution is for one to many low latency live broadasting with basic conferencing functions.

With Kurento, Millicast, Ant Media Server and Wowza Media Server conferencing solutions is provided however with Wowza the streams are pulled from the server as WebRTC or HLS. Demo java backend projects are provided.

With Millicast conferencing a demo node websocket service is provided to signal users when a publish stream begins.

The WebRTC plugin provides an integrated browser based encoder publishing application, with media device, resolution, codec, bitrate controls. And a subscriber player.

Media device selection can use whatever device the browser detects including video and audio inputs. Audio output control for the publisher is possible for browsers that support it.

Screensharing capabilities with browsers that support it are possible using screen sharing apis. The screen, browser, browser tab can be chosen , with an audio input for audio. This may be suitable for live broadcasting instruction videos or even gaming.

Switching between camera and screenshare video tracks is possible while peering. Retaining the original microphone audio stream.

Recording of the media device is possible using browser based media recorder api's for browsers that support it. With a download capability of the recorded data.

Device toggling can turn the video / audio device off before and while publishing.

Supported servers for now is Wowza, Kurento, Millicast WebRTC backends. Wowza only supports one to many peering and no data channels. Provided is a Kurento one to many signal server.

WebRTC settings can be configured as a player overlay,

Volume control of the audio input is made possible using the audio api. This also makes the possibility of an audio meter display

Virtual Meeting Background Removal

With an included virtual background plugin. Support for dynamic background removal rendering is provided.

This rendering view can be used as the main video stream on the publisher and will be automatically detected.

The auto background uses an AI BodyPix feature that doesn't require a Chromakey Green Screen render. It's not perfect but with fine tuned settings it's enough to make a transparent background to show a background image behind the video.

A Chromakey rendering option for Green Screens is supported with option to configure the flat background colour or choose an auto option for choosing any background of a solid colour.

Features

Low latency streaming via the browser.
One to many live streaming via WebRTC
Group call conferencing as WebRTC or HLS.
Group conferencing features for Wowza, Millicast and Kurento.
Ability to feature a participant in a conference by an admin.
Media device video and audio selection.
Encoder application selections of bitrate, resolution, framerate, audio input, audio output and video sources.
Support for external UI control if managed select menus is not suitabled.
Recording of local video stream support with a download option.
Toggling of audio and video device before and while publishing.
Screensharing video capabilities using screensharing apis instead of camera and microphone.
Screensharing toggling support between screen capture and camera. While peering the tracks are replaced in realtime. Audio is retained while screen capturing.
Microphone Audio peak meter display option.
Full peering events support with debug capabilities.
Audio volume control of the microphone input and audio meter display functionality.
Support for dynamic background removal for virtual meetings without Green Screens using Bodypix AI.
Support for Chromakey Green Screen background removal rendering.

(Current Version 8.0.0)

Compatibility

JWPlayer 8 and above

Demo

Contact for a site demo download.

Examples

Basic Example

configuration


  

(function(jwplayer) {

    var player = jwplayer("player").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    ui: {
                        audioInput: "audioSource",
                        audioOutput: "audioOutput",
                        videoSource: "videoSource",
                        bandwidth: "bandwidth",
                        resolution: "resolution",
                        rates: "rates",
                        container: "#encoder-menu"
                    },
                    /*dataCallback: function(devicesMap, deviceInfos, supportsAudioSink) {
                        return new Promise((resolve) => {
                            resolve({
                                videoFrameRate: 30,
                                videoResolution: {
                                    width: 1280,
                                    height: 720
                                },
                                videoSource: "743094318ec58aa8dab2120cd9d85f6fc309261a15c93b8a00c7e2339a375714",
                                //audioInput: "deviceid"
                            });
                        });
                    },*/
                    autoStartDevice: false,
                    serverURL: "rtc.electroteque.org",
                    //serverURL: "in-staging.videolinq.net",
                    applicationName: "webrtc",
                    //streamName: "myStream",
                    server: "wowza",
                    maxVideoBitrate: 2000,
                    minVideoBitrate: 2000,
                    startVideoBitrate: 300,
                    audioBitrate: 64,
                    videoFrameRate: 30,
                    preferredCodecs: {
                                  "audio": {
                                    codec: "opus",
                                  },
                                  "video": {
                                    codec: "H264",
                                    //codec: "VP9",
                                    level: "42e01f",
                                    //mode: 0
                                  }
                    },
                    opus: {
                        stereo: 1
                    },
                    userData: {param1:"value1"},
                    debug: true,
                    publisher: true,
                    recording: {
                        mimeType: "video/webm",
                        codec: "VP9",
                        name: "recording1"
                    },
                    exactVideoSize: false,
                    pcConstraints: {
                        //optional: [
                            //{ googIPv6: true },
                            //{ googImprovedWifiBwe: true },
                            //{ googDscp: true },
                            //{ googSuspendBelowMinBitrate: true },
                           // { googScreencastMinBitrate:300 },
                            //{ googCombinedAudioVideoBwe: true },
                            //turn off resolution changes
                            //{ googCpuOveruseDetection: false },
                           // { googCpuOveruseEncodeUsage: false },
                            //{ googCpuUnderuseThreshold: 55 },
                            //{ googCpuOverUseThreshold: 85 }
                        //]
                    },
                    exactVideoSize: false,
                     toggleScreen: true,
                    settings: true
                    //screen: true
                }
        }

            
    });
    player.on("ready", function() {
            player.on("devices", (event, devicesMap, deviceInfos) => {
                //get available devices here for building your own UI
                console.log("devices", devicesMap, deviceInfos);
                

            }).on("mediastart", (e, deviceInfo) => {
                //console.log("mediastart111", deviceInfo);
                //selected device has been activated
                console.log("Device Start ", deviceInfo);

            }).on("mediastop", (e) => {
                console.log("Device Stop");
            }).on("startpublish", (e, supportsParams) => {
                console.log("Publishing Started");
                //use this for enabling / disabling bandwidth select menus while publishing
                //browsers that don't support it cannot update bitrate controls while publishing
                //console.log("Browser Supports peer setParameters ", supportsParams);
            }).on("stoppublish", (e) => {
                console.log("Publishing Stopped");
            }).on("recordstart", (e) => {
                console.log("Recording Started");
            }).on("recordstop", (e) => {
                console.log("Recording Stopped");
            }).on("sendmessage", (e, message) => {
                console.log("Signal Server Send Message: ", message);
            }).on("gotmessage", (e, message) => {
                console.log("Signal Server Receive Message: ", message);
            }).on("offer", (e, offer) => {
               // console.log("WebRTC Offer ", offer.sdp);
            }).on("answer", (e, answer) => {
               // console.log("WebRTC Answer ", answer.sdp);
            }).on("bitratechanged", (e, params) => {
                console.log("WebRTC Bitrate Changed ", params);
            }).on("outputsuccess", (e, sinkId) => {
                console.log(`Success, audio output device attached: ${sinkId}`);
            }).on("outputerror", (e, message) => {
                console.log(message);
            }).on("ready", function(e, api, video) {
                console.log("READY", video);
            });
    });

Supported Codecs

H264, VP8, VP9
Opus audio codec.

Supported Servers

Wowza Streaming Server
Kurento Streaming Server
Millicast
Ant Media Server

Configuration

To enable a player as a publisher set the player needs the `publisher` config enabled.

Configuring stream name and server application

return with the configured options.


 {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }

Publisher Options

property / datatype	default	description
publisher boolean	false	Enable the player as the publisher. Which enables the full WebRTC encoder publishing application.
serverURL string		The host of the websocket service. For Wowza the url is configured from the host.
userData object		Custom data to send to the Wowza server
server string	wowza, kurento, millicast, wowza-conference, kurento-conference, millicast-conference	The server type. For conferencing, configure with the postfix -conference.
debug boolean	false	Turn on internal peering logs.
prefferedCodecs object	`{ "audio": { codec: "opus", }, "video": { codec: "H264", //codec: "VP9", level: "42e01f" } }`	Filter the codec used for audio and video. With a H264 level set for H264 codecs. This is required for Chrome to work with Wowza.
opus object	{ stereo: 1 },	Opus codec configs
ui object	`{ audioInput: "audioSource", audioOutput: "audioOutput", videoSource: "videoSource", bandwidth: "bandwidth", resolution: "resolution", rates: "rates", container: "#encoder-menu" }`	CSS selectors for managing external select menus with data and control of WebRTC and media options. Disable this if using external controls with the device data provided.
dataCallback function		To manage the encoder controls externally, the device data is provided in this callback to build the device menus. And return with the configured options. `function(devicesMap, deviceInfos, supportsAudioSink) { return new Promise((resolve) => { resolve({ videoFrameRate: 30, videoResolution: { width: 1280, height: 720 }, videoSource: "743094318ec58aa8dab2120cd9d85f6fc309261a15c93b8a00c7e2339a375714", //audioInput: "deviceid" }); }); }`
autoStartDevice boolean	false	Enable to auto activate the device on initialisation. Or turned on when clicking the player.
maxBitrateBitrate number	2000	Configure an initial max bitrate bitrate. This is required for Chrome to function with Wowza.
minVideoBitrate number	300	Minimum video bitrate. Used on Chrome only.
startVideoBitrate number	300	Starting video bitrate. Used on Chrome only.
audioVideoBitrate number	64	Audio codec bitrate.
recording object		Enable recording of the local stream with a download option. Config requires the filename for the download and the mimetype and codec. For H264 use video/mp4 and H264 as the codec. `{ mimeType: "video/webm", codec: "VP9", name: "recording1" }`
pcConstraints object		Set browser specific Peer connection options. `{ optional: [ { googIPv6: true }, { googImprovedWifiBwe: true }, { googDscp: true }, { googSuspendBelowMinBitrate: true }, { googScreencastMinBitrate:300 }, { googCombinedAudioVideoBwe: true }, //turn off resolution changes { googCpuOveruseDetection: false }, { googCpuOveruseEncodeUsage: false }, { googCpuUnderuseThreshold: 55 }, { googCpuOverUseThreshold: 85 } ] }`
toggleScreen boolean	false	Enable camera / share screen switching features
settings boolean	false	Enable the WebRTC device settings as a settings overlay

WebRTC Group Call Conferencing Options

property / datatype	default	description
roomName string		Required for Kurento group call conferencing.
conferenceContainer string		The container selector to host the remote participant's stream in a player.
hlsTemplate object	https://{serverURL}/{appName}/{streamName}/playlist.m3u8	When using Wowza. The HLS url template to use when subscribing in a group call as HLS.
conferenceSubscribeHls boolean	false	Set to true to subscribe to remote participants in a group call as HLS instead of WebRTC.
peerSubscribe boolean	false	Required for Kurento group call conferencing.

Subscriber Options

property / datatype	default	description
serverURL string		The host of the websocket service. For Wowza the url is configured from the host.
userData object		Custom data to send to the Wowza server
server string	wowza	The server type. For now Wowza is supported with other signalling servers after.
debug boolean	false	Turn on internal peering logs.

Millicast Options

Configuration example



                millicast: {
                        accountId: "",
                        publishToken: ""
                }

property / datatype	default	description
accountId string		Millicast account id shown in the api key section.
publishToken string		The publishing token created in the api section. This is used for generating the signal server url unless supplied using serverURL
subToken string		Subscriber token generated by the backend. This is used for generating the signal server url with authenticated subscribing enabled. Unless supplied using serverURL

Virtual Background Removal Plugin Options

property / datatype	default	description
renderType string	bodypix	The background rendering type. Either bodypix, or chromakey
backgroundType string	green	If Chromakey rendering is set. Choose from either Green Screen colour detection or auto for solid colour backgrounds.
backgroundColor array	[0, 255, 0]	Specify the background colour in an rgb value array.
accurate boolean	false	Use the fast or slow Bodypix architecture config. Slow is more accurate but more intensive.
internalResolution string	medium	The resolution for segmenting the detected body from the background. Medium or high.
segmentationThreshold number	0.5	An accuracy threshold for segmentation
segmentationThreshold number	0.5	An accuracy threshold for segmentation
fast object	{ architecture: 'MobileNetV1', outputStride: 16, multiplier: 0.75, quantBytes: 2 }	The fast least accurate Bodypix config
slow object	{ architecture: 'ResNet50', outputStride: 16, multiplier: 1, quantBytes: 2 },	The slow more accurate Bodypix config

Wowza WebRTC Configuration

Provided is a set of example configs for wowza with install script to get the WebRTC signal server working. Provided is a script to convert PEM SSL certificates into JKS format for Wowza.

An SSL certificate is required to be configured into the Secure SSL provider config inside VHost.xml.

The Websocket provider also needs to be configured inside VHost.xml

Configuration example


 
                      
        com.wowza.wms.webrtc.http.HTTPWebRTCExchangeSessionInfo
       
                      
        *webrtc-session.json
       
                      
        none

Wowza WebRTC Conferencing Configuration

To enable the conferencing examples. The supplied WebRTCCustomSignalServer Wowza IDE project, needs to be imported and built to be configured and loaded.

This provides the custom WebRTC signal server for WebRTC publishing with token protection. Another WebSocket provider is for group call signal messaging and to notify when a WebRTC publish begins.



            
        org.electroteque.webrtc.HTTPWebRTCExchangeSessionInfoCustom
       
            
        *webrtc-session.json
       
            
        none
       
          
      
          
            
        org.electroteque.webrtc.HTTPProviderGroupCallWebSocket
       
            
        *webrtc-room-session.json
       
            
        none

Kurento WebRTC Configuration

Kurento Media Server comes in two parts. The Kurento Media Server and a java / node JSONRPC client WebSocket server.

Provided is a one to many application aswell as a group call application to run the group call example.

The one to many application endpoint is "one2many"

The group call application endpoint is "groupcall"

Start the Kurento server first with


export MAKEFLAGS="-j$(nproc)"
./bin/kms-build-run.sh

Start the Kurento client Websocket server with


mvn -U clean spring-boot:run -Dkms.url=wss://localhost:8433/kurento

JavaScript API

Methods

The api methods are exposed on the Flowplayer 7 api.



    const ret = player.setStream(stream);

    const ret = player.setVideoStream(stream);

    const ret = player.toggleDevices();

    const ret = player.togglePublish();

    const ret =  player.toggleAudio();

    const ret = player.toggleVideo();

    const ret = player.toggleScreenShare();

    player.send(data);

    player.sendSignalMessage(message);

    player.setAudioGain(gain);

    const devices = player.getDevices();

    player.getDevice(constraint = null, videoIndex = -1, audioIndex = -1).then((device) => {});

method	returns	description
setStream		Switch to a new MediaStream from another source that isn't part of the Camera device list.
setVideoStream		Switch the video track of a new MediaStream from another source that isn't part of the Camera device list. This keeps the original audio track.
toggleDevices		Toggle enabling and playing the selected Camera / Microphone device.
togglePublish		Toggle publishing a WebRTC stream.
toggleAudio		Toggle the device audio track.
toggleVideo		Toggle the device video track.
toggleScreenShare		Toggle screen sharing.
send		Send a message over the WebRTC peering data channel.
sendSignalMessage		Send a message to the WebRTC signal server.
audioGain		Set the input audio gain of the microphone device.
getDevices		Get a map of the detected devices.
getDevice		Obtain the MediaStream of a new WebRTC device. Provide a constraint , or the index of the video and audio device provided in the devices map.

Subscriber Example

This example demonstrates setting up the WebRTC subscriber player.



(function(jwplayer) {

    var player2 = jwplayer("subscriber").setup({
        width: "100%",
        aspectratio: "16:9",
        

        playlist: [
            {
                sources: [
                    {type: "mp4", file: "myStream", appName: "webrtc", publisher: false, live: true },
                    {type: "application/x-mpegurl", src: "http://rtc.electroteque.org:1935/webrtc/myStream_360p/playlist.m3u8"}
                ]
            }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcsubscriber-8.0.0.js": {
                    server: "wowza",
                    serverURL: "rtc.electroteque.org",
                    //serverURL: "in-staging.videolinq.net",
                    debug: true
            }
        }

            
    });
    player2.on("ready", function() {
            player2.on("sendmessage", (e,api, message) => {
                console.log("Signal Server Send Message: ", message);
            }).on("gotmessage", (e, api, message) => {
                console.log("Signal Server Receive Message: ", message);
            }).on("offer", (e, api, offer) => {
                console.log("WebRTC Offer ", offer.sdp);
            }).on("answer", (e, api, answer) => {
                console.log("WebRTC Answer ", answer.sdp);
            }).on("resize", (e, api, size) => {
                console.log("Receiving Video Size ", size);
            });
    });


  

})(window.jwplayer);

Virtual Background Removal Example

This example demonstrates dynamic virtual background removal using Bodypix and Tensorflow AI.

Backgrounds can be applied using CSS on the publisher and subscriber. The video element needs to be shown or rendering stops.



(function(jwplayer) {

    player = jwplayer("virtual").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    serverURL: "rtc.electroteque.org",
                    applicationName: "webrtc",
                    server: "wowza",
                    publisher: true
                },

                "https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {

                },
                "https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {

                }
        },
        


            
    });

})(window.jwplayer);

Virtual Chromakey Green Screen Removal Example

This example demonstrates dynamic virtual background removal using Green Screen Chromakey.

If set to auto render type any solid background color can be used or a color can be chosen.



(function(jwplayer) {

    player = jwplayer("virtual").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    serverURL: "rtc.electroteque.org",
                    applicationName: "webrtc",
                    server: "wowza",
                    publisher: true
                },

                "https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {

                },
                "https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {

                }
        },
        


            
    });

})(window.jwplayer);

Electroteque Media