WebRTC Low Latency Streaming Adds WebRTC features into JWPlayer 8


Introduction

The WebRTC is a browser based low latency peering streaming solution with only the requirement of a signal server.

While WebRTC can allow for full 2 way peering, the intended solution is for one to many low latency live broadasting with basic conferencing functions.

With Kurento, Millicast, Ant Media Server and Wowza Media Server conferencing solutions is provided however with Wowza the streams are pulled from the server as WebRTC or HLS. Demo java backend projects are provided.

With Millicast conferencing a demo node websocket service is provided to signal users when a publish stream begins.

The WebRTC plugin provides an integrated browser based encoder publishing application, with media device, resolution, codec, bitrate controls. And a subscriber player.

Media device selection can use whatever device the browser detects including video and audio inputs. Audio output control for the publisher is possible for browsers that support it.

Screensharing capabilities with browsers that support it are possible using screen sharing apis. The screen, browser, browser tab can be chosen , with an audio input for audio. This may be suitable for live broadcasting instruction videos or even gaming.

Switching between camera and screenshare video tracks is possible while peering. Retaining the original microphone audio stream.

Recording of the media device is possible using browser based media recorder api's for browsers that support it. With a download capability of the recorded data.

Device toggling can turn the video / audio device off before and while publishing.

Supported servers for now is Wowza, Kurento, Millicast WebRTC backends. Wowza only supports one to many peering and no data channels. Provided is a Kurento one to many signal server.

WebRTC settings can be configured as a player overlay,

Volume control of the audio input is made possible using the audio api. This also makes the possibility of an audio meter display

Virtual Meeting Background Removal

With an included virtual background plugin. Support for dynamic background removal rendering is provided.

This rendering view can be used as the main video stream on the publisher and will be automatically detected.

The auto background uses an AI BodyPix feature that doesn't require a Chromakey Green Screen render. It's not perfect but with fine tuned settings it's enough to make a transparent background to show a background image behind the video.

A Chromakey rendering option for Green Screens is supported with option to configure the flat background colour or choose an auto option for choosing any background of a solid colour.

Features

  • Low latency streaming via the browser.
  • One to many live streaming via WebRTC
  • Group call conferencing as WebRTC or HLS.
  • Group conferencing features for Wowza, Millicast and Kurento.
  • Ability to feature a participant in a conference by an admin.
  • Media device video and audio selection.
  • Encoder application selections of bitrate, resolution, framerate, audio input, audio output and video sources.
  • Support for external UI control if managed select menus is not suitabled.
  • Recording of local video stream support with a download option.
  • Toggling of audio and video device before and while publishing.
  • Screensharing video capabilities using screensharing apis instead of camera and microphone.
  • Screensharing toggling support between screen capture and camera. While peering the tracks are replaced in realtime. Audio is retained while screen capturing.
  • Microphone Audio peak meter display option.
  • Full peering events support with debug capabilities.
  • Audio volume control of the microphone input and audio meter display functionality.
  • Support for dynamic background removal for virtual meetings without Green Screens using Bodypix AI.
  • Support for Chromakey Green Screen background removal rendering.

(Current Version 8.0.0)

Compatibility

  • JWPlayer 8 and above

Demo

Contact for a site demo download.

Examples

Basic Example

configuration


  

(function(jwplayer) {

    var player = jwplayer("player").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    ui: {
                        audioInput: "audioSource",
                        audioOutput: "audioOutput",
                        videoSource: "videoSource",
                        bandwidth: "bandwidth",
                        resolution: "resolution",
                        rates: "rates",
                        container: "#encoder-menu"
                    },
                    /*dataCallback: function(devicesMap, deviceInfos, supportsAudioSink) {
                        return new Promise((resolve) => {
                            resolve({
                                videoFrameRate: 30,
                                videoResolution: {
                                    width: 1280,
                                    height: 720
                                },
                                videoSource: "743094318ec58aa8dab2120cd9d85f6fc309261a15c93b8a00c7e2339a375714",
                                //audioInput: "deviceid"
                            });
                        });
                    },*/
                    autoStartDevice: false,
                    serverURL: "rtc.electroteque.org",
                    //serverURL: "in-staging.videolinq.net",
                    applicationName: "webrtc",
                    //streamName: "myStream",
                    server: "wowza",
                    maxVideoBitrate: 2000,
                    minVideoBitrate: 2000,
                    startVideoBitrate: 300,
                    audioBitrate: 64,
                    videoFrameRate: 30,
                    preferredCodecs: {
                                  "audio": {
                                    codec: "opus",
                                  },
                                  "video": {
                                    codec: "H264",
                                    //codec: "VP9",
                                    level: "42e01f",
                                    //mode: 0
                                  }
                    },
                    opus: {
                        stereo: 1
                    },
                    userData: {param1:"value1"},
                    debug: true,
                    publisher: true,
                    recording: {
                        mimeType: "video/webm",
                        codec: "VP9",
                        name: "recording1"
                    },
                    exactVideoSize: false,
                    pcConstraints: {
                        //optional: [
                            //{ googIPv6: true },
                            //{ googImprovedWifiBwe: true },
                            //{ googDscp: true },
                            //{ googSuspendBelowMinBitrate: true },
                           // { googScreencastMinBitrate:300 },
                            //{ googCombinedAudioVideoBwe: true },
                            //turn off resolution changes
                            //{ googCpuOveruseDetection: false },
                           // { googCpuOveruseEncodeUsage: false },
                            //{ googCpuUnderuseThreshold: 55 },
                            //{ googCpuOverUseThreshold: 85 }
                        //]
                    },
                    exactVideoSize: false,
                     toggleScreen: true,
                    settings: true
                    //screen: true
                }
        }

            
    });
    player.on("ready", function() {
            player.on("devices", (event, devicesMap, deviceInfos) => {
                //get available devices here for building your own UI
                console.log("devices", devicesMap, deviceInfos);
                

            }).on("mediastart", (e, deviceInfo) => {
                //console.log("mediastart111", deviceInfo);
                //selected device has been activated
                console.log("Device Start ", deviceInfo);

            }).on("mediastop", (e) => {
                console.log("Device Stop");
            }).on("startpublish", (e, supportsParams) => {
                console.log("Publishing Started");
                //use this for enabling / disabling bandwidth select menus while publishing
                //browsers that don't support it cannot update bitrate controls while publishing
                //console.log("Browser Supports peer setParameters ", supportsParams);
            }).on("stoppublish", (e) => {
                console.log("Publishing Stopped");
            }).on("recordstart", (e) => {
                console.log("Recording Started");
            }).on("recordstop", (e) => {
                console.log("Recording Stopped");
            }).on("sendmessage", (e, message) => {
                console.log("Signal Server Send Message: ", message);
            }).on("gotmessage", (e, message) => {
                console.log("Signal Server Receive Message: ", message);
            }).on("offer", (e, offer) => {
               // console.log("WebRTC Offer ", offer.sdp);
            }).on("answer", (e, answer) => {
               // console.log("WebRTC Answer ", answer.sdp);
            }).on("bitratechanged", (e, params) => {
                console.log("WebRTC Bitrate Changed ", params);
            }).on("outputsuccess", (e, sinkId) => {
                console.log(`Success, audio output device attached: ${sinkId}`);
            }).on("outputerror", (e, message) => {
                console.log(message);
            }).on("ready", function(e, api, video) {
                console.log("READY", video);
            });
    });
    

Supported Codecs

  • H264, VP8, VP9
  • Opus audio codec.

Supported Servers

  • Wowza Streaming Server
  • Kurento Streaming Server
  • Millicast
  • Ant Media Server

Configuration

To enable a player as a publisher set the player needs the `publisher` config enabled.

Configuring stream name and server application

return with the configured options.

 {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }

Publisher Options

property / datatype default description
publisher
boolean
false Enable the player as the publisher. Which enables the full WebRTC encoder publishing application.
serverURL
string
The host of the websocket service. For Wowza the url is configured from the host.
userData
object
Custom data to send to the Wowza server
server
string
wowza, kurento, millicast, wowza-conference, kurento-conference, millicast-conference The server type. For conferencing, configure with the postfix -conference.
debug
boolean
false Turn on internal peering logs.
prefferedCodecs
object

{
    "audio": {
        codec: "opus",
    },
    "video": {
        codec: "H264",
        //codec: "VP9",
        level: "42e01f"
    }
}

Filter the codec used for audio and video. With a H264 level set for H264 codecs. This is required for Chrome to work with Wowza.
opus
object
{ stereo: 1 }, Opus codec configs
ui
object

{
    audioInput: "audioSource",
    audioOutput: "audioOutput",
    videoSource: "videoSource",
    bandwidth: "bandwidth",
    resolution: "resolution",
    rates: "rates",
    container: "#encoder-menu"
}

CSS selectors for managing external select menus with data and control of WebRTC and media options. Disable this if using external controls with the device data provided.
dataCallback
function
To manage the encoder controls externally, the device data is provided in this callback to build the device menus. And return with the configured options.

function(devicesMap, deviceInfos, supportsAudioSink) {
                        return new Promise((resolve) => {
                            resolve({
                                videoFrameRate: 30,
                                videoResolution: {
                                    width: 1280,
                                    height: 720
                                },
                                videoSource: "743094318ec58aa8dab2120cd9d85f6fc309261a15c93b8a00c7e2339a375714",
                                //audioInput: "deviceid"
                            });
                        });
                    }

autoStartDevice
boolean
false Enable to auto activate the device on initialisation. Or turned on when clicking the player.
maxBitrateBitrate
number
2000 Configure an initial max bitrate bitrate. This is required for Chrome to function with Wowza.
minVideoBitrate
number
300 Minimum video bitrate. Used on Chrome only.
startVideoBitrate
number
300 Starting video bitrate. Used on Chrome only.
audioVideoBitrate
number
64 Audio codec bitrate.
recording
object
Enable recording of the local stream with a download option. Config requires the filename for the download and the mimetype and codec. For H264 use video/mp4 and H264 as the codec.

{
    mimeType: "video/webm",
    codec: "VP9",
    name: "recording1"
}

pcConstraints
object
Set browser specific Peer connection options.

{
    optional: [
        { googIPv6: true },
        { googImprovedWifiBwe: true },
        { googDscp: true },
        { googSuspendBelowMinBitrate: true },
        { googScreencastMinBitrate:300 },
        { googCombinedAudioVideoBwe: true },
        //turn off resolution changes
        { googCpuOveruseDetection: false },
        { googCpuOveruseEncodeUsage: false },
        { googCpuUnderuseThreshold: 55 },
        { googCpuOverUseThreshold: 85 }
    ]
}

toggleScreen
boolean
false Enable camera / share screen switching features
settings
boolean
false Enable the WebRTC device settings as a settings overlay

WebRTC Group Call Conferencing Options

property / datatype default description
roomName
string
Required for Kurento group call conferencing.
conferenceContainer
string
The container selector to host the remote participant's stream in a player.
hlsTemplate
object
https://{serverURL}/{appName}/{streamName}/playlist.m3u8 When using Wowza. The HLS url template to use when subscribing in a group call as HLS.
conferenceSubscribeHls
boolean
false Set to true to subscribe to remote participants in a group call as HLS instead of WebRTC.
peerSubscribe
boolean
false Required for Kurento group call conferencing.

Subscriber Options

property / datatype default description
serverURL
string
The host of the websocket service. For Wowza the url is configured from the host.
userData
object
Custom data to send to the Wowza server
server
string
wowza The server type. For now Wowza is supported with other signalling servers after.
debug
boolean
false Turn on internal peering logs.

Millicast Options

Configuration example



                millicast: {
                        accountId: "",
                        publishToken: ""
                }


property / datatype default description
accountId
string
Millicast account id shown in the api key section.
publishToken
string
The publishing token created in the api section. This is used for generating the signal server url unless supplied using serverURL
subToken
string
Subscriber token generated by the backend. This is used for generating the signal server url with authenticated subscribing enabled. Unless supplied using serverURL

Virtual Background Removal Plugin Options

property / datatype default description
renderType
string
bodypix The background rendering type. Either bodypix, or chromakey
backgroundType
string
green If Chromakey rendering is set. Choose from either Green Screen colour detection or auto for solid colour backgrounds.
backgroundColor
array
[0, 255, 0] Specify the background colour in an rgb value array.
accurate
boolean
false Use the fast or slow Bodypix architecture config. Slow is more accurate but more intensive.
internalResolution
string
medium The resolution for segmenting the detected body from the background. Medium or high.
segmentationThreshold
number
0.5 An accuracy threshold for segmentation
segmentationThreshold
number
0.5 An accuracy threshold for segmentation
fast
object
{ architecture: 'MobileNetV1', outputStride: 16, multiplier: 0.75, quantBytes: 2 } The fast least accurate Bodypix config
slow
object
{ architecture: 'ResNet50', outputStride: 16, multiplier: 1, quantBytes: 2 }, The slow more accurate Bodypix config

Wowza WebRTC Configuration

Provided is a set of example configs for wowza with install script to get the WebRTC signal server working. Provided is a script to convert PEM SSL certificates into JKS format for Wowza.

An SSL certificate is required to be configured into the Secure SSL provider config inside VHost.xml.

The Websocket provider also needs to be configured inside VHost.xml

Configuration example


 
                      
        com.wowza.wms.webrtc.http.HTTPWebRTCExchangeSessionInfo
       
                      
        *webrtc-session.json
       
                      
        none
       
                    
      

Wowza WebRTC Conferencing Configuration

To enable the conferencing examples. The supplied WebRTCCustomSignalServer Wowza IDE project, needs to be imported and built to be configured and loaded.

This provides the custom WebRTC signal server for WebRTC publishing with token protection. Another WebSocket provider is for group call signal messaging and to notify when a WebRTC publish begins.



            
        org.electroteque.webrtc.HTTPWebRTCExchangeSessionInfoCustom
       
            
        *webrtc-session.json
       
            
        none
       
          
      
          
            
        org.electroteque.webrtc.HTTPProviderGroupCallWebSocket
       
            
        *webrtc-room-session.json
       
            
        none
       
          
      

Kurento WebRTC Configuration

Kurento Media Server comes in two parts. The Kurento Media Server and a java / node JSONRPC client WebSocket server.

Provided is a one to many application aswell as a group call application to run the group call example.

The one to many application endpoint is "one2many"

The group call application endpoint is "groupcall"

Start the Kurento server first with


export MAKEFLAGS="-j$(nproc)"
./bin/kms-build-run.sh

Start the Kurento client Websocket server with


mvn -U clean spring-boot:run -Dkms.url=wss://localhost:8433/kurento

JavaScript API

Methods

The api methods are exposed on the Flowplayer 7 api.



    const ret = player.setStream(stream);

    const ret = player.setVideoStream(stream);

    const ret = player.toggleDevices();

    const ret = player.togglePublish();

    const ret =  player.toggleAudio();

    const ret = player.toggleVideo();

    const ret = player.toggleScreenShare();

    player.send(data);

    player.sendSignalMessage(message);

    player.setAudioGain(gain);

    const devices = player.getDevices();

    player.getDevice(constraint = null, videoIndex = -1, audioIndex = -1).then((device) => {});
method returns description
setStream Switch to a new MediaStream from another source that isn't part of the Camera device list.
setVideoStream Switch the video track of a new MediaStream from another source that isn't part of the Camera device list. This keeps the original audio track.
toggleDevices Toggle enabling and playing the selected Camera / Microphone device.
togglePublish Toggle publishing a WebRTC stream.
toggleAudio Toggle the device audio track.
toggleVideo Toggle the device video track.
toggleScreenShare Toggle screen sharing.
send Send a message over the WebRTC peering data channel.
sendSignalMessage Send a message to the WebRTC signal server.
audioGain Set the input audio gain of the microphone device.
getDevices Get a map of the detected devices.
getDevice Obtain the MediaStream of a new WebRTC device. Provide a constraint , or the index of the video and audio device provided in the devices map.

Subscriber Example

This example demonstrates setting up the WebRTC subscriber player.




(function(jwplayer) {

    var player2 = jwplayer("subscriber").setup({
        width: "100%",
        aspectratio: "16:9",
        

        playlist: [
            {
                sources: [
                    {type: "mp4", file: "myStream", appName: "webrtc", publisher: false, live: true },
                    {type: "application/x-mpegurl", src: "http://rtc.electroteque.org:1935/webrtc/myStream_360p/playlist.m3u8"}
                ]
            }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcsubscriber-8.0.0.js": {
                    server: "wowza",
                    serverURL: "rtc.electroteque.org",
                    //serverURL: "in-staging.videolinq.net",
                    debug: true
            }
        }

            
    });
    player2.on("ready", function() {
            player2.on("sendmessage", (e,api, message) => {
                console.log("Signal Server Send Message: ", message);
            }).on("gotmessage", (e, api, message) => {
                console.log("Signal Server Receive Message: ", message);
            }).on("offer", (e, api, offer) => {
                console.log("WebRTC Offer ", offer.sdp);
            }).on("answer", (e, api, answer) => {
                console.log("WebRTC Answer ", answer.sdp);
            }).on("resize", (e, api, size) => {
                console.log("Receiving Video Size ", size);
            });
    });


  

})(window.jwplayer);

Virtual Background Removal Example

This example demonstrates dynamic virtual background removal using Bodypix and Tensorflow AI.

Backgrounds can be applied using CSS on the publisher and subscriber. The video element needs to be shown or rendering stops.




(function(jwplayer) {

    player = jwplayer("virtual").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    serverURL: "rtc.electroteque.org",
                    applicationName: "webrtc",
                    server: "wowza",
                    publisher: true
                },

                "https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {

                },
                "https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {

                }
        },
        


            
    });

})(window.jwplayer);

Virtual Chromakey Green Screen Removal Example

This example demonstrates dynamic virtual background removal using Green Screen Chromakey.

If set to auto render type any solid background color can be used or a color can be chosen.




(function(jwplayer) {

    player = jwplayer("virtual").setup({
        width: "100%",
        aspectratio: "16:9",
        live: true,
        playlist: [
            {type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
        ],
        plugins: {
            "https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
                    serverURL: "rtc.electroteque.org",
                    applicationName: "webrtc",
                    server: "wowza",
                    publisher: true
                },

                "https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {

                },
                "https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {

                }
        },
        


            
    });

})(window.jwplayer);