WebRTC Low Latency Streaming Adds WebRTC features into JWPlayer 8
Introduction
The WebRTC is a browser based low latency peering streaming solution with only the requirement of a signal server.
While WebRTC can allow for full 2 way peering, the intended solution is for one to many low latency live broadasting with basic conferencing functions.
With Kurento, Millicast, Ant Media Server and Wowza Media Server conferencing solutions is provided however with Wowza the streams are pulled from the server as WebRTC or HLS. Demo java backend projects are provided.
With Millicast conferencing a demo node websocket service is provided to signal users when a publish stream begins.
The WebRTC plugin provides an integrated browser based encoder publishing application, with media device, resolution, codec, bitrate controls. And a subscriber player.
Media device selection can use whatever device the browser detects including video and audio inputs. Audio output control for the publisher is possible for browsers that support it.
Screensharing capabilities with browsers that support it are possible using screen sharing apis. The screen, browser, browser tab can be chosen , with an audio input for audio. This may be suitable for live broadcasting instruction videos or even gaming.
Switching between camera and screenshare video tracks is possible while peering. Retaining the original microphone audio stream.
Recording of the media device is possible using browser based media recorder api's for browsers that support it. With a download capability of the recorded data.
Device toggling can turn the video / audio device off before and while publishing.
Supported servers for now is Wowza, Kurento, Millicast WebRTC backends. Wowza only supports one to many peering and no data channels. Provided is a Kurento one to many signal server.
WebRTC settings can be configured as a player overlay,
Volume control of the audio input is made possible using the audio api. This also makes the possibility of an audio meter display
Virtual Meeting Background Removal
With an included virtual background plugin. Support for dynamic background removal rendering is provided.
This rendering view can be used as the main video stream on the publisher and will be automatically detected.
The auto background uses an AI BodyPix feature that doesn't require a Chromakey Green Screen render. It's not perfect but with fine tuned settings it's enough to make a transparent background to show a background image behind the video.
A Chromakey rendering option for Green Screens is supported with option to configure the flat background colour or choose an auto option for choosing any background of a solid colour.
Features
- Low latency streaming via the browser.
- One to many live streaming via WebRTC
- Group call conferencing as WebRTC or HLS.
- Group conferencing features for Wowza, Millicast and Kurento.
- Ability to feature a participant in a conference by an admin.
- Media device video and audio selection.
- Encoder application selections of bitrate, resolution, framerate, audio input, audio output and video sources.
- Support for external UI control if managed select menus is not suitabled.
- Recording of local video stream support with a download option.
- Toggling of audio and video device before and while publishing.
- Screensharing video capabilities using screensharing apis instead of camera and microphone.
- Screensharing toggling support between screen capture and camera. While peering the tracks are replaced in realtime. Audio is retained while screen capturing.
- Microphone Audio peak meter display option.
- Full peering events support with debug capabilities.
- Audio volume control of the microphone input and audio meter display functionality.
- Support for dynamic background removal for virtual meetings without Green Screens using Bodypix AI.
- Support for Chromakey Green Screen background removal rendering.
(Current Version 8.0.0)
Compatibility
- JWPlayer 8 and above
Demo
Contact for a site demo download.
Examples
Basic Example
configuration
(function(jwplayer) {
var player = jwplayer("player").setup({
width: "100%",
aspectratio: "16:9",
live: true,
playlist: [
{type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
],
plugins: {
"https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
ui: {
audioInput: "audioSource",
audioOutput: "audioOutput",
videoSource: "videoSource",
bandwidth: "bandwidth",
resolution: "resolution",
rates: "rates",
container: "#encoder-menu"
},
/*dataCallback: function(devicesMap, deviceInfos, supportsAudioSink) {
return new Promise((resolve) => {
resolve({
videoFrameRate: 30,
videoResolution: {
width: 1280,
height: 720
},
videoSource: "743094318ec58aa8dab2120cd9d85f6fc309261a15c93b8a00c7e2339a375714",
//audioInput: "deviceid"
});
});
},*/
autoStartDevice: false,
serverURL: "rtc.electroteque.org",
//serverURL: "in-staging.videolinq.net",
applicationName: "webrtc",
//streamName: "myStream",
server: "wowza",
maxVideoBitrate: 2000,
minVideoBitrate: 2000,
startVideoBitrate: 300,
audioBitrate: 64,
videoFrameRate: 30,
preferredCodecs: {
"audio": {
codec: "opus",
},
"video": {
codec: "H264",
//codec: "VP9",
level: "42e01f",
//mode: 0
}
},
opus: {
stereo: 1
},
userData: {param1:"value1"},
debug: true,
publisher: true,
recording: {
mimeType: "video/webm",
codec: "VP9",
name: "recording1"
},
exactVideoSize: false,
pcConstraints: {
//optional: [
//{ googIPv6: true },
//{ googImprovedWifiBwe: true },
//{ googDscp: true },
//{ googSuspendBelowMinBitrate: true },
// { googScreencastMinBitrate:300 },
//{ googCombinedAudioVideoBwe: true },
//turn off resolution changes
//{ googCpuOveruseDetection: false },
// { googCpuOveruseEncodeUsage: false },
//{ googCpuUnderuseThreshold: 55 },
//{ googCpuOverUseThreshold: 85 }
//]
},
exactVideoSize: false,
toggleScreen: true,
settings: true
//screen: true
}
}
});
player.on("ready", function() {
player.on("devices", (event, devicesMap, deviceInfos) => {
//get available devices here for building your own UI
console.log("devices", devicesMap, deviceInfos);
}).on("mediastart", (e, deviceInfo) => {
//console.log("mediastart111", deviceInfo);
//selected device has been activated
console.log("Device Start ", deviceInfo);
}).on("mediastop", (e) => {
console.log("Device Stop");
}).on("startpublish", (e, supportsParams) => {
console.log("Publishing Started");
//use this for enabling / disabling bandwidth select menus while publishing
//browsers that don't support it cannot update bitrate controls while publishing
//console.log("Browser Supports peer setParameters ", supportsParams);
}).on("stoppublish", (e) => {
console.log("Publishing Stopped");
}).on("recordstart", (e) => {
console.log("Recording Started");
}).on("recordstop", (e) => {
console.log("Recording Stopped");
}).on("sendmessage", (e, message) => {
console.log("Signal Server Send Message: ", message);
}).on("gotmessage", (e, message) => {
console.log("Signal Server Receive Message: ", message);
}).on("offer", (e, offer) => {
// console.log("WebRTC Offer ", offer.sdp);
}).on("answer", (e, answer) => {
// console.log("WebRTC Answer ", answer.sdp);
}).on("bitratechanged", (e, params) => {
console.log("WebRTC Bitrate Changed ", params);
}).on("outputsuccess", (e, sinkId) => {
console.log(`Success, audio output device attached: ${sinkId}`);
}).on("outputerror", (e, message) => {
console.log(message);
}).on("ready", function(e, api, video) {
console.log("READY", video);
});
});
Supported Codecs
- H264, VP8, VP9
- Opus audio codec.
Supported Servers
- Wowza Streaming Server
- Kurento Streaming Server
- Millicast
- Ant Media Server
Configuration
To enable a player as a publisher set the player needs the `publisher` config enabled.
Configuring stream name and server application
return with the configured options.
{type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
Publisher Options
property / datatype | default | description |
---|---|---|
publisher
boolean
|
false | Enable the player as the publisher. Which enables the full WebRTC encoder publishing application. |
serverURL
string
|
The host of the websocket service. For Wowza the url is configured from the host. | |
userData
object
|
Custom data to send to the Wowza server | |
server
string
|
wowza, kurento, millicast, wowza-conference, kurento-conference, millicast-conference | The server type. For conferencing, configure with the postfix -conference. |
debug
boolean
|
false | Turn on internal peering logs. |
prefferedCodecs
object
|
|
Filter the codec used for audio and video. With a H264 level set for H264 codecs. This is required for Chrome to work with Wowza. |
opus
object
|
{ stereo: 1 }, | Opus codec configs |
ui
object
|
|
CSS selectors for managing external select menus with data and control of WebRTC and media options. Disable this if using external controls with the device data provided. |
dataCallback
function
|
To manage the encoder controls externally, the device data is provided in this callback to build the device menus. And return with the configured options.
|
|
autoStartDevice
boolean
|
false | Enable to auto activate the device on initialisation. Or turned on when clicking the player. |
maxBitrateBitrate
number
|
2000 | Configure an initial max bitrate bitrate. This is required for Chrome to function with Wowza. |
minVideoBitrate
number
|
300 | Minimum video bitrate. Used on Chrome only. |
startVideoBitrate
number
|
300 | Starting video bitrate. Used on Chrome only. |
audioVideoBitrate
number
|
64 | Audio codec bitrate. |
recording
object
|
Enable recording of the local stream with a download option. Config requires the filename for the download and the mimetype and codec. For H264 use video/mp4 and H264 as the codec.
|
|
pcConstraints
object
|
Set browser specific Peer connection options.
|
|
toggleScreen
boolean
|
false | Enable camera / share screen switching features |
settings
boolean
|
false | Enable the WebRTC device settings as a settings overlay |
WebRTC Group Call Conferencing Options
property / datatype | default | description |
---|---|---|
roomName
string
|
Required for Kurento group call conferencing. | |
conferenceContainer
string
|
The container selector to host the remote participant's stream in a player. | |
hlsTemplate
object
|
https://{serverURL}/{appName}/{streamName}/playlist.m3u8 | When using Wowza. The HLS url template to use when subscribing in a group call as HLS. |
conferenceSubscribeHls
boolean
|
false | Set to true to subscribe to remote participants in a group call as HLS instead of WebRTC. |
peerSubscribe
boolean
|
false | Required for Kurento group call conferencing. |
Subscriber Options
property / datatype | default | description |
---|---|---|
serverURL
string
|
The host of the websocket service. For Wowza the url is configured from the host. | |
userData
object
|
Custom data to send to the Wowza server | |
server
string
|
wowza | The server type. For now Wowza is supported with other signalling servers after. |
debug
boolean
|
false | Turn on internal peering logs. |
Millicast Options
Configuration example
millicast: {
accountId: "",
publishToken: ""
}
property / datatype | default | description |
---|---|---|
accountId
string
|
Millicast account id shown in the api key section. | |
publishToken
string
|
The publishing token created in the api section. This is used for generating the signal server url unless supplied using serverURL | |
subToken
string
|
Subscriber token generated by the backend. This is used for generating the signal server url with authenticated subscribing enabled. Unless supplied using serverURL |
Virtual Background Removal Plugin Options
property / datatype | default | description |
---|---|---|
renderType
string
|
bodypix | The background rendering type. Either bodypix, or chromakey |
backgroundType
string
|
green | If Chromakey rendering is set. Choose from either Green Screen colour detection or auto for solid colour backgrounds. |
backgroundColor
array
|
[0, 255, 0] | Specify the background colour in an rgb value array. |
accurate
boolean
|
false | Use the fast or slow Bodypix architecture config. Slow is more accurate but more intensive. |
internalResolution
string
|
medium | The resolution for segmenting the detected body from the background. Medium or high. |
segmentationThreshold
number
|
0.5 | An accuracy threshold for segmentation |
segmentationThreshold
number
|
0.5 | An accuracy threshold for segmentation |
fast
object
|
{ architecture: 'MobileNetV1', outputStride: 16, multiplier: 0.75, quantBytes: 2 } | The fast least accurate Bodypix config |
slow
object
|
{ architecture: 'ResNet50', outputStride: 16, multiplier: 1, quantBytes: 2 }, | The slow more accurate Bodypix config |
Wowza WebRTC Configuration
Provided is a set of example configs for wowza with install script to get the WebRTC signal server working. Provided is a script to convert PEM SSL certificates into JKS format for Wowza.
An SSL certificate is required to be configured into the Secure SSL provider config inside VHost.xml.
The Websocket provider also needs to be configured inside VHost.xml
Configuration example
com.wowza.wms.webrtc.http.HTTPWebRTCExchangeSessionInfo
*webrtc-session.json
none
Wowza WebRTC Conferencing Configuration
To enable the conferencing examples. The supplied WebRTCCustomSignalServer Wowza IDE project, needs to be imported and built to be configured and loaded.
This provides the custom WebRTC signal server for WebRTC publishing with token protection. Another WebSocket provider is for group call signal messaging and to notify when a WebRTC publish begins.
org.electroteque.webrtc.HTTPWebRTCExchangeSessionInfoCustom
*webrtc-session.json
none
org.electroteque.webrtc.HTTPProviderGroupCallWebSocket
*webrtc-room-session.json
none
Kurento WebRTC Configuration
Kurento Media Server comes in two parts. The Kurento Media Server and a java / node JSONRPC client WebSocket server.
Provided is a one to many application aswell as a group call application to run the group call example.
The one to many application endpoint is "one2many"
The group call application endpoint is "groupcall"
Start the Kurento server first with
export MAKEFLAGS="-j$(nproc)"
./bin/kms-build-run.sh
Start the Kurento client Websocket server with
mvn -U clean spring-boot:run -Dkms.url=wss://localhost:8433/kurento
JavaScript API
Methods
The api methods are exposed on the Flowplayer 7 api.
const ret = player.setStream(stream);
const ret = player.setVideoStream(stream);
const ret = player.toggleDevices();
const ret = player.togglePublish();
const ret = player.toggleAudio();
const ret = player.toggleVideo();
const ret = player.toggleScreenShare();
player.send(data);
player.sendSignalMessage(message);
player.setAudioGain(gain);
const devices = player.getDevices();
player.getDevice(constraint = null, videoIndex = -1, audioIndex = -1).then((device) => {});
method | returns | description |
---|---|---|
setStream | Switch to a new MediaStream from another source that isn't part of the Camera device list. | |
setVideoStream | Switch the video track of a new MediaStream from another source that isn't part of the Camera device list. This keeps the original audio track. | |
toggleDevices | Toggle enabling and playing the selected Camera / Microphone device. | |
togglePublish | Toggle publishing a WebRTC stream. | |
toggleAudio | Toggle the device audio track. | |
toggleVideo | Toggle the device video track. | |
toggleScreenShare | Toggle screen sharing. | |
send | Send a message over the WebRTC peering data channel. | |
sendSignalMessage | Send a message to the WebRTC signal server. | |
audioGain | Set the input audio gain of the microphone device. | |
getDevices | Get a map of the detected devices. | |
getDevice | Obtain the MediaStream of a new WebRTC device. Provide a constraint , or the index of the video and audio device provided in the devices map. |
Subscriber Example
This example demonstrates setting up the WebRTC subscriber player.
(function(jwplayer) {
var player2 = jwplayer("subscriber").setup({
width: "100%",
aspectratio: "16:9",
playlist: [
{
sources: [
{type: "mp4", file: "myStream", appName: "webrtc", publisher: false, live: true },
{type: "application/x-mpegurl", src: "http://rtc.electroteque.org:1935/webrtc/myStream_360p/playlist.m3u8"}
]
}
],
plugins: {
"https://static.electroteque.org/js/jw7/webrtcsubscriber-8.0.0.js": {
server: "wowza",
serverURL: "rtc.electroteque.org",
//serverURL: "in-staging.videolinq.net",
debug: true
}
}
});
player2.on("ready", function() {
player2.on("sendmessage", (e,api, message) => {
console.log("Signal Server Send Message: ", message);
}).on("gotmessage", (e, api, message) => {
console.log("Signal Server Receive Message: ", message);
}).on("offer", (e, api, offer) => {
console.log("WebRTC Offer ", offer.sdp);
}).on("answer", (e, api, answer) => {
console.log("WebRTC Answer ", answer.sdp);
}).on("resize", (e, api, size) => {
console.log("Receiving Video Size ", size);
});
});
})(window.jwplayer);
Virtual Background Removal Example
This example demonstrates dynamic virtual background removal using Bodypix and Tensorflow AI.
Backgrounds can be applied using CSS on the publisher and subscriber. The video element needs to be shown or rendering stops.
(function(jwplayer) {
player = jwplayer("virtual").setup({
width: "100%",
aspectratio: "16:9",
live: true,
playlist: [
{type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
],
plugins: {
"https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
serverURL: "rtc.electroteque.org",
applicationName: "webrtc",
server: "wowza",
publisher: true
},
"https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {
},
"https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {
}
},
});
})(window.jwplayer);
Virtual Chromakey Green Screen Removal Example
This example demonstrates dynamic virtual background removal using Green Screen Chromakey.
If set to auto render type any solid background color can be used or a color can be chosen.
(function(jwplayer) {
player = jwplayer("virtual").setup({
width: "100%",
aspectratio: "16:9",
live: true,
playlist: [
{type: "mp4", file: "myStream", appName: "webrtc", publisher: true, live: true }
],
plugins: {
"https://static.electroteque.org/js/jw7/webrtcpublisher-8.0.0.js": {
serverURL: "rtc.electroteque.org",
applicationName: "webrtc",
server: "wowza",
publisher: true
},
"https://static.electroteque.org/js/jw7/webrtcpeakmeter-8.0.0.js": {
},
"https://static.electroteque.org/js/jw7/virtual-background-8.0.0.js": {
}
},
});
})(window.jwplayer);