WebRTC Video Streaming Client

Video streaming from IP cameras, DVRs and NVRs is a very popular Nabto Edge use case. And using Nabto Edge WebRTC is the modern and preferred approach for video streaming.

Client Library for Signaling

The Nabto Edge WebRTC Signaling Protocol defines how signaling messages are exchanged on a Nabto connection between a client (e.g. a mobile app) and a device (e.g. a camera) to establish a WebRTC connection.

Libraries are provided for iOS, Android and web based applications that implement this protocol.

The following shows how the EdgeWebRTC library is used to create a WebRTC connection and then request streams using Nabto Edge CoAP. In mobile apps, the WebRTC connection is created using a Nabto Edge connection for signaling. In web based applications, signaling is relayed through the Nabto Edge Basestation and a Nabto Edge Connection is not used.

func openVideoStream(_ nabtoConnection: Connection) throws {
    self.webrtcConnection = EdgeWebRTC.createPeerConnection(nabtoConnection)

    self.webrtcConnection.onConnected = {
        do {
            requestTracks(connection: nabtoConnection)
        } catch {
            print("Failed requesting tracks from device (\(error)")
        }
    }

    self.webrtcConnection.onTrack = { track in
        if let track = track as? EdgeVideoTrack {
            self.remoteTrack = track
            self.remoteTrack.add(self.videoView)
        }
    }

    self.webrtcConnection.onError = { error in
        // TODO: Handle or log errors here
    }

    try await self.webrtcConnection.connect()
}

func requestTracks(connection: Connection) throws {
    let trackInfo = """
        {"tracks": ["frontdoor-video", "frontdoor-audio"]}
        """
    let coap = try connection.createCoapRequest(method: "POST", path: "/webrtc/tracks")
    try coap.setRequestPayload(contentFormat: 50, data: trackInfo.data(using: .utf8)!)
    let coapResult = try coap.execute()
    if coapResult.status != 201 {
        // TODO: Show an error in UI
        print("Failed getting track info from device (coap code: \(coapResult.status)")
    }
}

fun openVideoStream(nabtoConnection: Connection) {
    this.peerConnection = EdgeWebRTCManager.getInstance().createRTCConnection(nabtoConnection)
    val connRef = WeakReference(nabtoConnection)

    peerConnection.onConnected {
        Log.i(tag, "Connected to peer")
        val trackInfo = """{"tracks": ["frontdoor-video", "frontdoor-audio"]}"""

        val coap = connRef.get()?.createCoap("POST", "/webrtc/tracks")
        coap?.setRequestPayload(50, trackInfo.toByteArray())
        coap?.execute()

        if (coap?.responseStatusCode != 201) {
            // TODO: Show an error in UI
            Log.e("MyTag", "Failed getting track info from device (coap code: ${coap?.responseStatusCode}")
        }
    }

    peerConnection.onTrack { track ->
        Log.i("MyTag", "Track of type ${track.type}")
        if (track.type == EdgeMediaTrackType.VIDEO) {
            remoteTrack = track as EdgeVideoTrack
            videoView?.let { remoteTrack.add(it) }
        }
    }

    peerConnection.onError { error ->
        // TODO: Handle or log error
    }

    peerConnection.connect()
}

async function openVideoStream() {
  webrtcConnection = EdgeWebrtc.createEdgeWebrtcConnection();
  webrtcConnection.setConnectionOptions(device);

  webrtcConnection.onTrack((event, trackId) => {
    let video = document.getElementById("received_video");
    video.srcObject = event.streams[0];
  });

  await webrtcConnection.connect();

  const payload = JSON.stringify({ tracks: ["frontdoor-video", "frontdoor-audio"] });
  await webrtcConnection.coapInvoke("POST",
                                    "/webrtc/tracks",
                                    CoapContentFormat.APPLICATION_JSON,
                                    Buffer.from(payload));
}

For Swift and Kotlin, the connection (e.g. as passed to createPeerConnection) could have been opened as described in the Connecting guide with authentication based on the Nabto Edge decentral access control paradigm. In WebRTC scenarios, the connections could typically also have been authenticated using centralized access control using OAuth tokens. Any authentication done on the Nabto Edge Connection will be inherited by the WebRTC connection. For web based applications, the Nabto Edge Basestation should not be authorized to perform any actions other than signaling, so authentication must be done on the WebRTC connection after it is established (eg. using session authentication).

Once the WebRTC connection is established, two tracks are requested using a CoAP request on the Nabto connection. These are application specific endpoints, defined in the Nabto Edge WebRTC application running on the camera.

In the snippet above, two track ids frontdoor-video and frontdoor-audio are assumed to exist on the target. In a real application, the WebRTC device application could be queried using additional application specific CoAP endpoints.

Once a video track is received from the other peer using the Nabto Edge WebRTC library, the information is passed on to the player that in turn starts playing the video stream. For simplicity, checks have been omitted - the player should only show the video feed, if the received track matches the requested track; the other WebRTC peer could in principle announce tracks that were not requested.

Summary of responsibilities: The EdgeWebRTC library establishes a WebRTC connection through using Nabto Edge Signaling. Additionally, tracks are requested using CoAP requests - this happens at the application level, ie. it is up to the solution to define the CoAP endpoints for this in the device and invoke them from the client using standard Nabto Edge CoAP calls. These tracks are then served by the other peer on the WebRTC connection when/if available.

WebRTC Player Component on iOS

As part of the Nabto Edge WebRTC library for iOS, integration to Google’s WebRTC library (the same underlying WebRTC library as used in Chromium) is provided through EdgeMetalVideoView. Embed it into a UIView as follows:

override func viewDidLoad() {
    super.viewDidLoad()

    self.videoView = EdgeMetalVideoView(frame: self.videoScreenView.frame)
    self.videoView.videoContentMode = .scaleAspectFit
    self.videoView.embed(into: self.videoScreenView)

    do {
        try self.openVideoStream()
    } catch {
        self.showDeviceErrorMsg("Could not start an RTC connection to devie \(error)")
    }
}

WebRTC Player Component on Android

As part of the Nabto Edge WebRTC library for Android, integration to the Google WebRTC library (the same WebRTC library as used in Chromium) is provided through an Android View. It can be used as follows:

<com.nabto.edge.client.webrtc.EdgeVideoView
  android:id="@+id/myVideoView"
  android:layout_width="match_parent"
  android:layout_height="match_parent"/>

This view must be initialized in code, for example in the onViewCreated lifecycle method of the Fragment that contains the video view:

Java:

  EdgeVideoView myVideoView = view.findViewById<EdgeVideoView>(R.id.myVideoView)
  EdgeWebRTCManager.getInstance().initVideoView(myVideoView)

Kotlin:

  val myVideoView = view.findViewById<EdgeVideoView>(R.id.myVideoView)
  EdgeWebRTCManager.getInstance().initVideoView(myVideoView)

First party support for Jetpack Compose is planned. Currently you may use the AndroidView composable if you need Jetpack Compose support immediately.