Sorry, you need to enable JavaScript to visit this website.

Immersive Video Sample on the Open Visual Cloud

BY Andrew Zhang ON Jan 16, 2020
The Open Visual Cloud project includes a diverse span of graphics-intensive technologies and applications. The latest Open Visual Cloud Immersive Video Sample demonstrates a software stack streaming 360 video content in UHD, using the open source software projects: SVT-HEVC, Open WebRTC Toolkit, FFmpeg, and OMAF Compliant Libraries to build end-to-end 360 video streaming services supporting 4K and 8K video. The sample can improve visual fidelity of 360 content with 4K and 8K video and lower the requirements on network bandwidth for delivery, and client devices’ decoder capability for consumption.
 
Developers can use the provided Dockerfiles to quickly load the Immersive Video Sample onto specific Intel® Xeon® Scalable Processors servers set up with well-working configurations.  By running a few lines of code, a developer can set up a container with all the open source pieces loaded and configured. Once built, the Immersive Video Sample contains end-to-end software pipelines and allows developers to immediately start developing their own 360 video streaming solution using either OMAF standard (MPEG-DASH) or low-latency WebRTC standard.
The Immersive Video Sample's code and documentation can be downloaded from the Open Visual Cloud Github repository: https://github.com/OpenVisualCloud/Immersive-Video-Sample
 
This project includes two different streaming frameworks. The OMAF version delivers tiled 360 video stream for both VOD and live streaming mode. The WebRTC version provides low latency tiled 360 streaming.
 
Tile-based Encoding is the key component in both OMAF server and WebRTC sever. Server uses SVT-HEVC to encode the 360 content. SVT-HEVC supports MCTS which is a “must have” feature for tiled 360 video streaming, as the encoder library in the whole solution. After tile-based encoding, OMAF server uses OMAF Packing library to pack the bitstream into mp4 segmentations which are compatible with OMAF DASH standard; WebRTC server packetizes the bitstream as RTP for low latency tiled streaming. 
 
Three components are provided for the client side:
 
  • OMAF Dash Access Library: This library provides basic functions to parse mpd file, select extractor track based on current viewport, download relative tile segments, parse the extractor and tile segments, and then output a standard HEVC frame with RWPK information for decoding.
  • WebRTC Linux Client Library: This library establishes a WebRTC connection with the OWT media server, and retrieves the tile-based FoV RTP stream. It supports dynamically switching the viewport, by sending a feedback message to the server to transfer different tiles in the RTP stream. The received RTP stream is depacketized as a standard HEVC frame with RWPK SEI for decoding. 
  • Reference OMAF player: This reference player gets an aggregated video bit stream based on OMAF Dash Access library or WebRTC Linux Client library, decodes the bit stream with or without HW acceleration, and then renders the picture to a sphere or cube mesh based on RWPK of each tile in the frame.
 
There is also a common library used to do basic process such as:
 
  • Provide a unified interface for tile-based HEVC bitstream processing and viewport-based content processing;
  • Support HEVC bitstream processing: VPS/SPS/PPS parsing and generating, 360 video relative SEI generating and parsing, HEVC tile-based bitstream aggregation;
  • Support viewport generation, viewport-based tile selection and extractor selection based on content coverage;
  • FFMPEG Plugins in this solution provide a quick way for user to try this tiled 360 video streaming solution using common FFmpeg command line.

To get started building the Immersive Video Sample and to set up the developer environment, check out these two tutorials:

 
Want to take a deep dive into the software architecture? Check out these links in the Github repo: