​​It’s been almost a year since we published our streaming video architecture for Super Bowl LIV in 2020. As the ball gets handed to us for the next Super Bowl, we thought it might be ideal timing to share some of the improvements being planned for what we are calling our “Season of Champions”, including the following marquee events:

Busy times ahead for sure!

In preparation for these major events, our team is starting early and is already lining up work for our engineering teams to build out the video workflows and players to support these large scale and high profile events.

This article will cover the following topics:

For Super Bowl LIV in 2020, we had built a highly redundant transmission pipeline. Each incoming contribution signal was an RTP-FEC feed with video at 720p60 20 Mbps, AVC. This time around, we’re looking to utilize the state of the art infrastructure from our broadcast facility to upgrade the feed to a 1080p60 HDR source. We’re exploring the following open questions for the contribution feed:

To deliver an optimal end user experience, we need to optimize the adaptive bitrate ladder for a wide variety of devices, while keeping CDN bandwidth utilization in mind. Other considerations include:

We have built our video workflow using infrastructure as code and are currently expanding our automation suite to spin up and tear down entire video workflows as described in the diagram below:

A key initiative is to better understand our end user Quality of Experience (QoE). We use several industry standard player QoE tools to understand our user engagement and QoE. We combine player QoE tools with CDN logs and synthetic monitoring tools to successfully operate large scale live events.

Today when alerts occur, it is cumbersome to debug and understand which part of the ecosystem is the root cause of the issue. For instance, if you are an end user watching an NFL game on Sunday afternoon and experience rebuffing, is it due to slow origin responses? Or caching inefficiencies at the CDN? Or poor buffer management in players?

We typically look at player QoE tools to diagnose the issue and then look at CDN logs to further validate, if required. Hence debugging a single session involves logging into multiple tools and correlating the data manually. This is where CMCD comes into the mix.

The Common Media Client Data (CMCD) is a specification developed by the CTA-Wave. CMCD provides a ‘consolidated’ view of metrics as observed from both the clients (players) and CDNs. Players report a standardized set of QoE tracking data for every CDN request. This tracking data can subsequently be captured and viewed in CDN logs. Players can send this tracking data either as HTTP headers or query string params on CDN object requests.

The CMCD specification can be found here.

This will give us the ability to visualize individual sessions, triage problems and perform mitigation steps.

For Super Bowl LIV, we built a backend service for CDN decisioning with the decision made at session start. Additional data vectors under consideration:

Another key initiative is striving for consistency of QoE implementation across our various players, enabling focus on iterative improvement to overall quality.Our player teams are currently working on:

Consistent and More Verbose Error Reporting:

We are also working on improving our player QoE implementation to add more verbose and structured error reporting. Instead of reporting generic “Playback failed” errors, we will be adding descriptive error messages explaining the reason why a video start failure or a video playback failure occurred

Consistent Retry Logic Across Players:

Retry logic in players can be tricky. Important considerations are:

We’re working on smart algorithms for doing reties without elevating rebuffering or video playback failures.

Overall, lots of interesting challenges ahead for the team. And by the way, if you’re interested in joining us to help build and solve some of the challenges described above… we’re hiring 😀

Be on the lookout for more from our Road To Super Bowl 2023 series and follow us on Medium so you’re the first to know when they’re available!

Make Your Mark Here. At FOX, we pride ourselves in shaking things up and making things happen. We’re a community of builders, operators and innovators and each and every day we experiment, collaborate, and co-create to develop the next world of news, sports & entertainment streaming technology.

While being one of the most well-known brands in the world, we provide our employees with the culture of a start-up — fast paced, non-hierarchical, full of smart ideas & innovation and most importantly, the knowledge that each member of the team is making a difference in defining what’s next for FOX Tech. Simply put, we love to do great work, with great people.

Learn more and join our team: https://tech.fox.com

We’ve always been a little bit different. And we like it that way. Today, it’s all about the “Now”​ at FOX as we are focused on delivering LIVE viewing, plain and simple. Each and every day, we experiment, collaborate, and co-create. Learn more: https://tech.fox.com

WHAT THE FI. BY POWEREDLOCAL

JEN-LI CHEN IN DATA SCIENCE