Video improvements for poor network conditions

[clone1018]

We'll be focused on making improvements that help viewers have a reliable experience viewing streams on Glimesh, even in poorer network conditions. One of the main objectives will be supporting the Glimesh Mobile App with the ability to view streams, but the exact implementation is still to be determined.

This is a roadmap epic to capture work involving improving video for poor network conditions.

Examples of potential work:

• NACKs
• HLS
• RTMP
• etc

This feature is still being groomed and could change frequently.

[danstiner]

My rough plan of what I'm going to look at:

• Re-enable NACKs Fix NACKs to more gracefully handle RTP sequence # rollover #95 🔙 Improved NACKs #135
• Add playout delay 🕑 Support playout delay extension #134 Enable playout-delay extension + NACKs + janus version bumps ops#41
  • This should workaround the issues seen in Large keyframes can result in consistent freezes on every keyframe #101
• Email webrtc list if playout delay works and work with them to get a real fix in for the issue
• Capture additional metrics for debugging and tuning (or eventually removing) the playout delay
• ftl-sdk traffic shaping improvements for OBS (or move ingest to RTMP/SRT/WebRTC/etc that have already solved this)

Other things not listed that might also help:

• Transcoding
• WebRTC in OBS
• Scalable video coding in ftl-sdk ( might be possible to send both 720p and 240p video layers, or at least temporal scaling might help with severe packet loss)
• Adaptive bitrate in ftl-sdk

[haydenmc]

I also would like to look at RTMP and HLS as a means of improving reliability at ingest and edge respectively.