Skip to content

Latest commit

 

History

History
139 lines (107 loc) · 9.38 KB

explainer.md

File metadata and controls

139 lines (107 loc) · 9.38 KB

RTCRtpReceiver Decoder Fallback

Authors

Much of this explainer synthesizes and consolidates prior discussions and contributions from members of the WebRTC working group.

Participate

Introduction

Game streaming platforms like Xbox Cloud Gaming and Nvidia GeForce Now rely on hardware decoding in browsers to deliver low-latency, power efficient experiences. However, there is currently no reliable way for these applications to detect when decoding silently falls back to software during a stream.

This proposal introduces a runtime event to notify applications when a decoder fallback occurs. The goal is to give developers actionable visibility into runtime behavior without exposing new fingerprinting vectors or hardware details.

User-Facing Problem

End users of game streaming services may experience increased latency, degraded quality, and battery drain when the browser switches from hardware to software decoding. Developers currently lack a way to detect this fallback in real time without prompting users for camera/mic permissions. In the past, developers used to rely on decoderImplementation info, but as of Chromium M110+ it requires getUserMedia() permissions. This is not ideal because the UI prompt is invasive, it’s excessive since it grants access to the camera and mic hardware when apps don’t need it, and it has a high failure rate since users have little reason to grant the permission unless they want to use voice chat. This gap makes it difficult to diagnose performance regressions and provide troubleshooting guidance.

Goals

  • Enable developers to detect runtime decoder fallback from hardware to software in a non-invasive way without requiring additional permissions (does not require getUserMedia() permissions).
  • Allow applications to diagnose regressions (e.g. codec negotiation issues, device specific problems).
  • Support user experience improvements by enabling apps to adapt (e.g. lowering resolution, re-negotiating codecs), alerting end users when software decode occurs, and displaying troubleshooting information.

Non-goals

  • Exposing vendor-specific hardware information.
  • Exposing deterministic codec support/capabilities beyond what MediaCapabilities already provides.
  • Providing detailed telemetry such as frame-level error counts or decoder identifiers.

User Research

Feedback from Xbox Cloud Gaming, Nvidia GeForce Now and similar partners shows:

  • Fallback is common in the field, and developers lack visibility into when/why it occurs.
  • Reliance on getUserMedia() to query decoderImplementation has a high failure rate because users often deny permissions that are irrelevant to media playback.
  • Previous workarounds (e.g. guessing based on decode times) have proven unreliable and masked bugs.
  • Relying on MediaCapabilities is insufficient because it only provides a static capability hint and does not reflect what happens at runtime, for example, when hardware decode fails mid-session and the browser silently falls back to software.
  • Without this signal, developers cannot confidently diagnose or reduce fallback incidence.

Proposed Approach

Introduce an event on RTCRtpReceiver (see slide 30) that fires when a decoder error occurs:

  • The engine falls back from hardware to software decoding
  • No software decoder is available (e.g. in the case of H.265)

This enables applications to alert users, re-negotiate codecs, and debug issues without requiring getUserMedia() permissions.

Example

const pc = new RTCPeerConnection();

pc.addEventListener('track', (event) => {
  const receiver = event.receiver;

  // Listen for decoder state changes
  receiver.addEventListener('decoderstatechange', (ev) => {

    // Adapt application behavior based on power efficiency
    if (!ev.powerEfficient) {
        // Notify the user
        showToast("Playback quality may be reduced");

        // Lower resolution or disable heavy post-processing
        adjustQuality('low');

        // Log telemetry signal with codec and RTP timestamp
        logMetric(`Decoder fallback: codec=${ev.codecString}, rtp=${ev.rtpTimestamp}`);
    }
  });
});

Proposed IDL

partial interface RTCRtpReceiver {
attribute EventHandler ondecoderstatechange;
};

interface RTCDecoderStateChangeEvent : Event {
constructor(DOMString type, RTCDecoderStateChangeEventInit eventInitDict);

// Media timeline reference
readonly attribute unsigned long rtpTimestamp;

// Codec now in effect after the change.
readonly attribute DOMString codecString; 

// Align with MediaCapabilitiesInfo, powerEfficient changes primarily based on hardware/software decoder
// https://www.w3.org/TR/media-capabilities/#media-capabilities-info
readonly attribute boolean powerEfficient;
};

Note: The event fires at the beginning of streaming.

Alternatives Considered

  1. Use decoderImplementation info via WebRTC Stats API
    • Rejected because it now requires getUserMedia() permissions, which are invasive and have a high failure rate.
    • Requires unnecessary permissions (camera/microphone).
  2. Use MediaCapabilitiesInfo.powerEfficient
    • Rejected because this is a static hint, not a runtime signal.
    • Does not update when the browser silently switches from hardware to software.
  3. Guess based on decode times
    • Unreliable and has masked bugs in production.
  4. Add decoderFallback field to RTCInboundRtpStreamStats
    • Rejected because relying on stats to trigger a change felt like an anti-pattern and the recommendation was to explore an event driven solution. Additionally, there were concerns around fingerprinting.
    • WebRTC March 2023 meeting – 21 March 2023

Privacy Considerations

  • The event does not expose hardware vendor or device identity, reducing fingerprinting risk.
  • Does not reveal deterministic codec/hardware support.

Counter-argument to fingerprinting concerns:

  • Information is already exposed via Media Capabilities: Hardware/software decode status is already partially exposed via the MediaCapabilitiesInfo.powerEfficient attribute. A “common implementation strategy” is to treat hardware usage as indicative of optimal power draw.
  • Fallback doesn’t directly reveal capability: The fallback event does not deterministically expose hardware support, as software fallback may occur for various reasons, making it a dynamic and contextual signal rather than a static fingerprint. Software fallback may occur because:
    • Device lacks hardware support for the specific codec.
    • The hardware decoder is temporarily unavailable.

Stakeholder Feedback

  • Web Developers: Positive
  • Chromium: Positive; actively pursuing proposal.
  • WebKit & Gecko: Overall positive feedback, but privacy/fingerprinting is a common concern.

Last discussed in the 2025-09-16 WebRTC WG Call: Slides 17-21 & minutes

References & Acknowledgements

Many thanks for valuable feedback and advice from:

Links to past working group meetings where this has been discussed: