Diagnose DRM failures
without chasing the wrong key.

Player fetches the license fine for the first minute, then a 503 (or 502) burst rolls in from the keyserver. The error envelope comes back as application/octet-stream (Widevine / FairPlay), not a clean CDN MISS — it's the keyserver, not the edge.

License-server-side burst — the keyserver pool is hot or pinned to a degraded replica, and the burst ratio (5xx_responses / total_responses in a 30s window) crossed the probe threshold. drm.license_reachable flips to fail when burst_ratio > 0.05 even if the prior probe was green.

Pre-warm the keyserver pool, raise the license-fetch burst headroom, and front the keyserver with a CDN that has a healthier peering topology for the keyserver region. Verify drm.license_reachable flips to pass and burst_ratio returns under 0.02 for 10 consecutive cycles.

Agent detects license_5xx_burst_ratio > 0.05 on cdn-A/fra02 and emits rebalance_hints=[{to: cdn-B, scope: 'level:L1', browser: '*'}] after two bad cycles. cdn-B/ams04 (which proxies the same keyserver through a healthier peering path) absorbs the next five license fetches.

Player.attach stalls at the EME step even though the manifest is serving green. The browser console shows "license-fetch timed out at 6000ms" and the first byte never arrives. Network probe is fine — it is the DRM-proxy path that is slow.

The DRM-proxy layer in front of the keyserver is consuming the entire budget on first-byte: TLS handshake succeeds, but the proxy does not forward the request until it has verified the proxy-token signature. drm.proxy_match returns warn when the signing-key version on the proxy doesn't match the issuer's v2 scheme.

Re-issue the proxy-token under the current signing scheme (v2) and increase the proxy forward-headroom budget so first-byte latency stays under 3000ms. After re-issue, drm.proxy_match flips back to pass and the finish-to-start gap on drm.license_reachable drops below 1.5s.

Agent detects proxy_token_sig_age > 24h and emits rebalance_hints=[renew_proxy_token]. On the next cycle the proxy-token is re-issued under the current signing scheme and drm.proxy_match returns pass; license-fetch first-byte latency drops from 5,820ms to 1,140ms.

Two SSDs with overlapping keys ship, but FairPlay clients still get KID_A on KID_B samples for the first 3–5 seconds after rotation. Errors read "KID mismatch: expected KID_B, manifest returned KID_A".

The packager emits pssh v1 box 3s late after the KID_B first sample. The edge cache still holds pssh v1 under the original key (KID_A), so the first wave of clients post-rotation reads a stale manifest and trips KID_B entitlement on what looks like a KID_A asset.

Align the packager pssh emission with the KID boundary (pssh v1 must land within 0.5s of the first KID_B sample, before the next msn rollover). Pre-warm the edge cache on KID_B pssh so first-cycle reads manifest the right generation.

Agent detects rotation_lag_s > 2.5 on the KID_B pssh and emits rebalance_hints=[prewarm_edge_cache_for_new_kid]. The next probe witnesses KID_B pssh served from the warmed POP, and drm.key_rotation_window returns to pass for one cycle.

After rotation, clients still resolve "default_KID=KID_A" even though cenc:default_KID was rewritten to KID_B in the new init segment. The browser skip-keys-after-KID-mismatch path triggers a hard re-key — usually a stall.

Edge / proxy cache is holding init.mp4 under the previous rotation tag (the SHA of init.mp4 is unchanged because tenc boxes overlap, but cenc:default_KID is different). Cache hit on the previous init.mp4 means the player reads the previous default_KID and queries KID_A on KID_B samples.

Strip an explicit cache-key from init.mp4 that includes cenc:default_KID (or rotate the init.mp4 SHA on each tenc change via an outgoing edge policy). Pair the cenc change with a soft purge so the next read is forced to the current init.

Agent detects init_seg_cache_age > rotation_window and emits rebalance_hints=[soft_purge_init_seg_for_kid_B]. The next probe sees init.mp4 served from the new cache-key, default_KID resolves to KID_B, and drm.key_rotation_window flips to pass.

After a cert rollover (the keyserver's cert rotates), some players still handshake-validate the keyserver against the previous cert. Console reads 'keys refused — cert expired' or 'cert chain incomplete'. The probe sees a 200 on the first request and then fails on the second.

DRM-proxy cert was rolled at the issuer but the proxy pair wasn't re-synced — so it serves both old and new certs depending on the POP. drm.cert_valid fails because cert.stale_age > 24h — the SP/Browser pool is half old-root, half new-root.

Re-sync the proxy pair and front-load the new cert on every POP. Bump the cert-stale-age threshold above the issuer rotation interval so legitimate rotations do not flag as failures — only true proxy missyncs flip the probe to fail.

Agent detects cert.stale_age > 24h and triggers a DRM-proxy token renewal task before the next probe cycle. Once renewed, drm.cert_valid returns pass and the cert_seq_no on the proxy serves both old and new root chains until the next soft-rotation.

License fetch hangs at the proxy boundary — first-byte latency is unbounded. DevTools reports the request 'pending' until it times out at the client. No 4xx, no 5xx — the proxy never responds.

The DRM-proxy was issued a token under signing scheme v1, but the keyserver has been moved under scheme v2. The proxy accepts the request (token signature validates under v1) but the keyserver v2 handshake on the proxy-target side fails, leaving the request stuck in a forwarding queue.

Re-issue the proxy-token under scheme v2 — both sides now agree. Set up a dual-scheme period so the proxy can decode both v1 and v2 tokens while the issuer rolls forward.

Agent detects proxy_token.scheme_v != keyserver.scheme_v and emits rebalance_hints=[renew_proxy_token + dual_sign_for_24h]. The next probe witnesses the proxy-token under v2 and drm.proxy_match returns pass.

Same content, same asset, same KID — Chrome plays cleanly with a Widevine license, but Safari throws 'license refused'. Player reports show Widevine at L1 pass, FairPlay at L1 fail on a 403.

SPDC cert rotation on the FairPlay endpoint landed upstream but the enterprise FairPlay deployment did not re-issue the new root to its Apple-side licensing endpoint. Widevine and PlayReady entitlement is consistent — only FairPlay flips to fail because its cert chain is stale.

Refresh the FairPlay cert chain on the Apple-side endpoint AND re-issue the asset under the current SPDC root. drm.entitlement_consistent flips to pass when all three key systems return consistent verdicts on the same KID.

Agent detects entitlement_inconsistent on FairPlay and emits rebalance_hints=[pin_fairplay_to_cdn_B + refresh_spdc_root]. cdn-B/eu-west serves FairPlay L1 from a current root while the upstream re-syncs — playback recovers for ~80% of the Safari cohort.

Hardened clients (PlayReady SL3000) fail on a license that only offers SL2000. Browser logs read "Session token missing hardening proof." Standard clients succeed. Entitlement looks green until the hardened cohort is dissected separately.

PlayReady rights manager is configured to issue SL2000 entitlements only (the lower-tier default). Hardened clients that require SL3000 receive a challenge because the issuer never offers it. drm.entitlement_consistent warns — standard clients are fine, hardened clients are downgraded.

Raise the rights manager to issue SL3000 entitlements where the device supports it AND verify the hardened client can satisfy the SL3000 challenge. Pair this with the SL2000 fallback so legacy clients are not broken during the rollout.

Agent detects playready_level_downgrade_count > 0 and emits rebalance_hints=[raise_rights_to_sl3000_for_hardened_clients]. On the next probe hardened clients receive SL3000 entitlements and the downgrade count drops to zero within 5 cycles.

FairPlay L1 playback fails for Safari clients routed through cdn-A/eu-west but succeeds for Safari clients routed through cdn-B/eu-west. Same manifest, same KID, same client build — only the POP changes.

POP-level cert-chain miss on cdn-A/eu-west (Apple root not yet propagated to that POP). drm.playback_matrix reports partial-consistent verdict mix: 4 pass, 1 fail. The failing cell is exactly browser=Safari(>=17), cdn=cdn-A/eu-west, level=L1.

Roll the Apple root cert to cdn-A/eu-west via the CDN provider and pin the manifest to prefer cdn-B for Safari L1 until cdn-A is repaired. Verify drm.playback_matrix returns consistent after the cert is propagated to the failing POP.

Agent detects playback_matrix inconsistent on safari+cdn-A/L1 and emits rebalance_hints=[pin_safari_L1_to_cdn_B_for_eu-west]. cdn-B absorbs the next five Safari L1 license fetches; cdn-A gets repaired by the CDN provider while streaming stays green.

Chrome on a desktop with a TEE/HDCP path throws "hardware DRM unavailable" and falls back to L3 (software). The fallback plays, but the security level drops from L1 to L3 and the player logs "DRM level: SL3000 unreachable".

Chrome's hardware path uses the platform's secure decoder; the device's TEE chain is reporting an incomplete attestation chain, so the Chrome CDM rejects L1 device certificates. drm.playback_matrix reports level=L1 / fail, level=L3 / pass; the agent logs level_downgrade_count>0.

Re-flash the TEE / device-cert chain so the attestation is current. Pair the fix with a known-good CDM build (current Chrome CDM) — older CDM builds may trip platform-level attestation checks that the current CDM already works around.

Agent detects chrome_L1_attestation_failures > threshold and emits rebalance_hints=[downgrade_chrome_to_L3_for_device_class + alert_cdm_team]. Chrome L3 (software) is acceptable short-term; the agent also surfaces a CDN tag for the incident so on-call gets paged the same cycle.