crabidy/architecture/youtube-rustypipe.md

# YouTube provider on rustypipe

> **Amended the same day** by D2-revised below: pure-Rust stream
> *fetching* turned out to be blocked by YouTube's PO-token
> enforcement, so stream URLs go through a minimal `yt-dlp` sidecar
> again while all metadata stays on rustypipe.

## Context and problem statement

The first `ytdy` iteration shelled out to `yt-dlp` (a Python tool). Two
problems surfaced in real use:

1. **Playback was broken.** Search worked, but picked tracks never
   played: `yt-dlp -f bestaudio` selects WebM/**Opus**, and the player is
   rodio + symphonia (`symphonia-all`) — symphonia has **no Opus
   decoder**. The stream downloaded fine and then failed to decode.
   (Download captures of YouTube tracks had the same latent problem:
   `.webm` files that the local player cannot decode.)
2. The user does not want a Python subprocess in the loop.

## Evaluation of pure-Rust extractors (2026-07-21, live-tested)

- `rusty_ytdl` 0.7.4 — search works, but **stream URLs come back
  empty** and `stream()` fails with "Video source empty": its cipher
  handling has fallen behind YouTube's rotation (last release early
  2025).
- `rustube` 0.6.0 — unmaintained since ~2022; not tested further.
- `rust-yt-downloader` 0.1.0 — a small CLI, not a library engine.
- `rustypipe` 0.11.4 — **works end to end**: search, video details,
  playlists, and `player()` returns deciphered stream URLs (verified:
  HTTP 206 fetch, and the full itag-140 m4a of a test video decodes
  through `rodio::Decoder` — 44.1 kHz samples out). Actively maintained
  (NewPipe-inspired Innertube client).

**Decision: replace the subprocess engine with `rustypipe`.** It removes
the Python dependency *and* the format problem: we pick the stream
ourselves and prefer `audio/mp4` (AAC — symphonia decodes it) over
Opus. Captures get `.m4a` audio the local player can play.

Robustness note: any non-yt-dlp extractor can break when YouTube
changes Innertube. rustypipe is the most actively maintained Rust
option, ships `rustypipe-botguard` (an optional *Rust* helper binary,
auto-detected on PATH) for PO-token attestation if YouTube starts
demanding it, and persists client state in a cache file. Accepted risk,
revisit if streams start failing.

## D1 — Extractor seam

The subprocess `Engine` is replaced by an `Extractor` trait owned by
`ytdy` (search videos, video details, audio stream URL, saved
playlists, playlist videos) with two implementations:

- `RustyPipeExtractor` — the real one, wrapping one `RustyPipe` client.
- a test fake — provider logic (path scheme, search-term store, node
  shapes, gating) is tested without network or fake shell scripts.

`Client` keeps its public `ProviderClient` surface, path scheme, and
the in-memory search-term store unchanged.

```d2
direction: right
tui -> server -> ytdy: "/youtube/..."
ytdy: {
  client: "Client\n(paths, search terms, nodes)"
  extractor: "Extractor trait"
  client -> extractor
}
ytdy.extractor -> rustypipe: "RustyPipeExtractor"
rustypipe -> youtube: "Innertube (HTTPS)"
tests -> ytdy.extractor: "FakeExtractor"
```

## D2 — Stream selection (the playback fix)

`audio_stream_url` prefers the highest-average-bitrate `audio/mp4`
stream (AAC — decodable by the player); only if none exists does it
fall back to the overall best audio stream, with a warning (it will
likely not decode locally, but the URL is still honest — e.g. a future
player may cope). Download captures inherit the same choice, so their
audio is `.m4a`.

## D2-revised — Stream *fetching* (2026-07-21, second round)

Real use immediately hit YouTube's tokenless-fetch enforcement. Live
findings against googlevideo (all reproduced, same day):

- Every rustypipe (iOS-client) stream URL serves **exactly its leading
  ~1 MiB**, measured to the byte (403 at cumulative 1 048 576): plain
  GETs 403, open-ended ranges 403, bounded ranges work only until the
  budget is spent, and a **fresh URL refuses to start at an offset** —
  so chaining fresh URLs per window is impossible.
- PO tokens would lift the cap, but rustypipe attaches them only to
  web-family clients, whose **signature deciphering is broken upstream
  right now** ("could not extract sig fn name", also on git master) —
  `rustypipe-botguard` was built and tested; ineffective through the
  iOS client (no `pot` parameter).
- `yt-dlp` (2026.07.04) still solves the cipher challenges; its URLs
  accept plain and ranged GETs for the whole file, **throttled to
  ~32 KB/s** — twice the itag-140 audio bitrate, so playback holds and
  captures are merely slow. Its TV/Android clients are SABR-blocked
  (no URLs at all), so this is the state of the art everywhere.

**Decisions:**

1. **Windowed HTTP fetching everywhere.** The player streams through a
   `WindowedHttpStream` (audio-player, a `stream-download`
   `SourceStream`) and the capture `Downloader` downloads in the same
   bounded ~1 MiB `Range` windows — the request pattern real players
   produce, correct for every provider (servers that ignore `Range`
   degrade to one 200 body; rejected windows fail typed, never loop).
2. **`yt-dlp` returns as a stream-URL-only sidecar.** All metadata
   (search, playlists, details) stays on the pure-Rust rustypipe
   extractor; only `get_urls_for_track` consults `yt-dlp` (argv-only,
   bounded, stderr-summarized). A missing binary degrades with a
   warning — streams then stop after their first ~1 MiB instead of
   failing entirely.
3. **The pure-Rust path stays wired.** `botguard_bin` is configurable
   (and PATH-auto-detected by rustypipe); when upstream deciphering
   recovers, PO-token'd rustypipe URLs make the sidecar unnecessary
   without code changes beyond removing the fallback preference.

The per-track capture deadline was raised to 30 minutes — at 32 KB/s a
long track legitimately takes that.

## D3 — Login and playlists

The `cookies` setting keeps its meaning: a path to a Netscape
`cookies.txt` export. rustypipe consumes it natively
(`user_auth_set_cookie_txt`) and — importantly — **persists the rotated
cookie in its cache file**, which outlives the (quickly stale) original
export. Init order: if the cache already holds a working login
(`user_auth_check_cookie`), use it; otherwise load the configured file;
on any failure degrade to logged-out with a warning (never a failed
init). "Logged in" gates the `playlists` subtree exactly as before,
now backed by `saved_playlists()` (the `userdata` feature) instead of a
live-unvalidated `feed/playlists` scrape.

Playlist nodes page through `Paginator::extend_limit` up to a track
cap (`MAX_PLAYLIST_TRACKS`, 1000) instead of loading whole playlists
blindly.

## D4 — Configuration and environment

`ytdy.toml`: `binary` is gone (nothing to spawn); `cookies` and
`search_results` stay; `call_timeout_secs` maps to the rustypipe
client timeout. New `RustyPipe` client state lives in
`<config>/crabidy/rustypipe/` (`storage_dir`) — it holds the rotated
auth cookie, so it is as secret as the cookies file; neither its
contents nor cookie values are ever logged. `yt-dlp` leaves
`devenv.nix`. Init never probes the network except when validating a
configured login; a failed login check degrades, a broken client build
disables the provider non-fatally (as before).

## D5 — Out of scope

- Installing `rustypipe-botguard` (optional PO-token helper); document
  only. Streams work without it today.
- Opus support in the player (a symphonia Opus decoder does not exist;
  an opus feature via a different rodio decoder is a separate project).
- YouTube Music (rustypipe supports it; nothing in crabidy asks yet).

## Risks

- Innertube changes can break rustypipe between releases; mitigations:
  cache-backed client data, optional botguard, active upstream.
- `saved_playlists` needs valid cookies; YouTube rotates them — the
  cache keeps the rotated value, but a long-cold server may need a
  fresh export. Degrades to logged-out, never fails.