clusterfuck
/
crabidy
4
1
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects 1 Releases 5 Wiki Activity
crabidy/architecture/youtube-rustypipe.md

7.8 KiB
Raw Blame History

YouTube provider on rustypipe

Amended the same day by D2-revised below: pure-Rust stream fetching turned out to be blocked by YouTube's PO-token enforcement, so stream URLs go through a minimal yt-dlp sidecar again while all metadata stays on rustypipe.

Context and problem statement

The first ytdy iteration shelled out to yt-dlp (a Python tool). Two problems surfaced in real use:

  1. Playback was broken. Search worked, but picked tracks never played: yt-dlp -f bestaudio selects WebM/Opus, and the player is rodio + symphonia (symphonia-all) — symphonia has no Opus decoder. The stream downloaded fine and then failed to decode. (Download captures of YouTube tracks had the same latent problem: .webm files that the local player cannot decode.)
  2. The user does not want a Python subprocess in the loop.

Evaluation of pure-Rust extractors (2026-07-21, live-tested)

  • rusty_ytdl 0.7.4 — search works, but stream URLs come back empty and stream() fails with "Video source empty": its cipher handling has fallen behind YouTube's rotation (last release early 2025).
  • rustube 0.6.0 — unmaintained since ~2022; not tested further.
  • rust-yt-downloader 0.1.0 — a small CLI, not a library engine.
  • rustypipe 0.11.4 — works end to end: search, video details, playlists, and player() returns deciphered stream URLs (verified: HTTP 206 fetch, and the full itag-140 m4a of a test video decodes through rodio::Decoder — 44.1 kHz samples out). Actively maintained (NewPipe-inspired Innertube client).

Decision: replace the subprocess engine with rustypipe. It removes the Python dependency and the format problem: we pick the stream ourselves and prefer audio/mp4 (AAC — symphonia decodes it) over Opus. Captures get .m4a audio the local player can play.

Robustness note: any non-yt-dlp extractor can break when YouTube changes Innertube. rustypipe is the most actively maintained Rust option, ships rustypipe-botguard (an optional Rust helper binary, auto-detected on PATH) for PO-token attestation if YouTube starts demanding it, and persists client state in a cache file. Accepted risk, revisit if streams start failing.

D1 — Extractor seam

The subprocess Engine is replaced by an Extractor trait owned by ytdy (search videos, video details, audio stream URL, saved playlists, playlist videos) with two implementations:

  • RustyPipeExtractor — the real one, wrapping one RustyPipe client.
  • a test fake — provider logic (path scheme, search-term store, node shapes, gating) is tested without network or fake shell scripts.

Client keeps its public ProviderClient surface, path scheme, and the in-memory search-term store unchanged.

direction: right
tui -> server -> ytdy: "/youtube/..."
ytdy: {
  client: "Client\n(paths, search terms, nodes)"
  extractor: "Extractor trait"
  client -> extractor
}
ytdy.extractor -> rustypipe: "RustyPipeExtractor"
rustypipe -> youtube: "Innertube (HTTPS)"
tests -> ytdy.extractor: "FakeExtractor"

D2 — Stream selection (the playback fix)

audio_stream_url prefers the highest-average-bitrate audio/mp4 stream (AAC — decodable by the player); only if none exists does it fall back to the overall best audio stream, with a warning (it will likely not decode locally, but the URL is still honest — e.g. a future player may cope). Download captures inherit the same choice, so their audio is .m4a.

D2-revised — Stream fetching (2026-07-21, second round)

Real use immediately hit YouTube's tokenless-fetch enforcement. Live findings against googlevideo (all reproduced, same day):

  • Every rustypipe (iOS-client) stream URL serves exactly its leading ~1 MiB, measured to the byte (403 at cumulative 1 048 576): plain GETs 403, open-ended ranges 403, bounded ranges work only until the budget is spent, and a fresh URL refuses to start at an offset — so chaining fresh URLs per window is impossible.
  • PO tokens would lift the cap, but rustypipe attaches them only to web-family clients, whose signature deciphering is broken upstream right now ("could not extract sig fn name", also on git master) — rustypipe-botguard was built and tested; ineffective through the iOS client (no pot parameter).
  • yt-dlp (2026.07.04) still solves the cipher challenges; its URLs accept plain and ranged GETs for the whole file, throttled to ~32 KB/s — twice the itag-140 audio bitrate, so playback holds and captures are merely slow. Its TV/Android clients are SABR-blocked (no URLs at all), so this is the state of the art everywhere.

Decisions:

  1. Windowed HTTP fetching everywhere. The player streams through a WindowedHttpStream (audio-player, a stream-download SourceStream) and the capture Downloader downloads in the same bounded ~1 MiB Range windows — the request pattern real players produce, correct for every provider (servers that ignore Range degrade to one 200 body; rejected windows fail typed, never loop).
  2. yt-dlp returns as a stream-URL-only sidecar. All metadata (search, playlists, details) stays on the pure-Rust rustypipe extractor; only get_urls_for_track consults yt-dlp (argv-only, bounded, stderr-summarized). A missing binary degrades with a warning — streams then stop after their first ~1 MiB instead of failing entirely.
  3. The pure-Rust path stays wired. botguard_bin is configurable (and PATH-auto-detected by rustypipe); when upstream deciphering recovers, PO-token'd rustypipe URLs make the sidecar unnecessary without code changes beyond removing the fallback preference.

The per-track capture deadline was raised to 30 minutes — at 32 KB/s a long track legitimately takes that.

D3 — Login and playlists

The cookies setting keeps its meaning: a path to a Netscape cookies.txt export. rustypipe consumes it natively (user_auth_set_cookie_txt) and — importantly — persists the rotated cookie in its cache file, which outlives the (quickly stale) original export. Init order: if the cache already holds a working login (user_auth_check_cookie), use it; otherwise load the configured file; on any failure degrade to logged-out with a warning (never a failed init). "Logged in" gates the playlists subtree exactly as before, now backed by saved_playlists() (the userdata feature) instead of a live-unvalidated feed/playlists scrape.

Playlist nodes page through Paginator::extend_limit up to a track cap (MAX_PLAYLIST_TRACKS, 1000) instead of loading whole playlists blindly.

D4 — Configuration and environment

ytdy.toml: binary is gone (nothing to spawn); cookies and search_results stay; call_timeout_secs maps to the rustypipe client timeout. New RustyPipe client state lives in <config>/crabidy/rustypipe/ (storage_dir) — it holds the rotated auth cookie, so it is as secret as the cookies file; neither its contents nor cookie values are ever logged. yt-dlp leaves devenv.nix. Init never probes the network except when validating a configured login; a failed login check degrades, a broken client build disables the provider non-fatally (as before).

D5 — Out of scope

  • Installing rustypipe-botguard (optional PO-token helper); document only. Streams work without it today.
  • Opus support in the player (a symphonia Opus decoder does not exist; an opus feature via a different rodio decoder is a separate project).
  • YouTube Music (rustypipe supports it; nothing in crabidy asks yet).

Risks

  • Innertube changes can break rustypipe between releases; mitigations: cache-backed client data, optional botguard, active upstream.
  • saved_playlists needs valid cookies; YouTube rotates them — the cache keeps the rotated value, but a long-cold server may need a fresh export. Degrades to logged-out, never fails.