clusterfuck
/
crabidy
4
1
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects 1 Releases 5 Wiki Activity
crabidy/architecture/youtube-rustypipe.md

5.3 KiB
Raw Blame History

YouTube provider on rustypipe (pure Rust)

Context and problem statement

The first ytdy iteration shelled out to yt-dlp (a Python tool). Two problems surfaced in real use:

  1. Playback was broken. Search worked, but picked tracks never played: yt-dlp -f bestaudio selects WebM/Opus, and the player is rodio + symphonia (symphonia-all) — symphonia has no Opus decoder. The stream downloaded fine and then failed to decode. (Download captures of YouTube tracks had the same latent problem: .webm files that the local player cannot decode.)
  2. The user does not want a Python subprocess in the loop.

Evaluation of pure-Rust extractors (2026-07-21, live-tested)

  • rusty_ytdl 0.7.4 — search works, but stream URLs come back empty and stream() fails with "Video source empty": its cipher handling has fallen behind YouTube's rotation (last release early 2025).
  • rustube 0.6.0 — unmaintained since ~2022; not tested further.
  • rust-yt-downloader 0.1.0 — a small CLI, not a library engine.
  • rustypipe 0.11.4 — works end to end: search, video details, playlists, and player() returns deciphered stream URLs (verified: HTTP 206 fetch, and the full itag-140 m4a of a test video decodes through rodio::Decoder — 44.1 kHz samples out). Actively maintained (NewPipe-inspired Innertube client).

Decision: replace the subprocess engine with rustypipe. It removes the Python dependency and the format problem: we pick the stream ourselves and prefer audio/mp4 (AAC — symphonia decodes it) over Opus. Captures get .m4a audio the local player can play.

Robustness note: any non-yt-dlp extractor can break when YouTube changes Innertube. rustypipe is the most actively maintained Rust option, ships rustypipe-botguard (an optional Rust helper binary, auto-detected on PATH) for PO-token attestation if YouTube starts demanding it, and persists client state in a cache file. Accepted risk, revisit if streams start failing.

D1 — Extractor seam

The subprocess Engine is replaced by an Extractor trait owned by ytdy (search videos, video details, audio stream URL, saved playlists, playlist videos) with two implementations:

  • RustyPipeExtractor — the real one, wrapping one RustyPipe client.
  • a test fake — provider logic (path scheme, search-term store, node shapes, gating) is tested without network or fake shell scripts.

Client keeps its public ProviderClient surface, path scheme, and the in-memory search-term store unchanged.

direction: right
tui -> server -> ytdy: "/youtube/..."
ytdy: {
  client: "Client\n(paths, search terms, nodes)"
  extractor: "Extractor trait"
  client -> extractor
}
ytdy.extractor -> rustypipe: "RustyPipeExtractor"
rustypipe -> youtube: "Innertube (HTTPS)"
tests -> ytdy.extractor: "FakeExtractor"

D2 — Stream selection (the playback fix)

audio_stream_url prefers the highest-average-bitrate audio/mp4 stream (AAC — decodable by the player); only if none exists does it fall back to the overall best audio stream, with a warning (it will likely not decode locally, but the URL is still honest — e.g. a future player may cope). Download captures inherit the same choice, so their audio is .m4a.

D3 — Login and playlists

The cookies setting keeps its meaning: a path to a Netscape cookies.txt export. rustypipe consumes it natively (user_auth_set_cookie_txt) and — importantly — persists the rotated cookie in its cache file, which outlives the (quickly stale) original export. Init order: if the cache already holds a working login (user_auth_check_cookie), use it; otherwise load the configured file; on any failure degrade to logged-out with a warning (never a failed init). "Logged in" gates the playlists subtree exactly as before, now backed by saved_playlists() (the userdata feature) instead of a live-unvalidated feed/playlists scrape.

Playlist nodes page through Paginator::extend_limit up to a track cap (MAX_PLAYLIST_TRACKS, 1000) instead of loading whole playlists blindly.

D4 — Configuration and environment

ytdy.toml: binary is gone (nothing to spawn); cookies and search_results stay; call_timeout_secs maps to the rustypipe client timeout. New RustyPipe client state lives in <config>/crabidy/rustypipe/ (storage_dir) — it holds the rotated auth cookie, so it is as secret as the cookies file; neither its contents nor cookie values are ever logged. yt-dlp leaves devenv.nix. Init never probes the network except when validating a configured login; a failed login check degrades, a broken client build disables the provider non-fatally (as before).

D5 — Out of scope

  • Installing rustypipe-botguard (optional PO-token helper); document only. Streams work without it today.
  • Opus support in the player (a symphonia Opus decoder does not exist; an opus feature via a different rodio decoder is a separate project).
  • YouTube Music (rustypipe supports it; nothing in crabidy asks yet).

Risks

  • Innertube changes can break rustypipe between releases; mitigations: cache-backed client data, optional botguard, active upstream.
  • saved_playlists needs valid cookies; YouTube rotates them — the cache keeps the rotated value, but a long-cold server may need a fresh export. Degrades to logged-out, never fails.