crabidy/architecture/youtube-provider.md

YouTube provider (ytdy)

Extraction engine superseded by youtube-rustypipe.md: the yt-dlp subprocess (D1, D5, the binary setting, and the bestaudio format choice) was replaced with the pure-Rust rustypipe client after playback turned out broken (bestaudio = Opus, which the rodio/symphonia player cannot decode) and the Python subprocess proved unwanted. The tree shape, search-term store, path scheme, and optional-login gating described here still hold.

Context and problem statement

A new library provider for YouTube, mounted at /youtube:

• Search works without any login — creatable search-term nodes like /tidal/search (% creates a term, results list as tracks).
• Login is optional. When configured, the user's playlists appear as an extra subtree; without it, the provider still works (search only).
• Captures (W, download) must work on YouTube nodes — search results and playlists are downloadable.

Assumptions (confirmed)

• The captures machinery is provider-agnostic: anything that answers get_lib_node/get_urls_for_track and raises is_downloadable gets w/W for free — no new wire or TUI work at all.
• The TUI's creatable/editable/deletable node flows (%/e/d) are generic; mirroring tidal's search-term semantics costs no TUI change.
• pkgs.yt-dlp (2026.06.09) exists in nixpkgs; the devenv already pins the toolchain, so the engine binary is declared, not assumed.
• The audio player streams plain https URLs; googlevideo stream URLs (from bestaudio) are plain https and also downloadable with the captures reqwest client.

Decisions

D1 — Extraction engine: yt-dlp subprocess

Options considered:

• (a) Pure-Rust extractor crates (rustypipe, rusty_ytdl): no external binary, but they chase YouTube's extraction changes with small maintainer teams, and logged-in user playlists are weakly or not supported.
• (b) Invidious/Piped instances: no extraction code at all, but a hard runtime dependency on third-party servers of unpredictable availability — worse than a local binary for a self-hosted player.
• (c) yt-dlp as a subprocess with -J JSON output: the de-facto standard extractor, fastest to track YouTube changes, supports search (ytsearchN:), playlists, cookies-based login, and direct stream URLs. Cost: a non-Rust runtime dependency and subprocess plumbing.

Decision: (c). The binary is declared in devenv.nix (dev) and is a documented runtime requirement (deploys). All calls go through one Engine seam (tokio::process::Command, kill_on_drop, per-call timeout, bounded stdout, serde_json parsing) so tests fake the binary with a script and a future pure-Rust engine stays swappable.

D2 — Crate ytdy, mounted at /youtube, non-fatal init

New workspace crate ytdy implementing ProviderClient, following tidaldy's shape. init probes <binary> --version (with timeout); a missing or broken binary disables the provider with a warning — the server and every other provider keep running. Settings (ytdy.toml):

binary = "yt-dlp"          # optional; PATH lookup by default
cookies = "/path/cookies.txt"  # optional; presence = "logged in"
search_results = 20        # ytsearchN cap

The orchestrator gains youtube_client: Option<Arc<ytdy::Client>> and /youtube routing arms (same completeness as the other providers); get_lib_root lists youtube only when the probe succeeded.

D3 — Tree shape

• /youtube — children: search (always, is_creatable), playlists (only when cookies are configured).
• /youtube/search/<term> — created via % like tidal search; terms live in memory (RwLock<Vec<String>>, dedup, recreated implicitly on stale paths), renamable and deletable. The node lists the top search_results results as tracks (ytsearchN:<term>, --flat-playlist); it is queueable (results are homogeneous tracks, unlike tidal's mixed search).
• /youtube/playlists — the user's playlists as children (https://www.youtube.com/feed/playlists with cookies, flat).
• /youtube/playlists/<pid> — playlist entries as tracks.
• Track paths: <node>/<videoid>; metadata from the flat entries (title, uploader as artist, duration). get_metadata_for_track runs a single-video -J when called directly.

D4 — Streams, downloads, login

• get_urls_for_track: yt-dlp -f bestaudio/best -g --no-playlist <video> → the stream URL(s). Cookies are passed to every call when configured (search and streams benefit too, e.g. age-gated videos).
• Downloadability mirrors tidal's central rule: a node is downloadable when it is queueable or lists tracks; children mirror is_queable. extension_for gains audio/webm → webm (bestaudio is usually opus in webm; googlevideo URLs carry no path extension).
• "Logged in" is exactly "a cookies file is configured and readable at init" — no OAuth flow of our own, no credential storage beyond the user-provided file. The cookies path is config; its contents are a secret and must never appear in logs or error reports.

D5 — Subprocess discipline

• One shared Engine with the binary path; every invocation: --no-warnings -J (or -g), argument-list only (never shell), per-call timeout (default 60 s, configurable for tests), stdout capped, stderr summarized into warnings (never echoed wholesale — it can contain URLs).
• Errors are typed (EngineError: spawn/timeout/exit-status/parse) and map to ProviderError::FetchError/MalformedPath at the trait boundary; no panic on any subprocess condition.
• Calls run sequentially per request (no engine-level queue); the orchestrator already spawns long walks.

D6 — Out of scope (explicitly)

• OAuth/device login flows, cookie refresh, or storing credentials.
• Uploads, likes, subscriptions feeds, comments; YouTube Music.
• Caching search or playlist results across calls (each listing is a fresh subprocess call, like tidal's fresh HTTP calls).
• SponsorBlock, chapters, DRM-protected content (skipped as unplayable).

Structure

direction: right

server: crabidy-server {
  orch: ProviderOrchestrator
}

ytdy: "ytdy (crate)" {
  client: "Client\n(ProviderClient)"
  terms: "search terms\n(in-memory, like tidal)"
  engine: "Engine\n(subprocess seam)"
  client -> terms
  client -> engine
}

bin: "yt-dlp binary\n(devenv / PATH)" {
  shape: hexagon
}

yt: "YouTube" { shape: cloud }
cookies: "cookies.txt\n(optional, user-provided)" { shape: page }

server.orch -> ytdy.client: "/youtube/..."
ytdy.engine -> bin: "argv + -J, timeout,\nbounded stdout"
bin -> yt
cookies -> ytdy.engine: "--cookies (when set)"

Key flow: search, then W

shape: sequence_diagram
tui: TUI
orch: Orchestrator
y: ytdy
e: yt-dlp

tui -> orch: "% on /youtube/search: 'lofi'"
orch -> y: "create_lib_node"
y -> e: "ytsearch20:lofi -J --flat-playlist"
e -> y: "entries (id, title, uploader, duration)"
y -> tui: "term node: 20 tracks, downloadable"
tui -> orch: "W → CaptureLibraryNode(download)"
orch -> y: "per track: get_urls_for_track"
y -> e: "-f bestaudio -g <id>"
e -> y: "googlevideo url"
orch -> orch: "captures store: stream to\n/captures/<name>/ (webm + toml)"

Risks and open questions

• feed/playlists extraction: the exact yt-dlp invocation for "my playlists" remains live-unvalidated (no cookies were available during development; search, streams, and download captures were validated live). Fallback if it turns out wrong: adjust the feed URL in playlists_node, or bookmark playlist URLs as search terms.
• yt-dlp breakage/drift: extraction can break until the devenv pin is bumped; errors stay typed and non-fatal.
• Stream URL lifetime: googlevideo URLs expire (~6 h); captures download immediately after resolving, playback resolves on play — both within the window.
• Terms of service: downloads are personal-use copies of streams the client can already play, same stance as tidal captures.