crabidy/architecture/youtube-provider.md

# YouTube provider (ytdy)

## Context and problem statement

A new library provider for YouTube, mounted at `/youtube`:

- **Search** works without any login — creatable search-term nodes like
  `/tidal/search` (`%` creates a term, results list as tracks).
- **Login is optional.** When configured, the user's **playlists** appear
  as an extra subtree; without it, the provider still works (search
  only).
- **Captures** (`W`, download) must work on YouTube nodes — search
  results and playlists are downloadable.

## Assumptions (confirmed)

- The captures machinery is provider-agnostic: anything that answers
  `get_lib_node`/`get_urls_for_track` and raises `is_downloadable` gets
  `w`/`W` for free — no new wire or TUI work at all.
- The TUI's creatable/editable/deletable node flows (`%`/`e`/`d`) are
  generic; mirroring tidal's search-term semantics costs no TUI change.
- `pkgs.yt-dlp` (2026.06.09) exists in nixpkgs; the devenv already pins
  the toolchain, so the engine binary is declared, not assumed.
- The audio player streams plain https URLs; googlevideo stream URLs
  (from `bestaudio`) are plain https and also downloadable with the
  captures reqwest client.

## Decisions

### D1 — Extraction engine: `yt-dlp` subprocess

Options considered:

- *(a)* **Pure-Rust extractor crates** (`rustypipe`, `rusty_ytdl`): no
  external binary, but they chase YouTube's extraction changes with
  small maintainer teams, and logged-in user playlists are weakly or not
  supported.
- *(b)* **Invidious/Piped instances**: no extraction code at all, but a
  hard runtime dependency on third-party servers of unpredictable
  availability — worse than a local binary for a self-hosted player.
- *(c)* **`yt-dlp` as a subprocess** with `-J` JSON output: the de-facto
  standard extractor, fastest to track YouTube changes, supports search
  (`ytsearchN:`), playlists, cookies-based login, and direct stream
  URLs. Cost: a non-Rust runtime dependency and subprocess plumbing.

**Decision: (c).** The binary is declared in `devenv.nix` (dev) and is a
documented runtime requirement (deploys). All calls go through one
`Engine` seam (`tokio::process::Command`, `kill_on_drop`, per-call
timeout, bounded stdout, `serde_json` parsing) so tests fake the binary
with a script and a future pure-Rust engine stays swappable.

### D2 — Crate `ytdy`, mounted at `/youtube`, non-fatal init

New workspace crate `ytdy` implementing `ProviderClient`, following
`tidaldy`'s shape. `init` probes `<binary> --version` (with timeout);
a missing or broken binary disables the provider with a warning — the
server and every other provider keep running. Settings (`ytdy.toml`):

```toml
binary = "yt-dlp"          # optional; PATH lookup by default
cookies = "/path/cookies.txt"  # optional; presence = "logged in"
search_results = 20        # ytsearchN cap
```

The orchestrator gains `youtube_client: Option<Arc<ytdy::Client>>` and
`/youtube` routing arms (same completeness as the other providers);
`get_lib_root` lists `youtube` only when the probe succeeded.

### D3 — Tree shape

- `/youtube` — children: `search` (always, `is_creatable`), `playlists`
  (only when cookies are configured).
- `/youtube/search/<term>` — created via `%` like tidal search; terms
  live in memory (`RwLock<Vec<String>>`, dedup, recreated implicitly on
  stale paths), renamable and deletable. The node lists the top
  `search_results` results as **tracks** (`ytsearchN:<term>`,
  `--flat-playlist`); it is queueable (results are homogeneous tracks,
  unlike tidal's mixed search).
- `/youtube/playlists` — the user's playlists as children
  (`https://www.youtube.com/feed/playlists` with cookies, flat).
- `/youtube/playlists/<pid>` — playlist entries as tracks.
- Track paths: `<node>/<videoid>`; metadata from the flat entries
  (title, uploader as artist, duration). `get_metadata_for_track` runs
  a single-video `-J` when called directly.

### D4 — Streams, downloads, login

- `get_urls_for_track`: `yt-dlp -f bestaudio/best -g --no-playlist
  <video>` → the stream URL(s). Cookies are passed to **every** call
  when configured (search and streams benefit too, e.g. age-gated
  videos).
- Downloadability mirrors tidal's central rule: a node is downloadable
  when it is queueable or lists tracks; children mirror `is_queable`.
  `extension_for` gains `audio/webm → webm` (bestaudio is usually opus
  in webm; googlevideo URLs carry no path extension).
- "Logged in" is exactly "a cookies file is configured and readable at
  init" — no OAuth flow of our own, no credential storage beyond the
  user-provided file. The cookies path is config; its **contents are a
  secret** and must never appear in logs or error reports.

### D5 — Subprocess discipline

- One shared `Engine` with the binary path; every invocation:
  `--no-warnings -J` (or `-g`), argument-list only (never shell),
  per-call timeout (default 60 s, configurable for tests), stdout
  capped, stderr summarized into warnings (never echoed wholesale — it
  can contain URLs).
- Errors are typed (`EngineError`: spawn/timeout/exit-status/parse) and
  map to `ProviderError::FetchError`/`MalformedPath` at the trait
  boundary; no panic on any subprocess condition.
- Calls run sequentially per request (no engine-level queue); the
  orchestrator already spawns long walks.

### D6 — Out of scope (explicitly)

- OAuth/device login flows, cookie refresh, or storing credentials.
- Uploads, likes, subscriptions feeds, comments; YouTube Music.
- Caching search or playlist results across calls (each listing is a
  fresh subprocess call, like tidal's fresh HTTP calls).
- SponsorBlock, chapters, DRM-protected content (skipped as unplayable).

## Structure

```d2
direction: right

server: crabidy-server {
  orch: ProviderOrchestrator
}

ytdy: "ytdy (crate)" {
  client: "Client\n(ProviderClient)"
  terms: "search terms\n(in-memory, like tidal)"
  engine: "Engine\n(subprocess seam)"
  client -> terms
  client -> engine
}

bin: "yt-dlp binary\n(devenv / PATH)" {
  shape: hexagon
}

yt: "YouTube" { shape: cloud }
cookies: "cookies.txt\n(optional, user-provided)" { shape: page }

server.orch -> ytdy.client: "/youtube/..."
ytdy.engine -> bin: "argv + -J, timeout,\nbounded stdout"
bin -> yt
cookies -> ytdy.engine: "--cookies (when set)"
```

## Key flow: search, then W

```d2
shape: sequence_diagram
tui: TUI
orch: Orchestrator
y: ytdy
e: yt-dlp

tui -> orch: "% on /youtube/search: 'lofi'"
orch -> y: "create_lib_node"
y -> e: "ytsearch20:lofi -J --flat-playlist"
e -> y: "entries (id, title, uploader, duration)"
y -> tui: "term node: 20 tracks, downloadable"
tui -> orch: "W → CaptureLibraryNode(download)"
orch -> y: "per track: get_urls_for_track"
y -> e: "-f bestaudio -g <id>"
e -> y: "googlevideo url"
orch -> orch: "captures store: stream to\n/captures/<name>/ (webm + toml)"
```

## Risks and open questions

- **`feed/playlists` extraction**: the exact yt-dlp invocation for "my
  playlists" remains **live-unvalidated** (no cookies were available
  during development; search, streams, and download captures were
  validated live). Fallback if it turns out wrong: adjust the feed URL
  in `playlists_node`, or bookmark playlist URLs as search terms.
- **yt-dlp breakage/drift**: extraction can break until the devenv pin
  is bumped; errors stay typed and non-fatal.
- **Stream URL lifetime**: googlevideo URLs expire (~6 h); captures
  download immediately after resolving, playback resolves on play —
  both within the window.
- **Terms of service**: downloads are personal-use copies of streams the
  client can already play, same stance as tidal captures.