121 lines
5.3 KiB
Markdown
121 lines
5.3 KiB
Markdown
# YouTube provider on rustypipe (pure Rust)
|
|
|
|
## Context and problem statement
|
|
|
|
The first `ytdy` iteration shelled out to `yt-dlp` (a Python tool). Two
|
|
problems surfaced in real use:
|
|
|
|
1. **Playback was broken.** Search worked, but picked tracks never
|
|
played: `yt-dlp -f bestaudio` selects WebM/**Opus**, and the player is
|
|
rodio + symphonia (`symphonia-all`) — symphonia has **no Opus
|
|
decoder**. The stream downloaded fine and then failed to decode.
|
|
(Download captures of YouTube tracks had the same latent problem:
|
|
`.webm` files that the local player cannot decode.)
|
|
2. The user does not want a Python subprocess in the loop.
|
|
|
|
## Evaluation of pure-Rust extractors (2026-07-21, live-tested)
|
|
|
|
- `rusty_ytdl` 0.7.4 — search works, but **stream URLs come back
|
|
empty** and `stream()` fails with "Video source empty": its cipher
|
|
handling has fallen behind YouTube's rotation (last release early
|
|
2025).
|
|
- `rustube` 0.6.0 — unmaintained since ~2022; not tested further.
|
|
- `rust-yt-downloader` 0.1.0 — a small CLI, not a library engine.
|
|
- `rustypipe` 0.11.4 — **works end to end**: search, video details,
|
|
playlists, and `player()` returns deciphered stream URLs (verified:
|
|
HTTP 206 fetch, and the full itag-140 m4a of a test video decodes
|
|
through `rodio::Decoder` — 44.1 kHz samples out). Actively maintained
|
|
(NewPipe-inspired Innertube client).
|
|
|
|
**Decision: replace the subprocess engine with `rustypipe`.** It removes
|
|
the Python dependency *and* the format problem: we pick the stream
|
|
ourselves and prefer `audio/mp4` (AAC — symphonia decodes it) over
|
|
Opus. Captures get `.m4a` audio the local player can play.
|
|
|
|
Robustness note: any non-yt-dlp extractor can break when YouTube
|
|
changes Innertube. rustypipe is the most actively maintained Rust
|
|
option, ships `rustypipe-botguard` (an optional *Rust* helper binary,
|
|
auto-detected on PATH) for PO-token attestation if YouTube starts
|
|
demanding it, and persists client state in a cache file. Accepted risk,
|
|
revisit if streams start failing.
|
|
|
|
## D1 — Extractor seam
|
|
|
|
The subprocess `Engine` is replaced by an `Extractor` trait owned by
|
|
`ytdy` (search videos, video details, audio stream URL, saved
|
|
playlists, playlist videos) with two implementations:
|
|
|
|
- `RustyPipeExtractor` — the real one, wrapping one `RustyPipe` client.
|
|
- a test fake — provider logic (path scheme, search-term store, node
|
|
shapes, gating) is tested without network or fake shell scripts.
|
|
|
|
`Client` keeps its public `ProviderClient` surface, path scheme, and
|
|
the in-memory search-term store unchanged.
|
|
|
|
```d2
|
|
direction: right
|
|
tui -> server -> ytdy: "/youtube/..."
|
|
ytdy: {
|
|
client: "Client\n(paths, search terms, nodes)"
|
|
extractor: "Extractor trait"
|
|
client -> extractor
|
|
}
|
|
ytdy.extractor -> rustypipe: "RustyPipeExtractor"
|
|
rustypipe -> youtube: "Innertube (HTTPS)"
|
|
tests -> ytdy.extractor: "FakeExtractor"
|
|
```
|
|
|
|
## D2 — Stream selection (the playback fix)
|
|
|
|
`audio_stream_url` prefers the highest-average-bitrate `audio/mp4`
|
|
stream (AAC — decodable by the player); only if none exists does it
|
|
fall back to the overall best audio stream, with a warning (it will
|
|
likely not decode locally, but the URL is still honest — e.g. a future
|
|
player may cope). Download captures inherit the same choice, so their
|
|
audio is `.m4a`.
|
|
|
|
## D3 — Login and playlists
|
|
|
|
The `cookies` setting keeps its meaning: a path to a Netscape
|
|
`cookies.txt` export. rustypipe consumes it natively
|
|
(`user_auth_set_cookie_txt`) and — importantly — **persists the rotated
|
|
cookie in its cache file**, which outlives the (quickly stale) original
|
|
export. Init order: if the cache already holds a working login
|
|
(`user_auth_check_cookie`), use it; otherwise load the configured file;
|
|
on any failure degrade to logged-out with a warning (never a failed
|
|
init). "Logged in" gates the `playlists` subtree exactly as before,
|
|
now backed by `saved_playlists()` (the `userdata` feature) instead of a
|
|
live-unvalidated `feed/playlists` scrape.
|
|
|
|
Playlist nodes page through `Paginator::extend_limit` up to a track
|
|
cap (`MAX_PLAYLIST_TRACKS`, 1000) instead of loading whole playlists
|
|
blindly.
|
|
|
|
## D4 — Configuration and environment
|
|
|
|
`ytdy.toml`: `binary` is gone (nothing to spawn); `cookies` and
|
|
`search_results` stay; `call_timeout_secs` maps to the rustypipe
|
|
client timeout. New `RustyPipe` client state lives in
|
|
`<config>/crabidy/rustypipe/` (`storage_dir`) — it holds the rotated
|
|
auth cookie, so it is as secret as the cookies file; neither its
|
|
contents nor cookie values are ever logged. `yt-dlp` leaves
|
|
`devenv.nix`. Init never probes the network except when validating a
|
|
configured login; a failed login check degrades, a broken client build
|
|
disables the provider non-fatally (as before).
|
|
|
|
## D5 — Out of scope
|
|
|
|
- Installing `rustypipe-botguard` (optional PO-token helper); document
|
|
only. Streams work without it today.
|
|
- Opus support in the player (a symphonia Opus decoder does not exist;
|
|
an opus feature via a different rodio decoder is a separate project).
|
|
- YouTube Music (rustypipe supports it; nothing in crabidy asks yet).
|
|
|
|
## Risks
|
|
|
|
- Innertube changes can break rustypipe between releases; mitigations:
|
|
cache-backed client data, optional botguard, active upstream.
|
|
- `saved_playlists` needs valid cookies; YouTube rotates them — the
|
|
cache keeps the rotated value, but a long-cold server may need a
|
|
fresh export. Degrades to logged-out, never fails.
|