From bdc961501d6f69712efb827967ea28593400b883 Mon Sep 17 00:00:00 2001 From: Test User Date: Sun, 26 Jul 2026 12:19:23 +0200 Subject: [PATCH] architecture: design the /rss subscription provider MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Podcast feeds you subscribe to by URL, including premium per-subscriber URLs, with listings that are never cached. Two findings shape the design. First, a premium feed URL *is* the credential, and library paths are displayed, logged, and persisted into saved queues and bookmarks — so the URL can never appear in one. Paths therefore carry a slug of the subscription name plus a short blake3 hash of the episode guid, and subscriptions live in rss.toml rather than being addressed by URL. Second, "not cached" has a client-side half: both clients cache library listings by path and only /crabidy, /fs and /orphans bypass it. Without adding /rss to those lists, a re-visit answers from the client's cache and the server's freshness is invisible. A listing always fetches; a listing-written memo (read only when resolving a track) keeps queueing 40 episodes to one fetch instead of 40, with no TTL to guess at. Also picks feed-rs over hand-rolled parsing (RSS 2.0/1.0/0.x, Atom and JSON Feed in one maintained crate — real podcast feeds are not uniform), and records the risks that cannot be engineered away: publishers who regenerate guids break bookmarks, and episodes ageing out of a feed cannot be resolved. Capture rather than bookmark what you want to keep. Co-Authored-By: Claude Opus 5 (1M context) --- architecture/rss-provider.md | 263 +++++++++++++++++++++++++++++++++++ 1 file changed, 263 insertions(+) create mode 100644 architecture/rss-provider.md diff --git a/architecture/rss-provider.md b/architecture/rss-provider.md new file mode 100644 index 0000000..d754dc7 --- /dev/null +++ b/architecture/rss-provider.md @@ -0,0 +1,263 @@ +# RSS provider — /rss + +Subscribe to podcast feeds by URL and play their episodes. Mounted at +`/rss` by a new `rssdy` crate. Listings are **never cached**: every visit +to a subscription fetches the feed, so a new episode shows up the moment +it is published. + +## Context and problem statement + +`/fyyd` already plays podcasts, but it is a *discovery* provider: it +searches fyyd's public directory and gets episode metadata from fyyd's +API. It cannot play a feed fyyd does not index, and it cannot play a +**private** feed at all. + +Plenty of podcasts are distributed as a plain RSS URL, and paid ones as a +per-subscriber URL with a token in it: + +```text +https://feeds.economist.com/v1/rss/the-economist-podcasts/f74365b0-… +``` + +That URL *is* the credential. Anyone holding it has the subscription. + +So the ask is: a list of feeds I subscribe to, each browsable as a folder +of episodes, always showing the newest ones, and safe to use with +tokened URLs. + +## Assumptions (decided) + +- **Subscriptions are configuration, not discovery.** There is no feed + search; you name the feeds you want. `%` on `/rss` adds one from a + pasted URL and persists it (D5), which is how you "subscribe to + several" without hand-editing TOML. +- **A feed URL is a secret**, because a premium URL is a bearer + credential. It is redacted from `Debug` and never logged, and — the + decision that shapes the whole tree — **it never appears in a library + path** (D1). +- **Nothing about feed content is cached** (D2). This is the explicit + requirement, and it has a client-side half people forget: both clients + cache library listings by path, so the server being fresh is not + enough. +- Feeds are read-only. No OPML import, no per-episode read state, no + "unplayed" tracking — this is a player, not a podcatcher. + +## Options considered + +### The parser + +1. **`feed-rs` 2.4** (chosen) — one crate for RSS 2.0, RSS 1.0, RSS + 0.x, Atom and JSON Feed, with the iTunes extensions mapped into its + model. Podcast feeds in the wild are not uniformly RSS 2.0, and this + is the only option that does not make us care. +2. `rss` + `atom_syndication` — two crates, two models, and we write the + normalisation. Rejected: more code for less coverage. +3. `quick-xml` by hand — full control, and a long tail of real-world + feed quirks to discover the hard way. Rejected. + +### Episode identity — the interesting problem + +A track path has to survive being **stored**: it goes into the queue, is +persisted in `current`, and a bookmark (`w`) writes it into a toml that +is replayed days later. It is also *displayed*. So the path may not +contain the feed URL, and it should not contain an arbitrary +publisher-chosen id verbatim (guids are unbounded and can be URLs +themselves). + +1. **Enclosure URL in the path** (base64 or percent-encoded). Resolves + with no feed fetch, and a bookmark keeps working forever. Rejected on + two counts: a premium enclosure URL leaks its token into the UI, the + logs, and every saved queue; and paths become unreadable. +2. **Index in the feed** (`/rss/economist/3`). Stable only until the next + episode is published, which is the one thing guaranteed to happen. + Rejected — a bookmark would silently point at a different episode. +3. **A short hash of the episode's guid** (chosen), falling back to the + enclosure URL when a feed omits the guid: `/rss//<16 hex>`. + Short, readable, token-free, and stable for as long as the publisher + keeps the guid stable (which is what guids are for). Resolving one + means fetching the feed and matching the hash — which we are doing + anyway (D2), and the memo (D3) keeps it to one fetch per action. + `blake3` is already a workspace dependency, and using it here keeps + the hash stable across builds (`DefaultHasher` explicitly is not). + +### Freshness versus hammering the feed + +"Always fetch" taken literally means listing a 40-episode feed, queueing +it, and playing a track costs 40+ feed fetches. Options: + +1. A TTL cache. Rejected: it turns the requirement into a timing + question ("is 30 seconds fresh?") and gets it wrong twice. +2. **A listing-driven memo** (chosen, D3). A *listing* always fetches and + replaces the memo entry for that feed. Track lookups + (`get_urls_for_track`, `get_metadata_for_track`, + `resolve_tracks_into`) read the memo and only fetch when they miss. + Freshness is exactly "what a visit shows", with no clock involved, and + an episode you can see is an episode you can play. + +## Decisions + +**D1 — Subscriptions are `(name, url)` pairs in `rss.toml`; the path +carries a slug of the name.** + +```toml +# Per-subscription. `name` is yours; the path uses a slug of it. +[[feeds]] +name = "The Economist Podcasts" +url = "https://feeds.economist.com/v1/rss/…/f74365b0-…" + +[[feeds]] +name = "Cautionary Tales" +url = "https://feeds.example.org/cautionary-tales" +``` + +`/rss/the-economist-podcasts` is the node. The URL stays in the config +file and never reaches a path, a log line, or a saved queue. Duplicate +slugs get a numeric suffix; a feed whose entry has no usable url is +skipped with a warning at load. + +**D2 — No caching of feed content, at either end.** + +- The provider holds no listing cache: `get_lib_node("/rss/")` + fetches. +- **`/rss` joins `MUTABLE_ROOTS` in both clients** (`cbd-tui/src/rpc.rs`, + `cbd-web/src/state.rs`), which is what actually makes a re-visit + re-fetch. Without this the client answers from its own cache and the + server's freshness is invisible. + +**D3 — One memo, written by listings, read by track lookups.** A bounded +map (last 8 feeds) of `slug -> episodes`, replaced on every listing of +that feed. Track lookups consult it and fetch on a miss. It is never +consulted to answer a listing. + +**D4 — Episode key is `blake3(guid)[..16]`**, or `blake3(enclosure_url)` +when the feed omits a guid. `Track.provider_item_id` is set to the guid, +so the content store de-duplicates captures of the same episode across +visits and across feeds. + +**D5 — `%`, `e`, `d` manage subscriptions.** `%` on `/rss` takes a pasted +feed URL, fetches it once, names the subscription from the feed's own +title (slug-deduped), appends it to `rss.toml` and writes the file back — +the same shape as SoundCloud's `resolve` node taking a URL as its +"title", and the same write-back the other providers already do for their +settings. `e` renames a subscription (the slug, and so the path, changes +with it). `d` unsubscribes: it removes the config entry and touches no +audio. A capture made from it stays under `/crabidy`. + +**D6 — Bounded and defensive.** Per-request timeout +(`call_timeout_secs`, default 30), a response-size cap (`max_feed_bytes`, +default 8 MiB) so a hostile or broken feed cannot exhaust memory, and +`episodes_per_feed` (default 200). A malformed entry — no enclosure, no +title, unparseable date — is skipped with a warning; only a feed that +cannot be fetched or parsed at all is an error, and it fails just that +node, never the server. + +**D7 — Behind the `rss` cargo feature**, default on, exactly like the +other providers: `dep:rssdy`, a `BUILT_IN_PROVIDERS` entry, a +`ProviderToggles` field, a name in `ALL_PROVIDERS`, and a row in +`check-features`. + +**D8 — A separate crate, not part of `fyyd`.** They share only "an +episode is a track whose audio is an enclosure URL". Their config, +identity model, and caching differ completely; folding them together +would mean one crate with two personalities. If a third podcast source +ever lands, the shared piece to extract is the episode→`Track` mapping, +not the provider. + +**D9 — Newest first.** Episodes are sorted by publication date +descending where dates parse, keeping feed order for the rest (a stable +sort, so a dateless feed lists exactly as published). Queueing a +subscription therefore plays newest first. + +**D10 — Episodes stream directly, like fyyd's.** The enclosure URL is a +plain media file; the existing player streams it with no sidecar and no +new player component. Captures (`W`) download it into the content store. + +## Structure + +```d2 +direction: right + +config: "rss.toml\n[[feeds]] name + url\n(url is a credential)" { + shape: document +} + +rssdy: "rssdy (/rss)" { + subs: "subscriptions\nname -> url" + memo: "episode memo\nslug -> episodes\n(last 8, listing-written)" + parse: "feed-rs\nRSS/Atom/JSON Feed" +} + +feeds: "publisher feeds\n(https, tokened URLs)" { shape: cloud } +player: "audio player\n(enclosure URL)" +clients: "cbd-tui · cbd-web\n/rss is never cached client-side" + +config -> rssdy.subs: loaded at init +clients -> rssdy: "GetLibraryNode(/rss/)" +rssdy.subs -> feeds: "fetch on every listing" +feeds -> rssdy.parse +rssdy.parse -> rssdy.memo: replaces the entry +rssdy.memo -> player: "enclosure URL for a track key" +``` + +Listing versus playing — the same feed, two paths through it: + +```d2 +direction: down + +visit: "visit /rss/" +fetch: "fetch + parse the feed" +memo_w: "replace memo[slug]" +list: "list episodes, newest first\npath = /rss//" + +play: "play or queue an episode" +memo_r: "memo[slug] hit?" +enclosure: "enclosure URL -> player" + +visit -> fetch +fetch -> memo_w +memo_w -> list + +play -> memo_r +memo_r -> enclosure: hit +memo_r -> fetch: "miss (restart, or a\nbookmark replayed later)" +``` + +## Boundaries and interfaces + +- **`rssdy::Client`** implements `ProviderClient` like every other + provider: `PROVIDER_ROOT = "/rss"`, `init(&str) -> Settings` with + write-back, and the node/track methods. Nothing new at the boundary. +- **`rssdy::api::Feeds` trait** wraps all network access (one method: + fetch and parse a feed URL), so the tree, slug, identity, and ordering + logic is unit-tested over a fake with no network — the pattern + `fyyd`/`absdy`/`soundclouddy` already use. +- **Server** — one more optional dependency and one more mount + registration. The mount registry means no dispatch code changes. +- **Clients** — one entry in each `MUTABLE_ROOTS`. No UI work: a + subscription is a queueable node like any other. + +## Risks + +- **A token leaking through a path or a log** is the failure that + actually matters. D1 and D4 keep URLs out of paths structurally, and + the redaction rule matches what SoundCloud and Jamendo already do for + signed media URLs. +- **Guid instability.** A publisher that regenerates guids per fetch + breaks bookmarks (the episode key changes). Nothing can fix that from + our side; `W` captures the audio, which survives it. +- **Episodes ageing out of a feed.** A bookmark to an episode the feed no + longer lists cannot resolve. Inherent to RSS; documented, and again the + reason to capture rather than bookmark things you want to keep. +- **`itunes:duration` coverage.** Durations come from whatever the feed + provides through `feed-rs`; absent duration is `None`, which the UI + already handles. To be confirmed against a real feed at implementation + time rather than assumed. +- **Feed size.** D6's byte cap is the guard; a 40 MiB feed is a bug in + someone else's publishing, not something to load into memory. + +## Open questions + +None blocking. Deferred by choice: OPML import/export, per-episode +played state, feed refresh in the background (there is no server-side +polling — a listing is the refresh), and `itunes:image`/artwork, which +the library model has no field for.