crabidy/architecture/rss-provider.md

RSS provider — /rss

Subscribe to podcast feeds by URL and play their episodes. Mounted at /rss by a new rssdy crate. Listings are never cached: every visit to a subscription fetches the feed, so a new episode shows up the moment it is published.

Context and problem statement

/fyyd already plays podcasts, but it is a discovery provider: it searches fyyd's public directory and gets episode metadata from fyyd's API. It cannot play a feed fyyd does not index, and it cannot play a private feed at all.

Plenty of podcasts are distributed as a plain RSS URL, and paid ones as a per-subscriber URL with a token in it:

https://feeds.economist.com/v1/rss/the-economist-podcasts/f74365b0-…

That URL is the credential. Anyone holding it has the subscription.

So the ask is: a list of feeds I subscribe to, each browsable as a folder of episodes, always showing the newest ones, and safe to use with tokened URLs.

Assumptions (decided)

• Subscriptions are configuration, not discovery. There is no feed search; you name the feeds you want. % on /rss adds one from a pasted URL and persists it (D5), which is how you "subscribe to several" without hand-editing TOML.
• A feed URL is a secret, because a premium URL is a bearer credential. It is redacted from Debug and never logged, and — the decision that shapes the whole tree — it never appears in a library path (D1).
• Nothing about feed content is cached (D2). This is the explicit requirement, and it has a client-side half people forget: both clients cache library listings by path, so the server being fresh is not enough.
• Feeds are read-only. No OPML import, no per-episode read state, no "unplayed" tracking — this is a player, not a podcatcher.

Options considered

The parser

1. feed-rs 2.4 (chosen) — one crate for RSS 2.0, RSS 1.0, RSS 0.x, Atom and JSON Feed, with the iTunes extensions mapped into its model. Podcast feeds in the wild are not uniformly RSS 2.0, and this is the only option that does not make us care.
2. rss + atom_syndication — two crates, two models, and we write the normalisation. Rejected: more code for less coverage.
3. quick-xml by hand — full control, and a long tail of real-world feed quirks to discover the hard way. Rejected.

Episode identity — the interesting problem

A track path has to survive being stored: it goes into the queue, is persisted in current, and a bookmark (w) writes it into a toml that is replayed days later. It is also displayed. So the path may not contain the feed URL, and it should not contain an arbitrary publisher-chosen id verbatim (guids are unbounded and can be URLs themselves).

1. Enclosure URL in the path (base64 or percent-encoded). Resolves with no feed fetch, and a bookmark keeps working forever. Rejected on two counts: a premium enclosure URL leaks its token into the UI, the logs, and every saved queue; and paths become unreadable.
2. Index in the feed (/rss/economist/3). Stable only until the next episode is published, which is the one thing guaranteed to happen. Rejected — a bookmark would silently point at a different episode.
3. A short hash of the episode's guid (chosen), falling back to the enclosure URL when a feed omits the guid: /rss/<slug>/<16 hex>. Short, readable, token-free, and stable for as long as the publisher keeps the guid stable (which is what guids are for). Resolving one means fetching the feed and matching the hash — which we are doing anyway (D2), and the memo (D3) keeps it to one fetch per action. blake3 is already a workspace dependency, and using it here keeps the hash stable across builds (DefaultHasher explicitly is not).

Freshness versus hammering the feed

"Always fetch" taken literally means listing a 40-episode feed, queueing it, and playing a track costs 40+ feed fetches. Options:

1. A TTL cache. Rejected: it turns the requirement into a timing question ("is 30 seconds fresh?") and gets it wrong twice.
2. A listing-driven memo (chosen, D3). A listing always fetches and replaces the memo entry for that feed. Track lookups (get_urls_for_track, get_metadata_for_track, resolve_tracks_into) read the memo and only fetch when they miss. Freshness is exactly "what a visit shows", with no clock involved, and an episode you can see is an episode you can play.

Decisions

D1 — Subscriptions are (name, url) pairs in rss.toml; the path carries a slug of the name.

# Per-subscription. `name` is yours; the path uses a slug of it.
[[feeds]]
name = "The Economist Podcasts"
url = "https://feeds.economist.com/v1/rss/…/f74365b0-…"

[[feeds]]
name = "Cautionary Tales"
url = "https://feeds.example.org/cautionary-tales"

/rss/the-economist-podcasts is the node. The URL stays in the config file and never reaches a path, a log line, or a saved queue. Duplicate slugs get a numeric suffix; a feed whose entry has no usable url is skipped with a warning at load.

D2 — No caching of feed content, at either end.

• The provider holds no listing cache: get_lib_node("/rss/<slug>") fetches.
• /rss joins MUTABLE_ROOTS in both clients (cbd-tui/src/rpc.rs, cbd-web/src/state.rs), which is what actually makes a re-visit re-fetch. Without this the client answers from its own cache and the server's freshness is invisible.

D3 — One memo, written by listings, read by track lookups. A bounded map (last 8 feeds) of slug -> episodes, replaced on every listing of that feed. Track lookups consult it and fetch on a miss. It is never consulted to answer a listing.

D4 — Episode key is blake3(guid)[..16], or blake3(enclosure_url) when the feed omits a guid. Track.provider_item_id is set to the guid, so the content store de-duplicates captures of the same episode across visits and across feeds.

D5 — %, e, d manage subscriptions. % on /rss takes a pasted feed URL, fetches it once, names the subscription from the feed's own title (slug-deduped), appends it to rss.toml and writes the file back — the same shape as SoundCloud's resolve node taking a URL as its "title", and the same write-back the other providers already do for their settings. e renames a subscription (the slug, and so the path, changes with it). d unsubscribes: it removes the config entry and touches no audio. A capture made from it stays under /crabidy.

D6 — Bounded and defensive. Per-request timeout (call_timeout_secs, default 30), a response-size cap (max_feed_bytes, default 8 MiB) so a hostile or broken feed cannot exhaust memory, and episodes_per_feed (default 200). A malformed entry — no enclosure, no title, unparseable date — is skipped with a warning; only a feed that cannot be fetched or parsed at all is an error, and it fails just that node, never the server.

D7 — Behind the rss cargo feature, default on, exactly like the other providers: dep:rssdy, a BUILT_IN_PROVIDERS entry, a ProviderToggles field, a name in ALL_PROVIDERS, and a row in check-features.

D8 — A separate crate, not part of fyyd. They share only "an episode is a track whose audio is an enclosure URL". Their config, identity model, and caching differ completely; folding them together would mean one crate with two personalities. If a third podcast source ever lands, the shared piece to extract is the episode→Track mapping, not the provider.

D9 — Newest first. Episodes are sorted by publication date descending where dates parse, keeping feed order for the rest (a stable sort, so a dateless feed lists exactly as published). Queueing a subscription therefore plays newest first.

D10 — Episodes stream directly, like fyyd's. The enclosure URL is a plain media file; the existing player streams it with no sidecar and no new player component. Captures (W) download it into the content store.

Structure

direction: right

config: "rss.toml\n[[feeds]] name + url\n(url is a credential)" {
  shape: document
}

rssdy: "rssdy (/rss)" {
  subs: "subscriptions\nname -> url"
  memo: "episode memo\nslug -> episodes\n(last 8, listing-written)"
  parse: "feed-rs\nRSS/Atom/JSON Feed"
}

feeds: "publisher feeds\n(https, tokened URLs)" { shape: cloud }
player: "audio player\n(enclosure URL)"
clients: "cbd-tui · cbd-web\n/rss is never cached client-side"

config -> rssdy.subs: loaded at init
clients -> rssdy: "GetLibraryNode(/rss/<slug>)"
rssdy.subs -> feeds: "fetch on every listing"
feeds -> rssdy.parse
rssdy.parse -> rssdy.memo: replaces the entry
rssdy.memo -> player: "enclosure URL for a track key"

Listing versus playing — the same feed, two paths through it:

direction: down

visit: "visit /rss/<slug>"
fetch: "fetch + parse the feed"
memo_w: "replace memo[slug]"
list: "list episodes, newest first\npath = /rss/<slug>/<hash(guid)>"

play: "play or queue an episode"
memo_r: "memo[slug] hit?"
enclosure: "enclosure URL -> player"

visit -> fetch
fetch -> memo_w
memo_w -> list

play -> memo_r
memo_r -> enclosure: hit
memo_r -> fetch: "miss (restart, or a\nbookmark replayed later)"

Boundaries and interfaces

• rssdy::Client implements ProviderClient like every other provider: PROVIDER_ROOT = "/rss", init(&str) -> Settings with write-back, and the node/track methods. Nothing new at the boundary.
• rssdy::api::Feeds trait wraps all network access (one method: fetch and parse a feed URL), so the tree, slug, identity, and ordering logic is unit-tested over a fake with no network — the pattern fyyd/absdy/soundclouddy already use.
• Server — one more optional dependency and one more mount registration. The mount registry means no dispatch code changes.
• Clients — one entry in each MUTABLE_ROOTS. No UI work: a subscription is a queueable node like any other.

Risks

• A token leaking through a path or a log is the failure that actually matters. D1 and D4 keep URLs out of paths structurally, and the redaction rule matches what SoundCloud and Jamendo already do for signed media URLs.
• Guid instability. A publisher that regenerates guids per fetch breaks bookmarks (the episode key changes). Nothing can fix that from our side; W captures the audio, which survives it.
• Episodes ageing out of a feed. A bookmark to an episode the feed no longer lists cannot resolve. Inherent to RSS; documented, and again the reason to capture rather than bookmark things you want to keep.
• itunes:duration coverage. Durations come from whatever the feed provides through feed-rs; absent duration is None, which the UI already handles. To be confirmed against a real feed at implementation time rather than assumed.
• Feed size. D6's byte cap is the guard; a 40 MiB feed is a bug in someone else's publishing, not something to load into memory.

Open questions

None blocking. Deferred by choice: OPML import/export, per-episode played state, feed refresh in the background (there is no server-side polling — a listing is the refresh), and itunes:image/artwork, which the library model has no field for.