architecture: design the /rss subscription provider
Podcast feeds you subscribe to by URL, including premium per-subscriber URLs, with listings that are never cached. Two findings shape the design. First, a premium feed URL *is* the credential, and library paths are displayed, logged, and persisted into saved queues and bookmarks — so the URL can never appear in one. Paths therefore carry a slug of the subscription name plus a short blake3 hash of the episode guid, and subscriptions live in rss.toml rather than being addressed by URL. Second, "not cached" has a client-side half: both clients cache library listings by path and only /crabidy, /fs and /orphans bypass it. Without adding /rss to those lists, a re-visit answers from the client's cache and the server's freshness is invisible. A listing always fetches; a listing-written memo (read only when resolving a track) keeps queueing 40 episodes to one fetch instead of 40, with no TTL to guess at. Also picks feed-rs over hand-rolled parsing (RSS 2.0/1.0/0.x, Atom and JSON Feed in one maintained crate — real podcast feeds are not uniform), and records the risks that cannot be engineered away: publishers who regenerate guids break bookmarks, and episodes ageing out of a feed cannot be resolved. Capture rather than bookmark what you want to keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Test User
parent
1b2f01a2b6
commit
bdc961501d
|
|
@ -0,0 +1,263 @@
|
|||
# RSS provider — /rss
|
||||
|
||||
Subscribe to podcast feeds by URL and play their episodes. Mounted at
|
||||
`/rss` by a new `rssdy` crate. Listings are **never cached**: every visit
|
||||
to a subscription fetches the feed, so a new episode shows up the moment
|
||||
it is published.
|
||||
|
||||
## Context and problem statement
|
||||
|
||||
`/fyyd` already plays podcasts, but it is a *discovery* provider: it
|
||||
searches fyyd's public directory and gets episode metadata from fyyd's
|
||||
API. It cannot play a feed fyyd does not index, and it cannot play a
|
||||
**private** feed at all.
|
||||
|
||||
Plenty of podcasts are distributed as a plain RSS URL, and paid ones as a
|
||||
per-subscriber URL with a token in it:
|
||||
|
||||
```text
|
||||
https://feeds.economist.com/v1/rss/the-economist-podcasts/f74365b0-…
|
||||
```
|
||||
|
||||
That URL *is* the credential. Anyone holding it has the subscription.
|
||||
|
||||
So the ask is: a list of feeds I subscribe to, each browsable as a folder
|
||||
of episodes, always showing the newest ones, and safe to use with
|
||||
tokened URLs.
|
||||
|
||||
## Assumptions (decided)
|
||||
|
||||
- **Subscriptions are configuration, not discovery.** There is no feed
|
||||
search; you name the feeds you want. `%` on `/rss` adds one from a
|
||||
pasted URL and persists it (D5), which is how you "subscribe to
|
||||
several" without hand-editing TOML.
|
||||
- **A feed URL is a secret**, because a premium URL is a bearer
|
||||
credential. It is redacted from `Debug` and never logged, and — the
|
||||
decision that shapes the whole tree — **it never appears in a library
|
||||
path** (D1).
|
||||
- **Nothing about feed content is cached** (D2). This is the explicit
|
||||
requirement, and it has a client-side half people forget: both clients
|
||||
cache library listings by path, so the server being fresh is not
|
||||
enough.
|
||||
- Feeds are read-only. No OPML import, no per-episode read state, no
|
||||
"unplayed" tracking — this is a player, not a podcatcher.
|
||||
|
||||
## Options considered
|
||||
|
||||
### The parser
|
||||
|
||||
1. **`feed-rs` 2.4** (chosen) — one crate for RSS 2.0, RSS 1.0, RSS
|
||||
0.x, Atom and JSON Feed, with the iTunes extensions mapped into its
|
||||
model. Podcast feeds in the wild are not uniformly RSS 2.0, and this
|
||||
is the only option that does not make us care.
|
||||
2. `rss` + `atom_syndication` — two crates, two models, and we write the
|
||||
normalisation. Rejected: more code for less coverage.
|
||||
3. `quick-xml` by hand — full control, and a long tail of real-world
|
||||
feed quirks to discover the hard way. Rejected.
|
||||
|
||||
### Episode identity — the interesting problem
|
||||
|
||||
A track path has to survive being **stored**: it goes into the queue, is
|
||||
persisted in `current`, and a bookmark (`w`) writes it into a toml that
|
||||
is replayed days later. It is also *displayed*. So the path may not
|
||||
contain the feed URL, and it should not contain an arbitrary
|
||||
publisher-chosen id verbatim (guids are unbounded and can be URLs
|
||||
themselves).
|
||||
|
||||
1. **Enclosure URL in the path** (base64 or percent-encoded). Resolves
|
||||
with no feed fetch, and a bookmark keeps working forever. Rejected on
|
||||
two counts: a premium enclosure URL leaks its token into the UI, the
|
||||
logs, and every saved queue; and paths become unreadable.
|
||||
2. **Index in the feed** (`/rss/economist/3`). Stable only until the next
|
||||
episode is published, which is the one thing guaranteed to happen.
|
||||
Rejected — a bookmark would silently point at a different episode.
|
||||
3. **A short hash of the episode's guid** (chosen), falling back to the
|
||||
enclosure URL when a feed omits the guid: `/rss/<slug>/<16 hex>`.
|
||||
Short, readable, token-free, and stable for as long as the publisher
|
||||
keeps the guid stable (which is what guids are for). Resolving one
|
||||
means fetching the feed and matching the hash — which we are doing
|
||||
anyway (D2), and the memo (D3) keeps it to one fetch per action.
|
||||
`blake3` is already a workspace dependency, and using it here keeps
|
||||
the hash stable across builds (`DefaultHasher` explicitly is not).
|
||||
|
||||
### Freshness versus hammering the feed
|
||||
|
||||
"Always fetch" taken literally means listing a 40-episode feed, queueing
|
||||
it, and playing a track costs 40+ feed fetches. Options:
|
||||
|
||||
1. A TTL cache. Rejected: it turns the requirement into a timing
|
||||
question ("is 30 seconds fresh?") and gets it wrong twice.
|
||||
2. **A listing-driven memo** (chosen, D3). A *listing* always fetches and
|
||||
replaces the memo entry for that feed. Track lookups
|
||||
(`get_urls_for_track`, `get_metadata_for_track`,
|
||||
`resolve_tracks_into`) read the memo and only fetch when they miss.
|
||||
Freshness is exactly "what a visit shows", with no clock involved, and
|
||||
an episode you can see is an episode you can play.
|
||||
|
||||
## Decisions
|
||||
|
||||
**D1 — Subscriptions are `(name, url)` pairs in `rss.toml`; the path
|
||||
carries a slug of the name.**
|
||||
|
||||
```toml
|
||||
# Per-subscription. `name` is yours; the path uses a slug of it.
|
||||
[[feeds]]
|
||||
name = "The Economist Podcasts"
|
||||
url = "https://feeds.economist.com/v1/rss/…/f74365b0-…"
|
||||
|
||||
[[feeds]]
|
||||
name = "Cautionary Tales"
|
||||
url = "https://feeds.example.org/cautionary-tales"
|
||||
```
|
||||
|
||||
`/rss/the-economist-podcasts` is the node. The URL stays in the config
|
||||
file and never reaches a path, a log line, or a saved queue. Duplicate
|
||||
slugs get a numeric suffix; a feed whose entry has no usable url is
|
||||
skipped with a warning at load.
|
||||
|
||||
**D2 — No caching of feed content, at either end.**
|
||||
|
||||
- The provider holds no listing cache: `get_lib_node("/rss/<slug>")`
|
||||
fetches.
|
||||
- **`/rss` joins `MUTABLE_ROOTS` in both clients** (`cbd-tui/src/rpc.rs`,
|
||||
`cbd-web/src/state.rs`), which is what actually makes a re-visit
|
||||
re-fetch. Without this the client answers from its own cache and the
|
||||
server's freshness is invisible.
|
||||
|
||||
**D3 — One memo, written by listings, read by track lookups.** A bounded
|
||||
map (last 8 feeds) of `slug -> episodes`, replaced on every listing of
|
||||
that feed. Track lookups consult it and fetch on a miss. It is never
|
||||
consulted to answer a listing.
|
||||
|
||||
**D4 — Episode key is `blake3(guid)[..16]`**, or `blake3(enclosure_url)`
|
||||
when the feed omits a guid. `Track.provider_item_id` is set to the guid,
|
||||
so the content store de-duplicates captures of the same episode across
|
||||
visits and across feeds.
|
||||
|
||||
**D5 — `%`, `e`, `d` manage subscriptions.** `%` on `/rss` takes a pasted
|
||||
feed URL, fetches it once, names the subscription from the feed's own
|
||||
title (slug-deduped), appends it to `rss.toml` and writes the file back —
|
||||
the same shape as SoundCloud's `resolve` node taking a URL as its
|
||||
"title", and the same write-back the other providers already do for their
|
||||
settings. `e` renames a subscription (the slug, and so the path, changes
|
||||
with it). `d` unsubscribes: it removes the config entry and touches no
|
||||
audio. A capture made from it stays under `/crabidy`.
|
||||
|
||||
**D6 — Bounded and defensive.** Per-request timeout
|
||||
(`call_timeout_secs`, default 30), a response-size cap (`max_feed_bytes`,
|
||||
default 8 MiB) so a hostile or broken feed cannot exhaust memory, and
|
||||
`episodes_per_feed` (default 200). A malformed entry — no enclosure, no
|
||||
title, unparseable date — is skipped with a warning; only a feed that
|
||||
cannot be fetched or parsed at all is an error, and it fails just that
|
||||
node, never the server.
|
||||
|
||||
**D7 — Behind the `rss` cargo feature**, default on, exactly like the
|
||||
other providers: `dep:rssdy`, a `BUILT_IN_PROVIDERS` entry, a
|
||||
`ProviderToggles` field, a name in `ALL_PROVIDERS`, and a row in
|
||||
`check-features`.
|
||||
|
||||
**D8 — A separate crate, not part of `fyyd`.** They share only "an
|
||||
episode is a track whose audio is an enclosure URL". Their config,
|
||||
identity model, and caching differ completely; folding them together
|
||||
would mean one crate with two personalities. If a third podcast source
|
||||
ever lands, the shared piece to extract is the episode→`Track` mapping,
|
||||
not the provider.
|
||||
|
||||
**D9 — Newest first.** Episodes are sorted by publication date
|
||||
descending where dates parse, keeping feed order for the rest (a stable
|
||||
sort, so a dateless feed lists exactly as published). Queueing a
|
||||
subscription therefore plays newest first.
|
||||
|
||||
**D10 — Episodes stream directly, like fyyd's.** The enclosure URL is a
|
||||
plain media file; the existing player streams it with no sidecar and no
|
||||
new player component. Captures (`W`) download it into the content store.
|
||||
|
||||
## Structure
|
||||
|
||||
```d2
|
||||
direction: right
|
||||
|
||||
config: "rss.toml\n[[feeds]] name + url\n(url is a credential)" {
|
||||
shape: document
|
||||
}
|
||||
|
||||
rssdy: "rssdy (/rss)" {
|
||||
subs: "subscriptions\nname -> url"
|
||||
memo: "episode memo\nslug -> episodes\n(last 8, listing-written)"
|
||||
parse: "feed-rs\nRSS/Atom/JSON Feed"
|
||||
}
|
||||
|
||||
feeds: "publisher feeds\n(https, tokened URLs)" { shape: cloud }
|
||||
player: "audio player\n(enclosure URL)"
|
||||
clients: "cbd-tui · cbd-web\n/rss is never cached client-side"
|
||||
|
||||
config -> rssdy.subs: loaded at init
|
||||
clients -> rssdy: "GetLibraryNode(/rss/<slug>)"
|
||||
rssdy.subs -> feeds: "fetch on every listing"
|
||||
feeds -> rssdy.parse
|
||||
rssdy.parse -> rssdy.memo: replaces the entry
|
||||
rssdy.memo -> player: "enclosure URL for a track key"
|
||||
```
|
||||
|
||||
Listing versus playing — the same feed, two paths through it:
|
||||
|
||||
```d2
|
||||
direction: down
|
||||
|
||||
visit: "visit /rss/<slug>"
|
||||
fetch: "fetch + parse the feed"
|
||||
memo_w: "replace memo[slug]"
|
||||
list: "list episodes, newest first\npath = /rss/<slug>/<hash(guid)>"
|
||||
|
||||
play: "play or queue an episode"
|
||||
memo_r: "memo[slug] hit?"
|
||||
enclosure: "enclosure URL -> player"
|
||||
|
||||
visit -> fetch
|
||||
fetch -> memo_w
|
||||
memo_w -> list
|
||||
|
||||
play -> memo_r
|
||||
memo_r -> enclosure: hit
|
||||
memo_r -> fetch: "miss (restart, or a\nbookmark replayed later)"
|
||||
```
|
||||
|
||||
## Boundaries and interfaces
|
||||
|
||||
- **`rssdy::Client`** implements `ProviderClient` like every other
|
||||
provider: `PROVIDER_ROOT = "/rss"`, `init(&str) -> Settings` with
|
||||
write-back, and the node/track methods. Nothing new at the boundary.
|
||||
- **`rssdy::api::Feeds` trait** wraps all network access (one method:
|
||||
fetch and parse a feed URL), so the tree, slug, identity, and ordering
|
||||
logic is unit-tested over a fake with no network — the pattern
|
||||
`fyyd`/`absdy`/`soundclouddy` already use.
|
||||
- **Server** — one more optional dependency and one more mount
|
||||
registration. The mount registry means no dispatch code changes.
|
||||
- **Clients** — one entry in each `MUTABLE_ROOTS`. No UI work: a
|
||||
subscription is a queueable node like any other.
|
||||
|
||||
## Risks
|
||||
|
||||
- **A token leaking through a path or a log** is the failure that
|
||||
actually matters. D1 and D4 keep URLs out of paths structurally, and
|
||||
the redaction rule matches what SoundCloud and Jamendo already do for
|
||||
signed media URLs.
|
||||
- **Guid instability.** A publisher that regenerates guids per fetch
|
||||
breaks bookmarks (the episode key changes). Nothing can fix that from
|
||||
our side; `W` captures the audio, which survives it.
|
||||
- **Episodes ageing out of a feed.** A bookmark to an episode the feed no
|
||||
longer lists cannot resolve. Inherent to RSS; documented, and again the
|
||||
reason to capture rather than bookmark things you want to keep.
|
||||
- **`itunes:duration` coverage.** Durations come from whatever the feed
|
||||
provides through `feed-rs`; absent duration is `None`, which the UI
|
||||
already handles. To be confirmed against a real feed at implementation
|
||||
time rather than assumed.
|
||||
- **Feed size.** D6's byte cap is the guard; a 40 MiB feed is a bug in
|
||||
someone else's publishing, not something to load into memory.
|
||||
|
||||
## Open questions
|
||||
|
||||
None blocking. Deferred by choice: OPML import/export, per-episode
|
||||
played state, feed refresh in the background (there is no server-side
|
||||
polling — a listing is the refresh), and `itunes:image`/artwork, which
|
||||
the library model has no field for.
|
||||
Loading…
Reference in New Issue