crabidy/architecture/soundcloud-provider.md

# soundcloud provider (streaming music)

## Context and problem statement

A new library provider mounted at `/soundcloud` that lets a user **search,
resolve share links, and play** SoundCloud tracks and playlists — and, **when
they opt in with a token, browse their own likes and playlists**.

- Like fyyd/tidal/youtube, SoundCloud is a **remote, search-driven** service.
  Unlike them, SoundCloud offers **no official public API**: the modern
  `api-v2.soundcloud.com` requires a `client_id` that SoundCloud embeds in its
  web app and **rotates periodically**, and personal-account access needs an
  **OAuth token**. So the provider must (a) obtain a `client_id` on its own and
  survive rotation, and (b) treat login as **optional** — public browse + play
  works with only a `client_id`; a token merely adds personal nodes.
- Content is organized as **tracks** and **playlists** (a playlist is a
  container of tracks). There is no per-track container level like abs books —
  the tree is `search-term → tracks`, `playlist → tracks`, and (logged in)
  `likes → tracks` / `playlists → playlist → tracks`, plus a **resolve** entry
  that turns a pasted permalink URL into a track or playlist.
- **Playing** a track is the biggest divergence from every existing provider.
  SoundCloud does not serve a plain file URL: each track carries a set of
  `media.transcodings`, and the playable ones are **HLS** — an `.m3u8` playlist
  of short **mp3 segments**. crabidy's player streams a single byte source, so
  this requires a new **HLS source** in `audio-player` that fetches the playlist
  and streams the mp3 segments in order as one continuous mp3 (mp3 frames
  byte-concatenate into a valid stream — the same fact `ffmpeg -c copy` relies
  on). rodio's existing symphonia mp3 path then decodes it, unchanged.
- **Captures** (`W`, download) come for free once nodes serve tracks and raise
  `is_downloadable`, exactly as for abs/fyyd — no wire or TUI work.

## The SoundCloud API (grounding)

Base `https://api-v2.soundcloud.com`. **Every** request carries `?client_id`
(plus `app_version`, `app_locale=en`), omitted from the cells below; personal
calls also send `Authorization: OAuth <token>`. `search/*` add
`&limit=<n>&offset=<o>&linked_partitioning=1`. Endpoints we use:

| Purpose | Endpoint |
| --- | --- |
| Resolve a permalink URL | `GET /resolve?url=<permalink>` |
| Search tracks | `GET /search/tracks?q=<t>` |
| Search playlists | `GET /search/playlists?q=<t>` |
| Track detail | `GET /tracks/<id>` |
| Playlist detail | `GET /playlists/<id>` |
| Transcoding → media URL | `GET <transcoding.url>` → `{"url": "<m3u8>"}` |
| (login) My likes | `GET /me/likes/tracks` (OAuth) |
| (login) My playlists | `GET /me/playlists` (OAuth) |

Objects (fields we read):

- **track**: `id` (numeric), `title`, `user.username` (→ artist), `duration`
  (ms), `permalink_url`, `media.transcodings[]`, `policy`/`streamable`,
  `publisher_metadata` (optional album/release).
- **transcoding**: `url` (a second API URL, not the CDN), `preset`,
  `format.{protocol, mime_type}`, `quality`. We select
  `protocol == "hls" && mime_type == "audio/mpeg"` (mp3-HLS), which SoundCloud
  offers for essentially every playable track.
- **playlist**: `id`, `title`, `user.username`, `tracks[]` — often returned as
  **stubs** (`{id}` only); missing tracks are hydrated in batches of ≤50 via
  `GET /tracks?ids=<csv>&client_id=…`.
- **resolve**: returns a track or a playlist object (discriminated by `kind`).

`client_id` acquisition (no login): `GET https://soundcloud.com`, find the
referenced JS bundles, fetch them, regex `client_id:"(\w+)"`; `app_version`
from `window.__sc_version="(\d+)"`. This is exactly the streamrip approach.

## Assumptions (decided here)

- The captures/creatable/editable/deletable TUI flows are provider-agnostic
  (confirmed by `/youtube`, `/fyyd`, `/abs`): search-term + resolve semantics
  cost no TUI or wire change. No proto change, no new `ProviderCommand`.
- A missing/rotated/invalid `client_id` must never crash startup or a browse.
  The provider **self-heals** by scraping and by re-scraping on `401/403`; only
  if scraping itself fails does the `/soundcloud` subtree degrade (typed
  errors, skipped tracks), never the app.
- **Login is optional.** With no `oauth_token`, personal nodes (`likes`,
  `playlists`) are simply **not shown**; public search/resolve/play still work.
  A token unlocks the personal nodes and is refreshed/persisted like tidal's.
- HLS media/segment URLs and the `client_id`/`oauth_token` are **secrets or
  ephemeral signed URLs**: redact from `Debug`/config dumps, never log the built
  stream/segment URLs (hard rule: redact secrets from logs and error reports).
- SoundCloud `id`s are numeric (URL-safe); only user-typed **search terms** and
  **pasted URLs** are percent-encoded into a path segment.

## Decisions

### D1 — Crate `soundclouddy`, mounted at `/soundcloud`, non-fatal init

New workspace crate `soundclouddy` implementing `ProviderClient`, shaped on
`fyyd`/`absdy` (remote, search-driven, plain leaf tracks). Wired into
`ProviderOrchestrator` with a `sc_client: Option<Arc<soundclouddy::Client>>`
field, `sc_owns()`/`sc_provider()` helpers, a `build()` block that reads
`soundcloud.toml` (non-fatal), a `get_lib_root` child gated on
`self.sc_client.is_some()`, and one routing arm in each dispatch method.
`crabidy-server` settings gain `soundcloud` in `ALL_PROVIDERS` (now 8), in
`ProviderToggles`, in `all()`, and in `provider_toggles()`. No `cli.rs`/
`main.rs` change (providers are pure path-prefix subtrees).

### D2 — HTTP behind a trait, faked in tests; client_id lifecycle inside the seam

All network access goes through one seam — an `Sc` trait (`resolve`,
`search_tracks`, `search_playlists`, `track_detail`, `playlist_detail`,
`hydrate_tracks`, `resolve_stream_url`, and, when logged in, `my_likes`,
`my_playlists`) behind `Box<dyn Sc>` — with a `reqwest`-based `ScApi` for
production and a `FakeApi` in tests (as `absdy` hides `reqwest` behind `Abs`).
The **client_id acquisition, caching, and re-scrape-on-401** live entirely
inside `ScApi` so provider logic (tree shaping, path parsing, term store) is
unit-tested with zero network. Errors map to `ProviderError::FetchError` at the
boundary; malformed paths → `MalformedPath`; empty create/rename → `InvalidInput`.

### D3 — Tree shape (search, resolve, optional personal)

Track ids and playlist ids are both numeric, so leaf/container paths use a
**type-tagged** canonical segment to disambiguate: `track/<id>` and
`playlist/<id>`. Browse nodes point their children at these canonical paths.

- `/soundcloud` — children: `search` (creatable), `resolve` (creatable), and —
  **only if logged in** — `likes` and `playlists`. Not itself queueable.
- `/soundcloud/search` / `/soundcloud/resolve` — `is_creatable`; children are
  the in-memory terms/URLs (`RwLock<Vec<String>>`, dedup), each editable and
  deletable, like tidal/youtube/fyyd/abs search terms.
- `/soundcloud/search/<term>` — matching **tracks** as queueable leaves (and,
  optionally, matching playlists as containers).
- `/soundcloud/resolve/<url>` — the resolved permalink: a single track leaf, or
  a playlist container.
- `/soundcloud/likes` (login) — the user's liked **tracks**.
- `/soundcloud/playlists` (login) — the user's playlists as containers.
- `/soundcloud/playlist/<id>` — a playlist's tracks (queueable, downloadable);
  the canonical container path, reached from search/resolve/likes/playlists.
- `/soundcloud/track/<id>` — the canonical **track leaf**. A track id alone is
  sufficient to resolve a stream, so every branch's track children point here
  and playback needs no browse context.

Terms/URLs are percent-encoded into one segment (`encode_segment`/
`decode_segment`); numeric ids are already URL-safe.

### D4 — Playback: progressive mp3 (HLS source retained as fallback)

**Revised after live testing (2026-07-24).** The original plan chose HLS mp3,
but live probing showed SoundCloud's plain `hls + audio/mpeg` transcoding
exchange **404s for anonymous clients** (every track, streamable or not), while
the **`progressive + audio/mpeg`** transcoding returns 200 with a direct,
range-streamable mp3 URL (`cf-media.sndcdn.com`, `206`, `audio/mpeg`). So the
provider now **prefers progressive**, which the player streams on its normal
windowed-HTTP path — no HLS needed for the common case. The `HlsStream` built
for the original plan is kept as a fallback for any track that offers only HLS.

- `get_urls_for_track(/soundcloud/track/<id>)`: `GET /tracks/<id>`, pick the
  best mp3 transcoding (`progressive` first, then `hls`), `GET
  <transcoding.url>?client_id=…` → the media URL, and **return it as `urls[0]`**
  (the player consumes only the first). One API round-trip — unlike abs's pure
  string-building — because the media URL is signed and ephemeral. A track
  whose exchange 404s (Go+/label preview, geo-blocked) surfaces
  `NotStreamable`/`NotFound` and is skipped, never a crash.
- **New `audio-player` component `HlsStream`** — a `stream-download`
  `SourceStream`, sibling to `WindowedHttpStream`: on create it fetches the
  `.m3u8` (a media playlist), parses `#EXTINF`/segment URIs (resolving relative
  URIs against the playlist URL, and following one level if handed a master
  playlist); on poll it streams each mp3 segment's bytes in order, advancing at
  segment boundaries, finishing after the last. The concatenated bytes are a
  valid mp3 → rodio's symphonia mp3 decoder handles them unchanged.
- **Routing**: `open_source` selects `HlsStream` when the URL path ends in
  `.m3u8` (SoundCloud's media URLs carry it); all other http URLs keep the
  windowed-HTTP path, and content-sniffing (opus vs the rest) is unchanged
  downstream. `#EXTM3U` content-sniff is a hardening fallback if needed.
- **Duration** comes from the track metadata (`duration` ms → `Track.duration`),
  not from the stream, so the seek bar is correct even though the concatenated
  HLS stream carries no container duration.
- Track fields: `title` = track title, `artist` = `user.username`, `album` from
  `publisher_metadata` when present else `None`, `duration` = ms→`Option<u32>`,
  `provider_item_id` = `"track:<id>"` (keys the capture store).

### D5 — Auth: client_id self-heal, optional OAuth login

- `Settings`: `client_id: Option<String>`, `app_version: Option<String>`
  (both **cached** after first scrape and round-tripped via `settings()` so we
  don't re-scrape every start), `oauth_token: Option<String>` (optional),
  and bounds (D6). Hand-written `Debug` redacts `client_id`/`oauth_token`.
- **No login (baseline)**: if `client_id` is unset, `ScApi` scrapes it from
  `soundcloud.com` at init; on any `401/403` it re-scrapes **once** and retries
  (rotation recovery). The freshly scraped id is persisted.
- **Optional login**: if `oauth_token` is present, `get_lib_root` adds `likes`
  and `playlists`, and personal calls send `Authorization: OAuth <token>`. If a
  refresh-token flow is configured later it mirrors tidal's persist-on-refresh;
  v1 accepts a static token and, on `401`, drops the personal subtree with a
  typed error (never a crash) — public browse is unaffected.

### D6 — Bounds and freshness

- `search_results` (default 50), `playlist_tracks_limit` (default 500, hydrated
  in ≤50-id batches), `call_timeout_secs` (default 30) bound each HTTP call, and
  `hls_total_deadline_secs` (default 300) bounds a whole HLS fetch (segments are
  retried with jitter under this deadline). Caps are `log`-ged so truncation is
  visible, not silent (hard rule: no silent caps).
- Listings are fetched fresh per call (no cross-call cache), like the other
  remote providers; only search terms / resolve URLs are stored in memory. The
  chosen transcoding may be briefly cached per track id to save the extra
  round-trip on replay.

### D7 — Out of scope (explicitly)

- **opus-HLS** (`audio/ogg; codecs=opus`) and **progressive** transcodings: v1
  targets mp3-HLS uniformly (offered for ~all tracks). opus-HLS is a phase-2
  add that reuses the new `OpusSource` per segment; the `HlsStream` seam makes
  it additive.
- **HLS seeking**: v1 HLS is forward-only and reports the source as
  non-seekable, so symphonia does not attempt an end-seek (which would need a
  known byte length). In-track seek is a later add (open at a segment offset).
- Go+ / high-quality / lossless streams (need a premium account), uploads,
  comments, reposts feed, waveforms, social graph, and playback-progress sync.
- Pagination past the configured caps (one page per listing).

## Structure

```d2
direction: right

server: crabidy-server {
  orch: ProviderOrchestrator
}

sc: "soundclouddy (crate)" {
  client: "Client\n(ProviderClient)"
  terms: "search terms + resolve URLs\n(in-memory)"
  api: "ScApi\n(reqwest seam: Sc trait,\nclient_id self-heal, opt OAuth)"
  client -> terms
  client -> api
}

player: "audio-player" {
  hls: "HlsStream\n(new SourceStream)"
  dec: "rodio / symphonia\n(mp3, opus)"
  hls -> dec: "concatenated mp3 bytes"
}

scapi: "SoundCloud\napi-v2 + HLS CDN" { shape: cloud }
web: "soundcloud.com\n(HTML + JS)" { shape: cloud }

server.orch -> sc.client: "/soundcloud/..."
sc.api -> scapi: "resolve / search / tracks / transcoding (JSON, timeout)"
sc.api -> web: "scrape client_id (init, on 401)"
server.orch -> player.hls: ".m3u8 media URL"
player.hls -> scapi: "GET m3u8 + mp3 segments"
```

## Key flow: search and play a track

```d2
shape: sequence_diagram
tui: TUI
orch: Orchestrator
s: soundclouddy
api: "SoundCloud api-v2"
hls: "HlsStream (audio-player)"
cdn: "HLS CDN"

tui -> orch: "open /soundcloud/search"
tui -> orch: "create term \"boards of canada\""
orch -> s: "create_lib_node(search, term)"
s -> tui: "term stored"
tui -> orch: "open /soundcloud/search/<term>"
orch -> s: "get_lib_node"
s -> api: "GET /search/tracks?q=…&client_id="
api -> s: "tracks (id, title, user, duration)"
s -> tui: "tracks as queueable leaves (/soundcloud/track/<id>)"
tui -> orch: "queue + play a track"
orch -> s: "get_urls_for_track(/soundcloud/track/<id>)"
s -> api: "GET /tracks/<id>  → pick hls+audio/mpeg transcoding"
s -> api: "GET <transcoding.url>?client_id=  → { url: m3u8 }"
s -> orch: "urls = [ m3u8 ]"
orch -> hls: "player.play(m3u8)"
hls -> cdn: "GET m3u8 (segments)"
hls -> cdn: "GET segment 1..N (mp3, in order)"
hls -> orch: "continuous mp3 → symphonia decodes"
```

## Risks and open questions

- **client_id scraping fragility.** The scrape regexes depend on
  soundcloud.com's HTML/JS shape and can break on a redesign. Mitigation: a
  config override (`client_id` in `soundcloud.toml`) always wins, and failures
  are typed (subtree degrades, app survives). The scrape is the one piece with
  no test-double coverage of the *live* format — flagged as a **live-test gate**.
- **Ephemeral media URLs.** The resolved `.m3u8` and its segments are signed and
  short-lived; playback must start promptly after resolution (like `/youtube`).
  A stale URL surfaces as a skipped track, never a crash. Never logged.
- **HLS without a known length.** The concatenated stream has no total byte
  length; reported non-seekable so symphonia won't end-seek (the rodio-0.22
  panic the opus work documented). **Gate:** verify an end-to-end mp3-HLS play
  does not panic and reaches EOS cleanly.
- **Master vs media playlist.** SoundCloud returns a media (segment) playlist
  for the chosen transcoding; `HlsStream` follows one level of master playlist
  defensively and picks the first variant.
- **Playlist stubs.** Playlist detail may return track stubs; hydration in
  ≤50-id batches is bounded by `playlist_tracks_limit`. A hydration miss drops
  that track (skipped), never an error.
- **OAuth token lifetime.** v1 accepts a static token; expiry drops the personal
  subtree with a typed `401` (public browse unaffected). A device-flow/refresh
  upgrade mirrors tidal and is additive.
- **Field / envelope drift.** DTOs decode defensively (`#[serde(default)]`);
  a renamed field is a local fix in `ScApi`. Live validation is a task-plan gate.