# Progressive queueing of large collections

## Context and problem statement

Queueing a nested node (an artist with many albums, a large playlist) today
freezes the UI's mental model: nothing changes for many seconds, then the full
queue appears at once. Three compounding causes, all in the resolve path:

1. `Playback::resolve_tracks` collects **every** track before the queue is
   touched, so the single `Queue` broadcast happens only at the very end and
   playback cannot start earlier.
2. The resolve runs **inline in the playback loop**, so every other playback
   command — pause, next, volume — is blocked for the duration.
3. `tidaldy` paginates collections to exhaustion (50 tracks per sequential
   request) inside one `get_lib_node` call, so even a single large playlist
   produces no intermediate result.

The feature: resolve progressively. Tracks are applied to the queue in chunks
as the provider produces them, each chunk is broadcast, playback starts with
the first chunk, and clients see a loading indicator (animated dots as a
pseudo last queue item) while resolution is still running.

Run autonomously per standing user instruction; every decision below records
the options considered and the rationale.

## Assumptions

- Chunks must arrive **in playback order** — the user explicitly wants "the
  first elements first, and later chunks follow". Order is known up front
  (collection order), so no reordering step is needed.
- The four resolve-based queue operations (`Replace`, `Queue`, `Append`,
  `Insert`) all benefit equally and should share one mechanism.
- Wire changes must stay additive (old clients keep working; they simply see
  the queue fill progressively without an indicator).
- Multi-client remains supported: the indicator must be server-derived state,
  not client-local guessing.

## Decisions

### D1 — End-to-end progressive resolution, not a client-only spinner

Options considered:

1. **Client-only indicator**: the TUI shows dots between sending a queue op
   and receiving the next `Queue` update. No wire or server change.
2. **Server-side chunked resolution** with a wire-visible "still resolving"
   flag; the indicator falls out of the flag.

**Decision: (2).** Option 1 papers over the latency without fixing it —
playback would still start only after the full resolve, other clients would
see nothing, and the "did it even register?" dead time remains. Option 2
fixes the actual complaint (start playing early, fill visibly) and gives
every client the indicator for free. The server broadcasts an immediate
`Queue` update (unchanged tracks, `resolving = true`) when the op is
accepted, so feedback appears within one round trip.

### D2 — `Queue.resolving` field, not a new stream-update variant

Options considered:

1. New `GetUpdateStream` oneof variant `resolving(bool)`.
2. New field `bool resolving = 4` on the `Queue` message itself.

**Decision: (2).** The flag is queue state and must be atomic with the track
snapshot it describes; a separate variant can arrive out of order relative to
`Queue` updates (the broadcast channel is lossy for slow clients). Additive
field, wire-compatible both ways: old clients ignore it, old servers never
set it.

### D3 — Chunked resolution lives in `ProviderClient`, with a default

Options considered:

1. Keep the BFS in `ProviderOrchestrator` (chunk = one node's tracks); no
   trait change. A 1000-track playlist is still one 20-request blob.
2. Add a chunked resolve method to the `ProviderClient` trait with a default
   implementation (the generic walk, one chunk per node); `tidaldy`
   overrides it to stream **page-sized chunks (50)** for the paginated
   collections (playlists, albums).

**Decision: (2).**

```rust
/// Streams the playable tracks under `path` into `chunk_tx` in playback
/// order. Zero or more chunks, then the sender is dropped: a dropped
/// SENDER means resolution finished. A dropped RECEIVER cancels
/// resolution (the provider stops fetching and returns Ok).
async fn resolve_tracks_into(
    &self,
    path: &str,
    chunk_tx: flume::Sender<Vec<Track>>,
) -> Result<(), ProviderError>;
```

The channel semantics are the contract and are documented on the trait —
this is a local bounded channel used as a stream, not a queue pretending to
be durable. Unreadable nodes are skipped with a warning (today's behavior);
only a completely unresolvable root returns an error. The default
implementation walks the tree **depth-first pre-order** over queueable
descendants, emitting one chunk per node. This replaces the old orchestrator
BFS, and fixes a latent ordering bug while doing so: the old worklist popped
LIFO, so an artist's albums were flattened in *reverse* order.

`tidaldy` overrides the method: playlist and album paths stream one chunk
per fetched page instead of paginating to exhaustion first (a new
`make_paginated_request` variant hands each page to a sink); everything else
follows the generic walk. Search-term track lists are a single page already.
The playlist arm also stops fetching the playlist metadata (title) — the
resolve needs only tracks.

`flume` is already a workspace dependency; `crabidy-core` adopts it for the
trait signature.

### D4 — Neither event loop blocks: spawned resolves, single-writer queue

Options considered:

1. Consume chunks inline in the playback loop's `handle_command` (loop still
   blocked for the whole resolve; pause/next dead — today's hidden defect).
2. Spawn the resolve; queue mutations travel back to the playback loop as
   internal commands, so the loop stays the **single writer** of queue state.

**Decision: (2), on both loops.**

- **Provider side**: `ProviderOrchestrator::run` wraps the orchestrator in an
  `Arc`; the `ResolveTracks` arm spawns the resolve onto its own task instead
  of awaiting it inline. Without this, the first chunk would deadlock the
  system: playback applies chunk 1 → `play()` → sends `GetTrackUrls` and
  awaits the reply — but the provider loop would still be busy resolving.
  Other provider commands keep flowing while (possibly several) resolves run.
- **Playback side**: each queue op registers a *pending op* (id from an
  `AtomicU64`, kind, insertion cursor) and spawns a forwarder task that
  drives `ProviderCommand::ResolveTracks` per path (concurrently, under the
  exponential read-ahead window of D8, but forwarding chunks in strict
  multi-path order) and forwards each chunk to the playback channel as
  `PlaybackCommand::ApplyResolvedChunk { op_id, tracks }`, followed by
  `ResolveFinished { op_id }`. Queue state is only ever mutated inside the
  loop, exactly as before; user commands interleave between chunks.

Backpressure is real at every hop: provider → forwarder over a small bounded
chunk channel, forwarder → playback over the existing `bounded(64)` command
channel. A slow consumer slows the HTTP fetching down instead of buffering
unboundedly.

### D5 — Chunk application semantics per op kind

Each pending op keeps an insertion cursor:

- **Replace**: first chunk `replace_with_tracks` (broadcast resets the
  queue, current position 0, playback starts); later chunks append.
- **Append**: every chunk `append_tracks`.
- **Queue** (after current): cursor starts at the current position; each
  chunk `insert_tracks(cursor)`, then `cursor += chunk.len()`.
- **Insert**: same, starting at the requested position.

Playback start reuses the existing `Option<Track>` returns from the
`QueueManager` mutations — only a chunk that makes a track current (replace,
or any insert into an empty queue) yields one, so exactly the first relevant
chunk starts the player and later chunks never restart a playing track.

That first start can *fail to find anything playable yet*: if the head
tracks are `is_skipped` or the collection's first track is very short, the
player can run dry before the next chunk resolves — and then, since later
chunks return `None`, playback would stay stopped with playable tracks
arriving right behind it. So the op carries a `wants_start` flag: set when a
chunk makes a track current, cleared only once a start is *confirmed*
(`play` returned that it handed a track to the player). While it is set, each
arriving chunk retries the start from the current position — `next_playable`
advances past skipped/unplayable heads to the first track that has now
resolved. The exponential read-ahead (D8) makes that next track arrive
sooner; the retry makes sure it actually plays when it does. An op whose
whole resolve finishes with nothing playable is dropped by `finish_resolve`
and the player simply stays stopped.

Interleaved edits from other clients during a resolve can shift the cursor's
target (e.g. removing tracks before it). This is accepted as benign:
`insert_tracks` already clamps, the queue self-heals on the next broadcast,
and simultaneous multi-client edits during a resolve are rare. Under
shuffle, arriving chunks are shuffled behind the current track like any
other insert — chunk order is irrelevant when shuffle is on. Each op counts
its applied tracks; an op that finishes with zero keeps today's
"resolved to no playable tracks" warning.

### D6 — `Replace` and `Clear` cancel in-flight resolves

Without cancellation, "replace the queue" or "clear the queue" during a
large resolve would be followed by the old op's remaining chunks trickling
back in — a corrupted queue, and the exact ghost behavior this feature is
meant to kill. Options: let stale chunks land (wrong), or cancel.

**Decision: cancel.** Each pending op carries an `Arc<AtomicBool>` shared
with its forwarder. `Replace` and `Clear` mark every pending op cancelled
and drop it from the map. The forwarder checks the flag per chunk and, when
set, drops the chunk receiver — the provider's next `send` fails and the
resolve task stops fetching (the documented receiver-drop semantics from
D3). Chunks already in flight for an unknown op id are ignored by the loop.
`Queue`/`Append`/`Insert` do **not** cancel: concurrent additive ops are
legal; their chunks interleave between ops while each op's internal order is
preserved.

### D7 — TUI indicator: an animated pseudo-item, outside the list model

While the latest `Queue` update carries `resolving = true`, the queue pane
renders one extra line after the last track: one to three dots cycling
(~400 ms per step, derived from elapsed time — the render loop already
redraws at least every 100 ms), in `COLOR_SECONDARY`. The pseudo-item is
appended at render time only and never enters `self.list`, so selection,
removal, and `get_size` cannot reach it — no new input states, nothing to
misclick.

### D8 — Exponential read-ahead across paths

The forwarder of D4 first resolved paths one at a time. That starts the first
track quickly (good) but fills the rest only as fast as one provider resolve
at a time. When enumeration is slow (per-path Tidal/YouTube/fyyd round trips)
and the leading tracks are very short or `is_skipped`, playback drains the
resolved queue faster than a sequential resolver refills it — and stalls into
silence, the exact thing progressive queueing exists to avoid.

Options considered:

1. **Sequential per path** (original): simplest, but a short/skipped head can
   outrun a slow resolver.
2. **Resolve every path at once**: fills fastest, but a large marked
   selection fires an unbounded burst of concurrent provider calls (rate
   limits, memory) the instant playback starts — wasteful when the user skips
   away after two tracks.
3. **Exponential read-ahead window**: a concurrency window that starts at 1
   and doubles (1, 2, 4, 8, 16, then steady 16) after each path completes.

**Decision: (3).** The first path resolves alone, so time-to-first-track is
unchanged from (1); the window then grows geometrically, so the resolved
queue runs exponentially ahead of linear playback and a short/skipped head
cannot catch it after the first couple of tracks. The cap (16) bounds
concurrent provider load. Chunks are still forwarded in strict path order —
the forwarder fully drains the oldest in-flight resolve before the next, so
concurrency never reorders the queue (D5's per-op cursor and "first chunk
starts the player" are untouched). Cancellation (D6) drops the in-flight
receivers, stopping every concurrent resolve at once.

This is a read-ahead over *paths*. A single collection path (one album or
playlist) is still enumerated by its provider's `resolve_tracks_into` — the
per-page streaming of D3 is that path's read-ahead — so the window is the win
for multi-item selections; single-collection latency stays a provider concern.

## Flows

```d2
shape: sequence_diagram
user: { shape: person }
tui: cbd-tui
rpc: gRPC handler
playback: playback loop
fwd: forwarder task
provider: provider loop
resolve: resolve task
tidal: Tidal API

user -> tui: queue large artist
tui -> rpc: Append(paths)
rpc -> playback: "PlaybackCommand::Append (fire-and-forget)"
playback -> playback: register pending op
playback -> tui: "Queue update (resolving=true)"
playback -> fwd: spawn
fwd -> provider: "ResolveTracks(path, chunk_tx)"
provider -> resolve: spawn
resolve -> tidal: fetch page 1
resolve -> fwd: chunk 1
fwd -> playback: "ApplyResolvedChunk(op, chunk 1)"
playback -> tui: "Queue update (resolving=true)"
playback -> playback: "play() first track"
resolve -> tidal: fetch page 2
resolve -> fwd: chunk 2
fwd -> playback: "ApplyResolvedChunk(op, chunk 2)"
playback -> tui: "Queue update (resolving=true)"
fwd -> playback: "ResolveFinished(op)"
playback -> tui: "Queue update (resolving=false)"
```

The dots pseudo-item is visible in the TUI exactly while updates carry
`resolving = true`; user commands (pause, next, remove) flow through the
playback loop between chunk applications instead of waiting for the end.

```d2
direction: right
core: "ProviderClient::resolve_tracks_into" {
  default: "default: pre-order walk,\none chunk per node"
}
tidaldy: "tidaldy override" {
  pages: "playlist/album:\none chunk per 50-track page"
}
playback: "playback loop" {
  ops: "pending ops:\ncursor + cancel flag"
}
core -> tidaldy: overridden by
tidaldy.pages -> playback.ops: "bounded chunks, in order"
playback.ops -> playback.ops: "apply + broadcast per chunk"
```

## Boundaries and risks

- **Proto**: one additive field (`Queue.resolving = 4`). No RPC shape
  changes; the queue ops stay fire-and-forget.
- **Trait**: one new `ProviderClient` method with a default implementation —
  existing providers (there is one) compile unchanged if they skip the
  override; the override is where the provider-level win lives.
- **Ordering fix is a behavior change**: multi-album artists now queue in
  listing order instead of reversed. Strictly a fix, noted here because
  someone may have gotten used to the bug.
- **Concurrent additive ops interleave between ops.** Each op's internal
  order is kept; the interleaving matches command arrival order at the loop.
  Accepted — same semantics a human doing two appends "at once" expects.
- **Old TUI + new server**: queue fills progressively, no indicator — pure
  improvement, no breakage. New TUI + old server: `resolving` is always
  false, indicator never shows, behavior as today.
- **Not in scope**: pagination of `get_lib_node` for *browsing* (the library
  pane still fetches collections to exhaustion before rendering), queue
  persistence, a progress percentage (total counts are known per collection
  but not aggregated across a nested walk).