crabidy/architecture/queue-order.md

# Queue order: de-duplication and sorting

## Context

The queue is an ordered list of tracks the server owns; clients mutate it
through RPCs and learn the result from the update stream
(`docs/src/queue.md`). Every existing mutation either *adds* tracks
(`Replace`/`Append`/`Queue`/`Insert`), *drops* them (`Remove`/`ClearQueue`),
or moves the cursor (`SetCurrent`). Nothing **reorders** what is already
there, and nothing notices that the same track is in the queue twice.

Both gaps show up in ordinary use:

- Queue an artist, then a playlist that contains three of the same tracks,
  and the queue plays them twice. Today the only fix is spotting the rows
  and pressing `d` on each.
- Queue five albums by appending them, and they play in the order they were
  appended, interleaved by however the provider listed them. There is no way
  to say "group this by artist" or "shortest first" short of clearing the
  queue and re-queueing in a different order.

So: two new queue-order operations — **dedup** (drop duplicate entries) and
**sort** (reorder by a strategy) — for every client, without re-resolving
anything through providers.

## Assumptions

- **A1 — The server owns queue state; clients ask and then observe.** Same
  rule as `architecture/seek.md` A2 and `architecture/mpris.md` A2: a client
  sends the operation, the playback loop performs it, and the resulting
  `Queue` snapshot is the truth. No client predicts the new order.
- **A2 — Order is queue state, shuffle is a modifier.** `tracks` is the
  queue order (what every client renders, and what playback follows with
  shuffle off); `play_order` is the *play* order shuffle permutes. Dedup and
  sort rewrite the former. They are not new modifiers: nothing about them is
  remembered, they are one-shot rewrites, and after one runs the queue is
  just a queue in a different order.
- **A3 — Both work on metadata already in the queue.** `Track` carries
  path, artist, title, album, duration and `provider_item_id`; no provider
  round trip, no network, no new provider trait method.
- **A4 — The current track keeps playing.** Neither operation interrupts
  audio: dedup never removes the playing track, sort never restarts it. The
  playing track's *position* changes, which the existing `QueueTrack`
  broadcast already covers.
- **A5 — A queue can be long.** Ten thousand entries after queueing a large
  artist is normal (`architecture/progressive-queueing.md`), so both
  operations must be O(n log n) with keys computed once per track, not per
  comparison, and must not hold the queue lock across an await.

## Options considered

### Where the operations run

**Option A — client-side, on top of the existing RPCs.** A client already
holds the full `Queue` snapshot, so it could find duplicate positions itself
and send `Remove`. Sorting would mean sending `Replace` with the paths in the
new order — which re-resolves every path through its provider (slow, and a
provider that has since changed its listing returns something else), loses
the playing track (a replace restarts playback at the head), and would make
each client reimplement the comparators. And the `Remove` positions are
computed from a snapshot that a still-running resolve can invalidate before
the request lands.

**Option B — server-side RPCs *(chosen)*.** Both run on the playback loop,
the single writer of queue state: atomic against resolve chunks and against
other clients, no re-resolution, the current track is identified by *index*
and not by re-lookup, and the result reaches every client (and the persister)
through the one broadcast site that already exists.

**Option C — a general `ReorderQueue(permutation)` RPC.** The client decides
the order and the server applies it; sort strategies would then be a purely
client-side concern, and the same RPC would later serve drag-and-drop and
"move this track up". More general, but a permutation is an argument about
*positions*, and positions shift under a running resolve — a stale
permutation is not merely a no-op, it scrambles the queue. It also puts five
comparators in every client. Kept as **Deferred**: a move/reorder RPC wants
a different argument shape (identify rows by path, not index) and is its own
feature.

**Decision: Option B.** Two RPCs, `DedupQueue` and `SortQueue`, handled on
the playback loop like every other queue verb.

### What counts as a duplicate

1. **Same `path`.** Exactly the same library entry queued twice.
2. **Same provider item** — `provider_item_id` scoped to its provider, with
   `path` as the fallback when the id is empty. Catches the same track
   reached by two routes: through an album, through a playlist, through a
   search result.
3. **Same artist and title**, normalized. Would also catch the same song
   from two different providers.

**Decision: 2, falling back to 1** (D3). 3 is rejected: identical
artist/title is routinely a *different recording* — a live take, a remaster,
a radio edit, the studio version — and the server cannot tell which. Dropping
one would be a silent, unrecoverable edit of the user's queue, and the false
positives land exactly on the collections (greatest-hits, live albums) where
a user is most deliberate. A fuzzy dedup belongs behind a client-side preview
where the user confirms each pair; that is Deferred.

## Decisions

- **D1 — Two RPCs, both `QueueOwner`.** `DedupQueue` and `SortQueue` mutate
  the queue, so they sit with `Remove`/`ClearQueue`/`SetCurrent` in the
  rights matrix (`architecture/roles-auth.md`), not with the one appender
  verb. The pinned method-list test in `auth.rs` keeps a new RPC from
  reaching the wire unmapped.
- **D2 — `DedupQueue` answers with a count.** Every other queue verb
  answers with an empty message because the update stream carries the truth.
  Dedup is the exception: "how many did that remove?" cannot be recovered
  from the new snapshot (a client would have to diff against a snapshot it
  may never have had), and `0` is the answer a user most needs — it says
  *there were no duplicates*, as opposed to *nothing happened*. So
  `DedupQueueResponse.removed`, produced on the loop and returned through a
  bounded result channel, the way `SaveQueue` already reports.
  `SortQueueResponse` stays empty: the new order *is* the answer, and it
  arrives on the stream.
- **D3 — Duplicate identity is `(provider, provider_item_id)`, or the whole
  path.** The provider is the first path segment; ids are provider-internal
  (the content store keys them the same way, `by_provider_id(provider, id)`),
  so leaving them unscoped would let two providers' numeric ids collide and
  merge two unrelated tracks. When the id is empty — most providers, and
  every local file — the key is the full path, which is exact. The
  consequence worth stating: a captured copy under `/crabidy` and its
  streaming original are **not** duplicates, because they are different
  providers. Conservative on purpose: a missed duplicate is a keypress, a
  wrong merge is lost queue state.
- **D4 — The survivor is the current track, else the earliest.** Within a
  group of duplicates, the entry at the current position survives if it is
  in the group; otherwise the earliest one does, and every later copy goes.
  "Keep the first" alone would remove the playing track whenever the playing
  copy was a later one — a dedup that stops the music is a bug, not a
  policy (A4).
- **D5 — Dedup is expressed as a removal.** It computes the positions to
  drop and hands them to `QueueManager::remove_tracks`, which already
  maintains `play_order`, shifts `current_offset`, and ignores out-of-range
  positions. Since the current track is never in that list, `remove_tracks`
  reports no successor to start and playback is untouched — the same code
  path a client's `d` takes, so there is one removal implementation, not two.
- **D6 — Five sort strategies, on a proto enum.** `ARTIST`, `ALBUM`,
  `TITLE`, `DURATION` and `REVERSE`, plus a `descending` flag.
  `UNSPECIFIED` (the proto3 default, i.e. a client that forgot the field) is
  `InvalidArgument`, never a silent default. `REVERSE` reverses the order
  the queue is in and ignores `descending` — it is not a key, and reversing
  descendingly is the same thing.
- **D7 — The sort is stable, and the keys are compound.** `ARTIST` sorts by
  (artist, album) and `ALBUM` by (album) alone; within an equal key, queue
  order survives — which for a queued album *is* its track order, and is
  much closer to what a user means than sorting an album's tracks
  alphabetically by title. A stable sort is what makes "sort by album, then
  by artist" compose across two presses, too.
- **D8 — Unknown sorts last, in both directions.** An empty artist/album/
  title and an absent duration go to the end ascending *and* descending. A
  length-less web radio stream is not "the longest track", and a missing
  album is not alphabetically first. One rule for every key, so a user never
  has to remember which end the blanks pile up at.
- **D9 — Text comparison is case-insensitive and locale-naive.** Keys are
  lowercased once per track (decorate–sort–undecorate, A5), then compared as
  Unicode strings. No collation, no article stripping: "The Beatles" sorts
  under T. Locale-aware collation would need a collation library and a
  locale the server does not have (it has no user, only clients); doing it
  half-way — special-casing English articles — would be wrong for every
  other language in a music library.
- **D10 — Sorting reorders `tracks`; what plays next depends on shuffle.**
  With shuffle **off**, `play_order` is rebuilt as the identity over the new
  order and `current_offset` follows the current track: the queue order *is*
  the play order, so a sort changes what plays next — the point of sorting.
  With shuffle **on**, `play_order` is remapped through the sort permutation,
  so the shuffled play sequence and the position within it are preserved
  exactly: the user asked for a random order, and sorting the *display* must
  not silently reshuffle. Either way the current track stays current and
  keeps playing (A4).
- **D11 — Both are legal while a resolve is in flight, and neither waits for
  it.** They apply to what the queue holds at that moment; chunks still
  being resolved land afterwards at their insert index or at the end,
  unsorted. Refusing (`FailedPrecondition`) would make both operations
  flaky exactly on the big queues that need them, and waiting would block
  the loop. Clients already render the `resolving` indicator, so "more is
  still arriving" is visible; a second press settles the rest.
- **D12 — Both persist through the existing broadcast site.**
  `broadcast_queue` hands the snapshot to the persister and to the stream,
  so a dedup or a sort survives a server restart with no new persistence
  code (`architecture/queue-persistence.md`).
- **D13 — Pure key logic lives in its own module.** `queue_order.rs` holds
  the duplicate key, the sort keys, and the permutation — no state, no locks,
  unit-testable directly. `QueueManager::dedup`/`sort` keep the
  `play_order`/`current_offset` bookkeeping, because that invariant is
  theirs. The split is what keeps D3/D7/D8 testable as data instead of
  through a queue.
- **D14 — TUI: `u` dedups, `S` opens a sort menu.** `u` is "unique" and is
  free in the queue scope. Sorting needs a choice, so `S` opens a modal
  overlay listing the five strategies with their letters (`a` artist, `l`
  album, `t` title, `d` duration, `r` reverse; the capital of each sorts
  descending), `Esc` closes. A menu rather than a chord sequence: the
  strategies are discoverable in the overlay instead of only in the help
  modal, and the app already has three modal overlays (help, input, search)
  to follow. It is modal in the same strict sense — while it is open, the
  bindings table is unreachable.
- **D15 — The dedup count is reported in the queue pane's title, briefly.**
  `Queue — removed 7 duplicates`, for a few seconds, then back to normal.
  The title already multiplexes VISUAL, the `/` query and the register count,
  so no layout changes and no new region; and the message is queue-scoped,
  which is where the user is looking after pressing `u`. The count travels
  as a typed `MessageToUi` variant, not a preformatted string — the wording
  is the UI layer's business.
- **D16 — The web client keeps the same keys and adds the two buttons a mouse
  needs.** `u` dedups and `S` opens the sort menu as a dialog, because the
  keymap is deliberately the TUI's (`architecture/web-client.md`); the menu's
  rows are clickable as well as typeable. The queue toolbar gains a **sort**
  button that opens that same dialog and a **dedup** button — without them
  both operations would be invisible to a mouse, and routing the button
  through the same menu keeps the strategy list in exactly one place (a
  second widget listing five strategies is a second thing to keep in step).
  The dedup count goes to the existing toast rather than a pane title.
- **D17 — CLI: `cbd queue dedup` and `cbd queue sort <key> [--desc]`.** The
  strategy is a `clap` `ValueEnum`, so the shell completes it and a typo is
  a parse error rather than an `InvalidArgument` round trip. `dedup` prints
  the count it got back (D2).

## Structure

```d2
direction: right

clients: Clients {
  tui: cbd-tui\nu / S menu
  web: cbd-web\nu / S / select
  cli: cbd queue\ndedup / sort
}

rpc: gRPC (QueueOwner) {
  dedup: DedupQueue\n-> removed: u32
  sort: SortQueue\n(strategy, descending)
}

loop: playback loop\n(single writer) {
  cmd: PlaybackCommand\nDedupQueue / SortQueue
  qm: QueueManager\ntracks + play_order
  keys: queue_order\nduplicate key, sort keys
}

out: One broadcast site {
  stream: Queue update\n(every client)
  persist: persister\n(latest wins)
}

clients.tui -> rpc.dedup
clients.web -> rpc.dedup
clients.cli -> rpc.dedup
clients.cli -> rpc.sort
clients.tui -> rpc.sort
clients.web -> rpc.sort
rpc.dedup -> loop.cmd
rpc.sort -> loop.cmd
loop.cmd -> loop.qm: mutate
loop.qm -> loop.keys: keys / permutation
loop.qm -> out.stream
loop.qm -> out.persist
rpc.dedup -> clients.cli: removed count
```

The sort's effect on the two orders (D10) — the same permutation, applied
differently depending on shuffle:

```d2
shape: sequence_diagram

client: client
loop: playback loop
off: QueueManager\nshuffle off
on: QueueManager\nshuffle on
stream: update stream

client -> loop: SortQueue(ARTIST, asc)
loop -> off: sort(ARTIST, asc)
off -> off: tracks := sorted
off -> off: play_order := identity
off -> off: offset := the current track's new index
loop -> on: sort(ARTIST, asc)
on -> on: tracks := sorted
on -> on: play_order := remapped through the permutation
on -> on: offset unchanged
off -> stream: Queue (new order, new position)
on -> stream: Queue (new order, new position)
loop -> client: SortQueueResponse {}
```

## Boundaries

- **`crabidy-server::queue_order`** — pure functions: the duplicate key of a
  track, the sort key of a track, and the permutation for a strategy. Knows
  nothing about locks, play order or clients.
- **`crabidy-server::QueueManager`** — gains `dedup()` and `sort()`. Owns
  the `play_order`/`current_offset` invariants; `dedup` delegates the actual
  removal to `remove_tracks` (D5).
- **`crabidy-server::playback`** — two new `PlaybackCommand` arms, each a
  lock–mutate–broadcast on the loop. `DedupQueue` carries a result channel
  for the count.
- **`crabidy-server::rpc` / `auth`** — the two methods, their argument
  validation (`UNSPECIFIED` → `InvalidArgument`), and their row in the rights
  matrix.
- **Clients** (`cbd-tui`, `cbd-web`, `cbd-cli`) — bindings/commands, the
  sort-menu modal, and rendering the count. No client computes an order.
- **Unchanged**: providers, the content store, the resolve pipeline, and
  every existing queue RPC.

## Risks

- **A pending insert's index goes stale.** Dedup shifts positions and sort
  moves everything, so the remaining chunks of an in-flight `InsertAt` op
  land somewhere else than the user pointed at. This is pre-existing — a
  plain `Remove` while resolving does the same — and bounded by D11's
  "settle it with a second press", but it is real.
- **Dedup is per-provider (D3), so the obvious cross-provider duplicate — a
  captured track and its streaming source — stays.** It is the conservative
  end of a trade-off, and it will read as a bug to someone.
- **A sort with shuffle on changes nothing audible** (D10) and may look
  broken. The clients show shuffle state, and the queue visibly reorders.
- **Very large queues copy their tracks once per sort.** A `Vec<Track>`
  permutation on ten thousand tracks is a handful of milliseconds on the
  loop; well under the loop's other work (a resolve chunk), but it is work
  done while no other command is served.

## Deferred

- **Fuzzy dedup** (artist/title matching) behind a client-side confirmation
  view — the only safe home for it (see the options above).
- **`ReorderQueue`/move**: drag-and-drop in the web client and `K`/`J` row
  moves in the TUI, on an RPC that identifies rows by path rather than
  index (Option C).
- **Sort by release year.** `Album.release_date` is a provider string and
  not always ISO 8601, so a year needs the same parse-or-drop treatment the
  TUI notification does; worth doing once that parse lives somewhere shared.
- **A remembered sort** ("keep the queue sorted by artist as tracks
  arrive") — a modifier, which is a different feature from a one-shot
  rewrite (A2), and one that fights progressive queueing.
- **Sort within a marked range only**, the visual-mode analogue of a partial
  sort.