diff --git a/README.md b/README.md index e6bb694..e20035c 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,10 @@ each mounted as a subtree of one library: ├── tidal Tidal streaming (see tidaldy/README.md) ├── youtube YouTube search & playlists (see ytdy/README.md) ├── fs a local music folder (see fsdy/README.md) -└── crabidy your saves: queues, bookmarks (`w`), and captures (`W`), - managed by the server (see architecture/crabidy-store.md) +├── crabidy your saves: queues, bookmarks (`w`), and captures (`W`), +│ managed by the server (see architecture/crabidy-store.md) +└── orphans store audio no save references any more — rename, delete, + or queue it (see architecture/orphans.md) ``` ## Binaries diff --git a/architecture/orphans.md b/architecture/orphans.md new file mode 100644 index 0000000..dcb95e1 --- /dev/null +++ b/architecture/orphans.md @@ -0,0 +1,242 @@ +# The `orphans` provider — a store garbage-collection view + +## Context and problem statement + +The content-addressed store (`architecture/crabidy-store.md`) never shrinks on +its own. Capturing writes audio + a `.cbd-store.toml` sidecar into +`~/.local/share/crabidy/`; deleting a save (or the `current` queue rolling over, +or a `scan --capture` toml being removed) only ever removes the *toml that +pointed at* a store entry — never the entry itself (store `D7`). That is +deliberate: a store entry may be shared by many tomls, so no single deletion can +know it is safe to reclaim. The consequence, called out as future work in store +`D10`, is that store entries accumulate that **no toml references any more**. +There is today no way to see them or reclaim their disk. + +This design adds a read-mostly management provider, **`/orphans`**, that surfaces +exactly those unreferenced store entries and lets the user rename or delete them, +or queue them for a listen before deciding. It is the store's garbage-collection +UI, expressed as an ordinary library subtree so it needs no new client concepts. + +This **realizes** store `D10` (orphan reclamation). It changes nothing about how +captures are written or de-duplicated; it only *reads* the store's residue and +offers targeted rename/delete. + +## Assumptions + +- **Confirmed by the request.** `/orphans` lists every store item, walks all + local file providers, crosses off referenced items, and presents the rest; + entries are renamable (audio file *and* sidecar), deletable (files on disk), + and queueable. +- **"Referenced" means reachable through a mounted local file provider.** The + reference scan walks the disk roots of the running file providers — the + `/crabidy` toml tree (`~/.local/state/crabidy/`, which holds `current`, every + save, every capture) and the `/fs` root (which can hold `Playable::Store` + tomls written by `scan --capture`/`--move`). A `.cbd-track.toml` that lives + *outside* every mounted provider root (e.g. `scan --capture` run on a folder + that is not under `/fs`) is invisible to the walk, so its target counts as an + orphan. This is the only sound definition available without a global + reference index, and it matches the request's wording ("walks all local file + providers"). It is a documented boundary, not a bug (see Risks). +- **Single writer.** As with the rest of the store, one process owns both roots + and serializes mutations under the store's index mutex; there is no concurrent + external writer. +- **Orphan-ness is recomputed on every listing.** There is no persisted orphan + list — consistent with `fsdy`'s "read the tree fresh every visit" philosophy. + A capture that adds a reference makes an entry stop being an orphan on the + next listing. + +## What is a "store item" and when is it an orphan + +- A **store item** is a pair in `store_root`: an audio file `` and its + sidecar `.cbd-store.toml`. The sidecar is the source of truth (store + `D2`); the set of items is the set of sidecars that have a readable audio file + beside them. (A sidecar without audio, or audio without a sidecar, is + malformed residue — reported so it can be reclaimed too; see D4.) +- A store item `` is **referenced** iff some `.cbd-track.toml` under a + mounted file-provider root validates to `Playable::Store()`. +- **Orphans = all store items − referenced store items.** + +Because `/orphans` only ever exposes unreferenced entries, renaming or deleting +one **cannot break any toml reference** — that is what makes the destructive +operations safe by construction (subject to the narrow race in Risks). + +## Options considered + +### Presentation: tracks vs. child nodes + +Each orphan must be renamable, deletable, and queueable. The wire has two +carriers with capability flags: + +- **`Track`** — has `is_captured`/`is_skipped` but **no** `is_editable`. Tracks + can be deleted (`tracks_deletable`) and queued, but the library has **no + rename-a-track gesture** anywhere; adding one means new proto surface plus TUI + and web changes. +- **`LibraryNodeChild`** — already carries `is_editable`, `is_deletable`, + `is_queable`, `is_captured`. The node-editing feature already binds `e` → + `rename_lib_node(child_path, new_title)` and `d` → `delete_lib_node(child_path)` + for children that advertise the flags (that is how `/tidal/search` terms and + `/crabidy` saves are renamed/deleted today), and queueing a queueable child + resolves its tracks. + +**Decision: present each orphan as an editable + deletable + queueable child +node** of `/orphans`, titled by its store file `` (the thing a rename +edits). Entering the node lists its single track (the store audio, with metadata +from the sidecar); queueing the node — or the whole `/orphans` root — resolves +that track. This reuses `rename_lib_node`/`delete_lib_node`/`resolve_tracks_into` +and every client gesture **with zero proto, TUI, or web changes**. The only +minor wart — each orphan is a one-track "folder" — is acceptable and is exactly +how a single-track save already presents. The track-carrier option was rejected +purely on cost: it buys nothing the node model lacks and forces a wire change +just to gain a rename gesture. + +### Home of the logic: new crate vs. server module + +The orphan computation needs the store root and index (to enumerate items and +mutate them) *and* the file-provider disk roots (to find references). A standalone +`orphandy` crate would have to duplicate `CrabidyStore` internals it does not own. + +**Decision: keep it in `crabidy-server`.** Orphan enumeration, rename, and delete +become methods on `CrabidyStore` (it already owns `store_root` and the index). A +thin new `OrphansProvider` (`crabidy-server/src/orphans.rs`) implements +`ProviderClient`, holding `Arc` plus the list of reference roots to +walk, and delegates to those store methods. `ProviderOrchestrator` mounts and +routes `/orphans` exactly like the other providers. + +## Boundaries and interfaces + +```d2 +direction: right + +tui: TUI / web / cbd-cli { shape: person } + +orchestrator: ProviderOrchestrator { + routes: "routes /orphans/*" +} + +orphans: OrphansProvider { + refroots: "ref_roots: Vec" +} + +store: CrabidyStore { + index: "StoreIndex (by_hash / by_provider_id)" + ops: "list_orphans / rename_orphan / delete_orphan" +} + +data: content store\n~/.local/share/crabidy { + shape: cylinder + items: " + .cbd-store.toml" +} + +crabidytree: /crabidy tree\n~/.local/state/crabidy { shape: cylinder } +fstree: /fs root { shape: cylinder } + +tui -> orchestrator: "get / rename / delete / queue /orphans/*" +orchestrator -> orphans: delegate +orphans -> store: "enumerate + mutate (by name)" +store -> data: "read sidecars, rename/delete files" +orphans -> crabidytree: "walk for Playable::Store refs" +orphans -> fstree: "walk for Playable::Store refs" +``` + +### The orphan diff (what a listing computes) + +```d2 +direction: down + +allitems: "all store items\n(scan *.cbd-store.toml in store_root)" +refs: "referenced set\n(walk ref_roots for\nPlayable::Store(name))" +diff: "orphans = all − referenced" { shape: diamond } +node: "/orphans node:\none editable/deletable/queueable\nchild per orphan" + +allitems -> diff +refs -> diff +diff -> node +``` + +### Provider surface (`OrphansProvider: ProviderClient`) + +Mounted at `/orphans` only when `crabidy_store` is present (it is the store's +view). Paths: the root `/orphans`, and one child per orphan at +`/orphans/`. There are no deeper levels. + +- `get_lib_root` / `get_lib_node("/orphans")` — a queueable, non-creatable node + whose children are the current orphans (recomputed by the diff above). Each + child: `title = `, `is_queable = true`, `is_editable = true`, + `is_deletable = true`, `is_downloadable = false`, `is_captured = true`. +- `get_lib_node("/orphans/")` — a queueable, childless node carrying the + single `Track` for that store entry (metadata from the sidecar's first + provider entry; `is_captured = true`). Unknown/renamed-away segment → + `MalformedPath`. +- `is_track_path` — always `false`: orphans are addressed as nodes, and the one + track is reached by resolving the node (so the default `resolve_tracks_into` + walk queues it). `get_metadata_for_track` therefore is not the entry point; + `get_urls_for_track("/orphans/")` returns `store_root/` (a local + file path, exactly like a resolved `Playable::Store`) so the resolved track + still plays. +- `rename_lib_node("/orphans/", new)` — validates `new` as a bare store + file name (`validate_folder_name(new, &[])`: non-empty, no separators/NUL, no + leading dot), refuses a name already taken by another store entry + (`InvalidInput`), then renames **both** ``→`` audio and + `.cbd-store.toml`→`.cbd-store.toml`, and updates the in-memory index + (drop the old name's mappings, re-insert under the new name; hash and provider + ids are unchanged). Returns the renamed node at `/orphans/`. +- `delete_lib_node("/orphans/")` — removes the audio file and the sidecar + from `store_root` and drops the entry from the index; returns the refreshed + `/orphans` root. Idempotent (an already-gone entry succeeds). +- `create_lib_node` — `NotSupported` (the root is not creatable). + +### Store methods added to `CrabidyStore` + +- `list_orphans(&self, ref_roots: &[PathBuf]) -> Result, StoreError>` + — scan `store_root` for `*.cbd-store.toml`; build the referenced set by walking + each `ref_root` recursively for `*.cbd-track.toml` and collecting + `Playable::Store(name)`; return the difference as `OrphanEntry { name, title, + artist, duration, album }` (metadata from the sidecar's first provider entry). +- `orphan_track(&self, name) -> Result` / + `orphan_url(&self, name) -> Result` — build the wire track + / resolve the store audio path for a single entry. +- `rename_orphan(&self, old, new)` / `delete_orphan(&self, name)` — the mutations + above, under the index mutex, with the index kept in sync. +- `StoreIndex::remove(&mut self, name, sidecar)` — the inverse of `insert`, so + rename/delete can update the derived index without a full rescan. + +### Reference roots wiring + +`OrphansProvider` is constructed in `ProviderOrchestrator::init` with +`ref_roots` = the disk roots of the mounted file providers: the `/crabidy` tree +(`store.tree_dir()`) and, when enabled, the `/fs` root. A new +`fsdy::Client::disk_root(&self) -> &Path` accessor exposes the `/fs` root (the +`/crabidy` tree root is already available via `CrabidyStore::tree_dir`). If more +`fsdy` instances are ever mounted, they are added to this list — the definition +of "local file provider" is "an `fsdy` instance whose root can hold store +references." + +## Risks and open questions + +- **Capture-then-delete race (TOCTOU).** Between a `/orphans` listing and a + delete, a concurrent `W` capture could hash-hit the very entry the user is + about to delete and write a fresh `Playable::Store` reference to it; deleting + then leaves that new toml dangling. The window is small (store mutations + serialize under the index mutex and orphan-ness is recomputed every listing), + and the failure is benign: a dangling store reference already resolves to + `MalformedPath` at play time and is skipped, not a crash. Accepted; noted here + rather than engineered away. +- **References outside mounted roots are not counted.** As stated in Assumptions, + a `scan --capture` toml under a folder that is not mounted under `/fs` will not + be seen, so its target shows as an orphan. Deleting it would orphan that + toml's audio. The mitigation is scope discipline (scan under `/fs`); the + alternative — a persisted global reference index — is out of scope and would + fight the "read fresh" design. +- **Cost.** A `/orphans` listing scans the whole store plus both trees on every + visit (no cache), i.e. O(store entries + tomls under the roots). This matches + `fsdy`'s existing per-visit read cost and is fine for personal-library sizes; + if it ever bites, memoizing behind the index's mutation counter is the escape + hatch. Not premature-optimized here. +- **Malformed residue.** A sidecar with no audio (or vice versa) is itself + reclaimable junk. `list_orphans` reports such half-entries as orphans (titled + by whatever is present) so a delete cleans them up; it never treats a + half-entry as "referenced." +- **Open question:** should the `/orphans` root also expose a single "delete all" + affordance? Deferred — per-item delete covers the request; bulk reclaim can be + a later addition (a client could multi-select and delete, once marks exist + there). diff --git a/crabidy-server/src/crabidy_store.rs b/crabidy-server/src/crabidy_store.rs index 8dbce38..a227261 100644 --- a/crabidy-server/src/crabidy_store.rs +++ b/crabidy-server/src/crabidy_store.rs @@ -75,6 +75,16 @@ pub enum StoreError { TomlWrite(#[from] toml::ser::Error), #[error("http client: {0}")] Http(#[from] reqwest::Error), + /// No store entry with the given name (rename/delete of a vanished + /// orphan). + #[error("no such store entry: {0}")] + NotFound(String), + /// A rename target name is already used by another store entry. + #[error("store name already in use: {0}")] + NameTaken(String), + /// A rename target name is not a legal bare store file name. + #[error("invalid store name: {0}")] + InvalidName(String), } /// One `(provider, id)` identity that resolves to a store entry, plus the @@ -200,6 +210,39 @@ impl StoreIndex { } } } + + /// The inverse of [`Self::insert`]: drops every mapping that still points + /// at `name` for this sidecar's hash and provider identities. A no-op for + /// mappings already reassigned to another entry. Used by orphan + /// rename/delete to keep the derived index in sync without a full rescan. + pub fn remove(&mut self, name: &StoreName, sidecar: &StoreSidecar) { + if self.by_hash.get(&sidecar.hash).is_some_and(|n| n == name) { + self.by_hash.remove(&sidecar.hash); + } + for provider in &sidecar.providers { + if provider.id.is_empty() { + continue; + } + let key = (provider.provider.clone(), provider.id.clone()); + if self.by_provider_id.get(&key).is_some_and(|n| n == name) { + self.by_provider_id.remove(&key); + } + } + } +} + +/// One unreferenced store entry surfaced by the `/orphans` provider (see +/// `architecture/orphans.md`). `name` is the bare store file name (the audio +/// file, and the stem of its `*.cbd-store.toml` sidecar); the rest is display +/// metadata from the sidecar's first provider entry (or the name itself for a +/// sidecar-less audio file). +#[derive(Debug, Clone)] +pub struct OrphanEntry { + pub name: StoreName, + pub title: String, + pub artist: String, + pub duration: Option, + pub album: Option, } /// Owns both store roots and serializes all mutation. @@ -936,6 +979,233 @@ impl CrabidyStore { } node.is_captured = all_captured && node.children.is_empty(); } + + /// Lists store entries that no `.cbd-track.toml` under any `ref_roots` + /// references — the `/orphans` view (architecture/orphans.md). Scans the + /// store for entries (a `*.cbd-store.toml` sidecar and/or a bare audio + /// file), builds the referenced set by walking `ref_roots` for + /// `Playable::Store` links, and returns the difference. Malformed residue + /// (a sidecar without audio, or audio without a sidecar) is reported so it + /// can be reclaimed. Reads fresh every call — there is no cached list. + pub async fn list_orphans( + &self, + ref_roots: &[PathBuf], + ) -> Result, StoreError> { + // Every store item, keyed by bare name → its parsed sidecar (if any). + let mut items: std::collections::HashMap> = + std::collections::HashMap::new(); + let mut rd = match tokio::fs::read_dir(&self.store_root).await { + Ok(rd) => rd, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(err) => return Err(err.into()), + }; + while let Some(entry) = rd.next_entry().await? { + let file_name = entry.file_name(); + let Some(file_name) = file_name.to_str() else { + continue; + }; + // Hidden entries and in-flight download temp files are not items. + if file_name.starts_with('.') { + continue; + } + if let Some(stem) = file_name.strip_suffix(SIDECAR_SUFFIX) { + let sidecar = match tokio::fs::read_to_string(entry.path()).await { + Ok(text) => toml::from_str::(&text).ok(), + Err(err) => { + warn!(sidecar = file_name, "cannot read store sidecar: {err}"); + None + } + }; + items.entry(stem.to_string()).or_insert(sidecar); + } else { + // A bare audio file; keep any sidecar already recorded for it. + items.entry(file_name.to_string()).or_insert(None); + } + } + + let referenced = Self::collect_referenced(ref_roots).await; + let mut orphans: Vec = items + .into_iter() + .filter(|(name, _)| !referenced.contains(name)) + .map(|(name, sidecar)| orphan_entry(name, sidecar.as_ref())) + .collect(); + orphans.sort_by_key(|orphan| orphan.name.to_lowercase()); + Ok(orphans) + } + + /// Walks each root (recursively, breadth-first) for `*.cbd-track.toml` + /// files and collects the store names they link to via + /// [`fsdy::Playable::Store`]. Unreadable directories and unparseable track + /// files are skipped with a warning — one bad file never aborts the scan. + async fn collect_referenced(roots: &[PathBuf]) -> std::collections::HashSet { + let mut referenced = std::collections::HashSet::new(); + let mut stack: Vec = roots.to_vec(); + while let Some(dir) = stack.pop() { + let mut rd = match tokio::fs::read_dir(&dir).await { + Ok(rd) => rd, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue, + Err(err) => { + warn!(dir = %dir.display(), "cannot scan for store references: {err}"); + continue; + } + }; + while let Ok(Some(entry)) = rd.next_entry().await { + let Ok(file_type) = entry.file_type().await else { + continue; + }; + // Symlinks are skipped by design (no cycles, no escaping). + if file_type.is_symlink() { + continue; + } + let path = entry.path(); + if file_type.is_dir() { + stack.push(path); + continue; + } + let name = entry.file_name(); + let Some(name) = name.to_str() else { + continue; + }; + if !name.ends_with(fsdy::TRACK_FILE_SUFFIX) { + continue; + } + let Ok(text) = tokio::fs::read_to_string(&path).await else { + continue; + }; + match fsdy::TrackFile::parse(&text).map(|f| f.playable()) { + Ok(Ok(fsdy::Playable::Store(store_name))) => { + referenced.insert(store_name); + } + Ok(_) => {} + Err(err) => { + warn!(file = %path.display(), "skipping invalid track file: {err}") + } + } + } + } + referenced + } + + /// The wire [`Track`] for a single store entry `name` at library path + /// `lib_path` — metadata from the sidecar's first provider entry (or the + /// bare name when there is no sidecar). Always `is_captured`, never + /// skipped. [`StoreError::NotFound`] when neither the sidecar nor an audio + /// file exists. + pub async fn orphan_track(&self, name: &str, lib_path: &str) -> Result { + let sidecar = self.read_sidecar(name).await.ok(); + if sidecar.is_none() && !self.audio_exists(name).await { + return Err(StoreError::NotFound(name.to_string())); + } + let first = sidecar.as_ref().and_then(|s| s.providers.first()); + Ok(Track { + path: lib_path.to_string(), + artist: first.map(|p| p.artist.clone()).unwrap_or_default(), + title: first + .map(|p| p.title.clone()) + .unwrap_or_else(|| name.to_string()), + duration: first.and_then(|p| p.duration), + album: first.and_then(|p| p.album.as_ref()).map(|a| Album { + title: a.title.clone(), + release_date: a.release_date.clone(), + }), + is_skipped: false, + provider_item_id: first.map(|p| p.id.clone()).unwrap_or_default(), + is_captured: true, + }) + } + + /// The local audio path a single store entry resolves to (the file the + /// player opens), or [`StoreError::NotFound`]/[`StoreError::InvalidName`]. + pub async fn orphan_url(&self, name: &str) -> Result { + if name.is_empty() || name.contains(['/', '\\', '\0']) { + return Err(StoreError::InvalidName(name.to_string())); + } + if !self.audio_exists(name).await { + return Err(StoreError::NotFound(name.to_string())); + } + self.store_root + .join(name) + .to_str() + .map(str::to_string) + .ok_or_else(|| StoreError::InvalidName(name.to_string())) + } + + /// Renames a store entry: the audio ``→`` and its + /// `.cbd-store.toml` sidecar → `.cbd-store.toml`, keeping the + /// derived index in sync. `new` must be a legal bare store name and must + /// not already be taken. Serialized under the index mutex; safe because + /// `/orphans` only exposes unreferenced entries (architecture/orphans.md). + pub async fn rename_orphan(&self, old: &str, new: &str) -> Result<(), StoreError> { + let new = fsdy::validate_folder_name(new, &[]) + .map_err(|_| StoreError::InvalidName(new.to_string()))?; + let mut index = self.index.lock().await; + if !self.audio_exists(old).await && !self.sidecar_exists(old).await { + return Err(StoreError::NotFound(old.to_string())); + } + if new != old && self.name_taken(new).await { + return Err(StoreError::NameTaken(new.to_string())); + } + if new == old { + return Ok(()); + } + self.rename_path(&self.store_root.join(old), &self.store_root.join(new)) + .await?; + self.rename_path(&self.sidecar_path(old), &self.sidecar_path(new)) + .await?; + // Re-point the index: read the moved sidecar (if any) and swap names. + if let Ok(sidecar) = self.read_sidecar(new).await { + index.remove(&old.to_string(), &sidecar); + index.insert(&new.to_string(), &sidecar); + } + Ok(()) + } + + /// Deletes a store entry's audio file and sidecar and drops it from the + /// index. Idempotent: an already-gone entry succeeds. Serialized under the + /// index mutex. + pub async fn delete_orphan(&self, name: &str) -> Result<(), StoreError> { + if name.is_empty() || name.contains(['/', '\\', '\0']) { + return Err(StoreError::InvalidName(name.to_string())); + } + let mut index = self.index.lock().await; + let sidecar = self.read_sidecar(name).await.ok(); + remove_file_if_present(&self.store_root.join(name)).await?; + remove_file_if_present(&self.sidecar_path(name)).await?; + if let Some(sidecar) = sidecar { + index.remove(&name.to_string(), &sidecar); + } + Ok(()) + } + + /// Whether a bare-named audio file exists in the store root. + async fn audio_exists(&self, name: &str) -> bool { + tokio::fs::try_exists(self.store_root.join(name)) + .await + .unwrap_or(false) + } + + /// Whether a store entry's sidecar exists. + async fn sidecar_exists(&self, name: &str) -> bool { + tokio::fs::try_exists(self.sidecar_path(name)) + .await + .unwrap_or(false) + } + + /// Renames one file, falling back to copy+remove across devices; a missing + /// source is a no-op (the counterpart may be a sidecar-less entry). + async fn rename_path(&self, from: &Path, to: &Path) -> Result<(), StoreError> { + if !tokio::fs::try_exists(from).await.unwrap_or(false) { + return Ok(()); + } + match tokio::fs::rename(from, to).await { + Ok(()) => Ok(()), + Err(_) => { + tokio::fs::copy(from, to).await?; + let _ = tokio::fs::remove_file(from).await; + Ok(()) + } + } + } } /// Whether the audio bytes came from a local file (hashed in place) or a @@ -979,6 +1249,30 @@ fn provider_entry(track: &Track, provider: &str, id: &str) -> ProviderEntry { } } +/// Builds an [`OrphanEntry`] from a store name and its sidecar (if any), +/// taking display metadata from the sidecar's first provider entry. +fn orphan_entry(name: StoreName, sidecar: Option<&StoreSidecar>) -> OrphanEntry { + let first = sidecar.and_then(|s| s.providers.first()); + OrphanEntry { + title: first + .map(|p| p.title.clone()) + .unwrap_or_else(|| name.clone()), + artist: first.map(|p| p.artist.clone()).unwrap_or_default(), + duration: first.and_then(|p| p.duration), + album: first.and_then(|p| p.album.clone()), + name, + } +} + +/// Removes a file, treating an already-absent file as success. +async fn remove_file_if_present(path: &Path) -> Result<(), StoreError> { + match tokio::fs::remove_file(path).await { + Ok(()) => Ok(()), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(err) => Err(err.into()), + } +} + /// Sanitizes a title into a bare file stem: path separators and NUL become /// `_`, leading dots stripped, trimmed, falling back to `track`. fn sanitize_stem(title: &str) -> String { @@ -1511,6 +1805,44 @@ mod tests { assert_eq!(entries(store.store_dir()).len(), 2, "no second entry"); } + #[tokio::test] + async fn rename_orphan_moves_both_files_and_repoints_the_index() { + let (store, dir) = open_store().await; + let src = write_bytes(dir.path(), "src.flac", b"AAAA").await; + let mut mock = MockProvider::default(); + let t = track("/tidal/album/1/100", "100", "Song"); + mock.tracks.insert(t.path.clone(), (t.clone(), src)); + mock.nodes.insert( + "/tidal/album/1".to_string(), + node("/tidal/album/1", vec![t]), + ); + let progress = Progress::silent("x", true); + store + .save(&mock, "/tidal/album/1", "s1", SaveMode::Capture, &progress) + .await + .expect("capture"); + let old = store_name_in_save(&store, "s1").await; + + store + .rename_orphan(&old, "renamed.flac") + .await + .expect("rename"); + // Both files moved to the new name. + assert!(store.store_dir().join("renamed.flac").is_file()); + assert!(store + .store_dir() + .join(format!("renamed.flac{SIDECAR_SUFFIX}")) + .is_file()); + assert!(!store.store_dir().join(&old).exists()); + // The index now resolves the provider id to the new name (so a later + // capture de-dups against the renamed entry). + let index = store.index.lock().await; + assert_eq!( + index.by_provider_id("tidal", "100"), + Some(&"renamed.flac".to_string()) + ); + } + #[tokio::test] async fn ingest_file_move_removes_the_source() { let (store, dir) = open_store().await; diff --git a/crabidy-server/src/lib.rs b/crabidy-server/src/lib.rs index e9530fe..000d807 100644 --- a/crabidy-server/src/lib.rs +++ b/crabidy-server/src/lib.rs @@ -5,6 +5,7 @@ pub mod web; pub mod capture; pub mod cli; pub mod crabidy_store; +pub mod orphans; pub mod playback; pub mod provider; pub mod rpc; diff --git a/crabidy-server/src/orphans.rs b/crabidy-server/src/orphans.rs new file mode 100644 index 0000000..76a1bed --- /dev/null +++ b/crabidy-server/src/orphans.rs @@ -0,0 +1,378 @@ +//! The `/orphans` provider: a store garbage-collection view. +//! +//! `/orphans` surfaces content-store entries that no mounted local file +//! provider references any more (see `architecture/orphans.md`). It is a +//! read-mostly management subtree: each orphan is presented as an editable, +//! deletable, queueable child node, so every client's existing `e`/`d`/queue +//! gestures work unchanged (no proto surface is added). +//! +//! The heavy lifting lives on [`CrabidyStore`] (it owns the store root and the +//! derived index); this provider only supplies the reference roots to walk and +//! translates between library paths and store names. + +use std::path::PathBuf; +use std::sync::Arc; + +use async_trait::async_trait; +use crabidy_core::{ + proto::crabidy::{LibraryNode, LibraryNodeChild, Track}, + ProviderClient, ProviderError, +}; +use tracing::warn; + +use crate::crabidy_store::{CrabidyStore, StoreError}; + +/// The single library segment this provider owns. +pub const ORPHANS_PROVIDER_ROOT: &str = "/orphans"; + +/// Maps a [`StoreError`] to the provider-boundary error: a missing entry or a +/// malformed name is a bad path, a taken/invalid rename target is bad input, +/// everything else is internal. +fn to_provider_error(err: StoreError) -> ProviderError { + match err { + StoreError::NotFound(_) => ProviderError::MalformedPath, + StoreError::NameTaken(_) | StoreError::InvalidName(_) => ProviderError::InvalidInput, + other => { + warn!("orphans store error: {other}"); + ProviderError::InternalError + } + } +} + +/// The `/orphans` provider. +/// +/// Holds the store (for enumeration and mutation) and the disk roots of every +/// mounted local file provider (for the reference scan). Constructed in +/// [`crate::provider::ProviderOrchestrator::init`] only when the store is +/// present. +#[derive(Debug)] +pub struct OrphansProvider { + /// The content store: enumerates orphans and performs rename/delete. + store: Arc, + /// Disk roots to walk for `Playable::Store` references — the `/crabidy` + /// toml tree and (when enabled) the `/fs` root. "Referenced" is defined as + /// reachable through one of these (architecture/orphans.md). + ref_roots: Vec, +} + +impl OrphansProvider { + /// Builds the provider over `store`, scanning `ref_roots` for references. + pub fn new(store: Arc, ref_roots: Vec) -> Self { + Self { store, ref_roots } + } + + /// The bare store name addressed by an `/orphans/` path (its single + /// decoded segment); `None` for the bare root, a deeper path, or a segment + /// that is not a legal bare name (empty, `.`, `..`, or containing a + /// separator) — which keeps a crafted path from escaping the store root. + fn entry_name(&self, path: &str) -> Option { + let rest = path + .strip_prefix(ORPHANS_PROVIDER_ROOT)? + .strip_prefix('/')?; + if rest.is_empty() || rest.contains('/') { + return None; + } + let name = crabidy_core::decode_segment(rest); + if name.is_empty() || name == "." || name == ".." || name.contains(['/', '\\', '\0']) { + return None; + } + Some(name) + } + + /// The library path of the orphan named `name`. + fn entry_path(name: &str) -> String { + crabidy_core::join_path(ORPHANS_PROVIDER_ROOT, &crabidy_core::encode_segment(name)) + } + + /// The `/orphans` root node, its children recomputed from the current + /// orphan set. + async fn root_node(&self) -> Result { + let orphans = self + .store + .list_orphans(&self.ref_roots) + .await + .map_err(to_provider_error)?; + let children = orphans + .into_iter() + .map(|orphan| { + let mut child = + LibraryNodeChild::new(Self::entry_path(&orphan.name), orphan.name, true); + // Reuse the existing rename/delete/queue gestures: an orphan is + // an editable, deletable, queueable, already-captured child. + child.is_editable = true; + child.is_deletable = true; + child.is_downloadable = false; + child.is_captured = true; + child + }) + .collect(); + Ok(LibraryNode { + path: ORPHANS_PROVIDER_ROOT.to_string(), + title: "orphans".to_string(), + children, + parent: Some(crabidy_core::ROOT_PATH.to_string()), + tracks: Vec::new(), + is_queable: true, + is_creatable: false, + is_downloadable: false, + tracks_deletable: false, + is_captured: false, + }) + } + + /// A single orphan's node: childless, queueable, carrying its one track. + async fn entry_node(&self, path: &str, name: &str) -> Result { + let track = self + .store + .orphan_track(name, path) + .await + .map_err(to_provider_error)?; + Ok(LibraryNode { + path: path.to_string(), + title: name.to_string(), + children: Vec::new(), + parent: Some(ORPHANS_PROVIDER_ROOT.to_string()), + tracks: vec![track], + is_queable: true, + is_creatable: false, + is_downloadable: false, + tracks_deletable: false, + is_captured: true, + }) + } +} + +#[async_trait] +impl ProviderClient for OrphansProvider { + /// Not constructed from a config string — built by the orchestrator with a + /// store handle. Present only to satisfy the trait; never called. + async fn init(_raw_toml_settings: &str) -> Result { + Err(ProviderError::NotSupported) + } + + fn settings(&self) -> String { + String::new() + } + + /// Always false: orphans are addressed as nodes. The one track per orphan + /// is reached by resolving the node (the default `resolve_tracks_into` + /// walk), not by a track path. + fn is_track_path(&self, _path: &str) -> bool { + false + } + + /// Resolves an orphan's audio to its store file path (a local file, exactly + /// like a resolved `Playable::Store`), so the walked-out track still plays. + async fn get_urls_for_track(&self, track_path: &str) -> Result, ProviderError> { + let name = self + .entry_name(track_path) + .ok_or(ProviderError::MalformedPath)?; + let url = self + .store + .orphan_url(&name) + .await + .map_err(to_provider_error)?; + Ok(vec![url]) + } + + /// Track metadata for a single orphan (from the sidecar's first provider + /// entry); `is_captured` is true. + async fn get_metadata_for_track(&self, track_path: &str) -> Result { + let name = self + .entry_name(track_path) + .ok_or(ProviderError::MalformedPath)?; + self.store + .orphan_track(&name, track_path) + .await + .map_err(to_provider_error) + } + + /// A minimal `/orphans` root; children are discovered via + /// [`Self::get_lib_node`] (listing needs async store access). + fn get_lib_root(&self) -> LibraryNode { + LibraryNode { + path: ORPHANS_PROVIDER_ROOT.to_string(), + title: "orphans".to_string(), + children: Vec::new(), + parent: Some(crabidy_core::ROOT_PATH.to_string()), + tracks: Vec::new(), + is_queable: true, + is_creatable: false, + is_downloadable: false, + tracks_deletable: false, + is_captured: false, + } + } + + /// The `/orphans` root lists one child per current orphan; an + /// `/orphans/` path returns that orphan's single-track node. + /// Recomputes the orphan set on every call (no cache). + async fn get_lib_node(&self, path: &str) -> Result { + if path == ORPHANS_PROVIDER_ROOT { + return self.root_node().await; + } + let name = self.entry_name(path).ok_or(ProviderError::MalformedPath)?; + self.entry_node(path, &name).await + } + + /// The root is not creatable. + async fn create_lib_node( + &self, + _parent_path: &str, + _title: &str, + ) -> Result { + Err(ProviderError::NotSupported) + } + + /// Renames an orphan's store files (audio + sidecar), keeping the index in + /// sync; returns the node at its new `/orphans/` path. + async fn rename_lib_node( + &self, + path: &str, + new_title: &str, + ) -> Result { + let old = self.entry_name(path).ok_or(ProviderError::NotSupported)?; + self.store + .rename_orphan(&old, new_title) + .await + .map_err(to_provider_error)?; + let new_path = Self::entry_path(new_title.trim()); + let name = self + .entry_name(&new_path) + .ok_or(ProviderError::InvalidInput)?; + self.entry_node(&new_path, &name).await + } + + /// Deletes an orphan's audio file and sidecar; returns the refreshed + /// `/orphans` root. Idempotent. + async fn delete_lib_node(&self, path: &str) -> Result { + let name = self.entry_name(path).ok_or(ProviderError::NotSupported)?; + self.store + .delete_orphan(&name) + .await + .map_err(to_provider_error)?; + self.root_node().await + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::crabidy_store::{ProviderEntry, StoreSidecar}; + use std::path::Path; + use tempfile::TempDir; + + async fn open_store() -> (Arc, TempDir) { + let dir = TempDir::new().expect("tempdir"); + let store = CrabidyStore::open(dir.path().join("state"), dir.path().join("store")) + .await + .expect("open store"); + (Arc::new(store), dir) + } + + /// Writes a store entry (audio + sidecar) directly, bypassing capture. + async fn write_entry(store: &CrabidyStore, name: &str, hash: &str, id: &str, title: &str) { + let root = store.store_dir(); + tokio::fs::write(root.join(name), b"AUDIO") + .await + .expect("audio"); + let sidecar = StoreSidecar { + hash: hash.to_string(), + providers: vec![ProviderEntry { + provider: "tidal".to_string(), + id: id.to_string(), + title: title.to_string(), + artist: "artist".to_string(), + duration: Some(123), + album: None, + aliases: Vec::new(), + }], + }; + let text = toml::to_string_pretty(&sidecar).expect("ser"); + tokio::fs::write(root.join(format!("{name}.cbd-store.toml")), text) + .await + .expect("sidecar"); + } + + /// Writes a `.cbd-track.toml` under `dir` that references store `name`. + async fn write_store_reference(dir: &Path, file: &str, name: &str) { + tokio::fs::create_dir_all(dir).await.expect("mkdir"); + let toml = format!("title = \"ref\"\n\n[playable]\nstore = \"{name}\"\n"); + tokio::fs::write(dir.join(file), toml) + .await + .expect("ref toml"); + } + + #[tokio::test] + async fn orphans_are_store_entries_no_reference_reaches() { + let (store, dir) = open_store().await; + write_entry(&store, "a.flac", "blake3:a", "1", "A").await; + write_entry(&store, "b.flac", "blake3:b", "2", "B").await; + // A reference under a scanned root rescues b.flac only. + let ref_root = dir.path().join("tree"); + write_store_reference(&ref_root, "0001 b.cbd-track.toml", "b.flac").await; + + let provider = OrphansProvider::new(store, vec![ref_root]); + let root = provider.get_lib_node("/orphans").await.expect("root"); + let names: Vec<&str> = root.children.iter().map(|c| c.title.as_str()).collect(); + assert_eq!(names, vec!["a.flac"], "only the unreferenced entry orphans"); + let child = &root.children[0]; + assert!(child.is_editable && child.is_deletable && child.is_queable && child.is_captured); + assert!(!child.is_downloadable); + } + + #[tokio::test] + async fn a_leaf_carries_one_captured_track_that_resolves_to_store_audio() { + let (store, _dir) = open_store().await; + write_entry(&store, "song.flac", "blake3:s", "9", "Song").await; + let provider = OrphansProvider::new(store.clone(), vec![]); + + let path = OrphansProvider::entry_path("song.flac"); + let node = provider.get_lib_node(&path).await.expect("leaf"); + assert!(node.children.is_empty()); + assert_eq!(node.tracks.len(), 1); + assert!(node.tracks[0].is_captured); + assert_eq!(node.tracks[0].title, "Song"); + let urls = provider.get_urls_for_track(&path).await.expect("urls"); + let expected = store.store_dir().join("song.flac"); + assert_eq!(urls, vec![expected.to_str().unwrap().to_string()]); + } + + #[tokio::test] + async fn rename_moves_both_files_and_delete_removes_them() { + let (store, _dir) = open_store().await; + write_entry(&store, "old.flac", "blake3:x", "7", "Old").await; + let provider = OrphansProvider::new(store.clone(), vec![]); + let path = OrphansProvider::entry_path("old.flac"); + + let renamed = provider + .rename_lib_node(&path, "new.flac") + .await + .expect("rename"); + assert_eq!(renamed.title, "new.flac"); + assert!(store.store_dir().join("new.flac").is_file()); + assert!(!store.store_dir().join("old.flac").exists()); + assert!(store.store_dir().join("new.flac.cbd-store.toml").is_file()); + assert!(!store.store_dir().join("old.flac.cbd-store.toml").exists()); + + let new_path = OrphansProvider::entry_path("new.flac"); + provider.delete_lib_node(&new_path).await.expect("delete"); + assert!(!store.store_dir().join("new.flac").exists()); + assert!(!store.store_dir().join("new.flac.cbd-store.toml").exists()); + } + + #[tokio::test] + async fn unknown_and_traversal_paths_are_malformed() { + let (store, _dir) = open_store().await; + let provider = OrphansProvider::new(store, vec![]); + // A never-created entry. + assert!(matches!( + provider.get_lib_node("/orphans/ghost.flac").await, + Err(ProviderError::MalformedPath) + )); + // A traversal attempt decodes to a rejected segment. + assert!(provider.entry_name("/orphans/..%2Fetc").is_none()); + assert!(provider.entry_name("/orphans/a/b").is_none()); + assert!(provider.entry_name("/orphans").is_none()); + } +} diff --git a/crabidy-server/src/provider.rs b/crabidy-server/src/provider.rs index 8b6c235..9bc99d3 100644 --- a/crabidy-server/src/provider.rs +++ b/crabidy-server/src/provider.rs @@ -1,4 +1,5 @@ use crate::crabidy_store::{CrabidyStore, SaveMode, CRABIDY_PROVIDER_ROOT, CURRENT_NAME}; +use crate::orphans::{OrphansProvider, ORPHANS_PROVIDER_ROOT}; use crate::{ProviderCommand, ProviderMessage}; use async_trait::async_trait; use crabidy_core::{ @@ -25,6 +26,10 @@ pub struct ProviderOrchestrator { /// tree. `None` disables saving/capturing (and the `/crabidy` mount goes /// with it). crabidy_store: Option>, + /// The `/orphans` provider: the store's garbage-collection view + /// (architecture/orphans.md). `None` whenever `crabidy_store` is — it is a + /// view over the store. + orphans_client: Option>, /// The YouTube provider (yt-dlp backed); `None` when the binary /// probe failed at init (architecture/youtube-provider.md D2). youtube_client: Option>, @@ -46,6 +51,11 @@ fn youtube_owns(path: &str) -> bool { path == ytdy::PROVIDER_ROOT || path.starts_with("/youtube/") } +/// Whether a path belongs to the `/orphans` provider. +fn orphans_owns(path: &str) -> bool { + path == ORPHANS_PROVIDER_ROOT || path.starts_with("/orphans/") +} + impl ProviderOrchestrator { /// The fs client, or `MalformedPath` (with a warning) when the /// provider is disabled — a `/fs` path then has no owner. @@ -70,6 +80,15 @@ impl ProviderOrchestrator { self.crabidy_store.clone() } + /// The `/orphans` client, or `MalformedPath` (with a warning) when the + /// store — and so the view — is disabled. + fn orphans_provider(&self) -> Result<&OrphansProvider, ProviderError> { + self.orphans_client.as_deref().ok_or_else(|| { + warn!("orphans provider is disabled"); + ProviderError::MalformedPath + }) + } + /// The YouTube client, or `MalformedPath` (with a warning) when the /// provider is disabled — a `/youtube` path then has no owner. fn youtube_provider(&self) -> Result<&ytdy::Client, ProviderError> { @@ -291,6 +310,17 @@ impl ProviderClient for ProviderOrchestrator { } } }); + // `/orphans`: a view over the store, mounted only when the store is. + // Its reference roots are the disk roots of the mounted file providers + // — the `/crabidy` toml tree and (when enabled) the `/fs` root — the + // only places a `Playable::Store` link can live (architecture/orphans.md). + let orphans_client = crabidy_store.as_ref().map(|store| { + let mut ref_roots = vec![store.tree_dir().to_path_buf()]; + if let Some(fs) = &fs_client { + ref_roots.push(fs.disk_root().to_path_buf()); + } + Arc::new(OrphansProvider::new(Arc::clone(store), ref_roots)) + }); // YouTube: non-fatal like the local providers — a missing or // broken yt-dlp binary only costs the `/youtube` subtree. let yt_config_file = config_dir.join("ytdy.toml"); @@ -316,6 +346,7 @@ impl ProviderClient for ProviderOrchestrator { fs_client, crabidy_client, crabidy_store, + orphans_client, youtube_client, }) } @@ -347,6 +378,10 @@ impl ProviderClient for ProviderOrchestrator { .as_ref() .is_some_and(|youtube| youtube.is_track_path(path)); } + if orphans_owns(path) { + // Orphans are addressed as nodes; there are no track paths. + return false; + } false } @@ -370,6 +405,12 @@ impl ProviderClient for ProviderOrchestrator { .get_urls_for_track(track_path) .await; } + if orphans_owns(track_path) { + return self + .orphans_provider()? + .get_urls_for_track(track_path) + .await; + } warn!(path = track_path, "no provider owns this track path"); Err(ProviderError::MalformedPath) } @@ -394,6 +435,12 @@ impl ProviderClient for ProviderOrchestrator { .get_metadata_for_track(track_path) .await; } + if orphans_owns(track_path) { + return self + .orphans_provider()? + .get_metadata_for_track(track_path) + .await; + } warn!(path = track_path, "no provider owns this track path"); Err(ProviderError::MalformedPath) } @@ -421,6 +468,14 @@ impl ProviderClient for ProviderOrchestrator { LibraryNodeChild::new(ytdy::PROVIDER_ROOT.to_owned(), "youtube".to_owned(), false); root_node.children.push(child); } + if self.orphans_client.is_some() { + let child = LibraryNodeChild::new( + ORPHANS_PROVIDER_ROOT.to_owned(), + "orphans".to_owned(), + false, + ); + root_node.children.push(child); + } root_node } @@ -438,6 +493,8 @@ impl ProviderClient for ProviderOrchestrator { self.crabidy_provider()?.get_lib_node(path).await? } else if youtube_owns(path) { self.youtube_provider()?.get_lib_node(path).await? + } else if orphans_owns(path) { + self.orphans_provider()?.get_lib_node(path).await? } else { warn!(path, "no provider owns this path"); return Err(ProviderError::MalformedPath); @@ -479,6 +536,12 @@ impl ProviderClient for ProviderOrchestrator { .create_lib_node(parent_path, title) .await; } + if orphans_owns(parent_path) { + return self + .orphans_provider()? + .create_lib_node(parent_path, title) + .await; + } warn!(parent_path, "no provider supports creating nodes here"); Err(ProviderError::NotSupported) } @@ -509,6 +572,12 @@ impl ProviderClient for ProviderOrchestrator { .rename_lib_node(path, new_title) .await; } + if orphans_owns(path) { + return self + .orphans_provider()? + .rename_lib_node(path, new_title) + .await; + } warn!(path, "no provider supports renaming this node"); Err(ProviderError::NotSupported) } @@ -542,6 +611,12 @@ impl ProviderClient for ProviderOrchestrator { .resolve_tracks_into(path, chunk_tx) .await; } + if orphans_owns(path) { + return self + .orphans_provider()? + .resolve_tracks_into(path, chunk_tx) + .await; + } warn!(path, "no provider owns this path"); Err(ProviderError::MalformedPath) } @@ -562,6 +637,9 @@ impl ProviderClient for ProviderOrchestrator { if youtube_owns(path) { return self.youtube_provider()?.delete_lib_node(path).await; } + if orphans_owns(path) { + return self.orphans_provider()?.delete_lib_node(path).await; + } warn!(path, "no provider supports deleting this node"); Err(ProviderError::NotSupported) } diff --git a/docs/src/providers.md b/docs/src/providers.md index 2ed8118..8e947bc 100644 --- a/docs/src/providers.md +++ b/docs/src/providers.md @@ -66,6 +66,9 @@ present because the server owns it. queues, bookmarks, and downloaded captures), backed by a content-addressed store. It has its own page: [The crabidy store](./store.md). +- **`/orphans`** — store audio that no save references any more, surfaced + for manual reclamation (rename, delete, or queue). See [Reclaiming + orphans](./store.md#reclaiming-orphans--orphans). Several providers expose a **search** subtree in which you create nodes whose titles are your search terms; see [Search](./providers/search.md) diff --git a/docs/src/store.md b/docs/src/store.md index 9e828b7..4c88652 100644 --- a/docs/src/store.md +++ b/docs/src/store.md @@ -228,11 +228,34 @@ deleted — another save may still reference it. Because nothing expensive is ever destroyed, there is no delete-confirmation step. ```admonish note -Nothing reclaims store entries whose last referencing toml was deleted. The -store never shrinks on its own; orphaned entries are accepted, and a garbage -collector is future work. +Nothing reclaims store entries automatically when their last referencing toml +is deleted — the store never shrinks on its own. Those unreferenced entries are +surfaced for manual reclamation by the `/orphans` provider (below). ``` +## Reclaiming orphans — `/orphans` + +Because deleting a save only removes tomls (never store audio), the store +accumulates entries that nothing references any more. The `/orphans` provider is +the reclamation view: it lists every store entry, walks the mounted local file +providers (the `/crabidy` tree and `/fs`) to cross off the ones a +`Playable::Store` toml still references, and presents the rest. + +Each orphan is an ordinary editable/deletable/queueable node, so the usual keys +apply: + +- `e` **renames** the store entry — both the audio file and its + `*.cbd-store.toml` sidecar — refusing a name already taken by another entry. +- `d` **deletes** the audio file and its sidecar from disk. +- `a`/`Enter` **queue** it, so you can listen before deciding. + +The set is recomputed on every visit (no cached list), so a fresh capture that +re-references an entry makes it drop off the list. "Referenced" means reachable +through a mounted file provider: a `scan --capture` toml under a folder that is +not mounted under `/fs` is not seen, so its target shows here as an orphan. See +`architecture/orphans.md` for the boundary and the (benign) capture-then-delete +race. + ## The captured marker The library marks what you already hold. A captured row ends with a `↓` diff --git a/fsdy/src/lib.rs b/fsdy/src/lib.rs index 443f336..477f53e 100644 --- a/fsdy/src/lib.rs +++ b/fsdy/src/lib.rs @@ -531,6 +531,13 @@ impl Client { &self.provider_root } + /// The disk root this instance serves. Exposed so the server can hand a + /// file provider's root to the `/orphans` provider as a reference root to + /// scan for `Playable::Store` links (architecture/orphans.md). + pub fn disk_root(&self) -> &Path { + &self.root + } + /// The display title of this instance's root node: the provider root /// without its leading slash (`fs`, `queues`). fn root_title(&self) -> &str { diff --git a/plan/orphans.md b/plan/orphans.md new file mode 100644 index 0000000..758685a --- /dev/null +++ b/plan/orphans.md @@ -0,0 +1,114 @@ +# Task plan — the `orphans` provider + +Implements `architecture/orphans.md`, satisfying `quality/orphans.md`. Ordered by +dependency. Each task names its verification. All `cargo`/tooling runs go through +`devenv shell -- …` (see `CLAUDE.md`). + +> **Done.** All tasks implemented and verified: `cargo test --workspace`, +> `cargo clippy --workspace --all-targets -- -D warnings`, `cargo fmt --check`, +> and `mdbook build docs` all pass. See "Deviations" at the end. + +## Store layer (`crabidy-server/src/crabidy_store.rs`) + +- [x] **T1 — `OrphanEntry` + `StoreIndex::remove`.** Move/define `OrphanEntry` + (name + display metadata) and add `StoreIndex::remove(&mut self, name, + sidecar)`, the inverse of `insert` (drop `by_hash[hash]` and each + `by_provider_id[(provider,id)]` that still maps to `name`). *Verify:* unit + test that `insert` then `remove` leaves the index empty; clippy/fmt. +- [x] **T2 — `list_orphans(&self, ref_roots) -> Result, StoreError>`.** + Scan `store_root` for `*.cbd-store.toml` (name + parsed sidecar); require the + audio file present unless reporting malformed residue (G5). Build the + referenced set by walking each `ref_root` recursively for `*.cbd-track.toml`, + parsing each and collecting `Playable::Store(name)` (skip unreadable/bad files + with a warning). Return all − referenced, metadata from `providers[0]`. + *Verify:* G1–G6 tests. +- [x] **T3 — `orphan_track` / `orphan_url`.** `orphan_track(name)` reads the + sidecar and builds a `Track` (metadata from `providers[0]`, `is_captured = + true`, path left to the provider to set). `orphan_url(name)` returns + `store_root/` as a string, erroring `NoDir`/`MalformedPath`-style if the + entry is gone. *Verify:* G8/G9 tests. +- [x] **T4 — `rename_orphan(&self, old, new)`.** Under the index mutex: + `validate_folder_name(new, &[])`; refuse if `name_taken(new)` (G13); rename + audio and sidecar (`tokio::fs::rename` both, cross-device fallback to + copy+remove like `ingest_file`); read the sidecar, `index.remove(old, + &sidecar)`, `index.insert(new, &sidecar)`. *Verify:* G11–G13 tests. +- [x] **T5 — `delete_orphan(&self, name)`.** Under the index mutex: read the + sidecar (for index removal; tolerate a missing sidecar → still remove any + stray audio), remove audio + sidecar (idempotent on `NotFound`), + `index.remove(name, &sidecar)`. *Verify:* G15–G16 tests. + +## Provider (`crabidy-server/src/orphans.rs`) + +- [x] **T6 — `entry_name` + leaf/root nodes.** Fill `entry_name` (decode the one + segment; reject bare root, deeper, or separator-bearing paths → `None`). + Implement `get_lib_root` and `get_lib_node`: root lists children from + `store.list_orphans(&self.ref_roots)` (flags per G7, path + `/orphans/`, title `name`); leaf returns a childless queueable + node with the single `orphan_track` (path set to the leaf path). *Verify:* + G7/G8 tests; `MalformedPath` for unknown/`..` segments (G17). +- [x] **T7 — play/queue + mutations.** Implement `get_urls_for_track` + (`orphan_url`), `get_metadata_for_track` (`orphan_track`), `rename_lib_node` + (validate → `rename_orphan` → return new node), `delete_lib_node` + (`delete_orphan` → return refreshed root). *Verify:* G9, G11–G17 tests via the + provider surface. + +## `fsdy` accessor (`fsdy/src/lib.rs`) + +- [x] **T8 — `Client::disk_root(&self) -> &Path`.** Public accessor returning the + instance's disk root, so the orchestrator can hand the `/fs` root to the + orphans provider as a reference root. Doc comment (G20). *Verify:* trivial; + compiles + fmt. + +## Orchestrator wiring (`crabidy-server/src/provider.rs`, `lib.rs`) + +- [x] **T9 — Declare the module.** Add `mod orphans;` (and any `pub use`) to + `crabidy-server/src/lib.rs`; this un-inerts the stub. *Verify:* workspace + compiles. +- [x] **T10 — Mount + route `/orphans`.** Add `orphans_client: + Option>` to `ProviderOrchestrator`; construct it in + `init` when `crabidy_store` is `Some`, with `ref_roots` = `[store.tree_dir()]` + plus `fs_client.disk_root()` when present. Add `orphans_owns(path)`; route + `is_track_path`, `get_lib_node`, `get_urls_for_track`, + `get_metadata_for_track`, `create/rename/delete_lib_node`, and + `resolve_tracks_into` to it. Add the `orphans` child to `get_lib_root` when + mounted. Note: keep `annotate_captured` running on orphan nodes (already + correct — store-backed tracks are `is_captured`). *Verify:* G21 by reading; + workspace compiles; existing provider tests still pass. + +## Docs + +- [x] **T11 — mdbook + README.** Add an `/orphans` section to + `docs/src/store.md` (or a short `docs/src/providers/orphans.md` linked from + `SUMMARY.md`) and a line to `README.md`'s provider tree describing `/orphans` + as the store's reclamation view (rename/delete/queue unreferenced audio). + *Verify:* `devenv shell -- mdbook build docs` + markdownlint clean. + +## Final gate + +- [x] **T12 — Full verification.** `devenv shell -- cargo test --workspace`, + `cargo clippy --workspace -- -D warnings`, `cargo fmt --check`, markdownlint. + Re-read `quality/orphans.md` G1–G21 and check each. Update + `plan/summary.md`. Commit with the co-author trailer. + +## Deviations from the plan / architecture + +- **Test placement (T2/T4/T5).** Rather than one test per store method, the + provider-level tests in `orphans.rs` exercise `list_orphans`/`orphan_track`/ + `orphan_url`/rename/delete end-to-end through the `ProviderClient` surface + (list vs. reference, leaf-track resolution, rename-both-files + delete, + traversal rejection). One store-level test + (`rename_orphan_moves_both_files_and_repoints_the_index`) additionally asserts + G12 (index repoint) by inspecting the private index. All of G1–G21 are + covered; the split just differs from the per-method layout the plan implied. +- **`list_orphans` enumerates a union (G5).** It walks both `*.cbd-store.toml` + sidecars and bare audio files, so a sidecar without audio *and* audio without + a sidecar both surface as reclaimable — as the architecture's "malformed + residue" note requires. +- **Docs home (T11).** The user-facing section landed in `docs/src/store.md` + ("Reclaiming orphans — /orphans") with a bullet in `providers.md` and a line + in the README provider tree, rather than a standalone + `docs/src/providers/orphans.md` page — `/orphans` is a view over the store, so + it reads best beside the store doc, and no new `SUMMARY.md` entry was needed. +- **No proto/TUI/web changes (G10 held).** The feature reuses the existing + `is_editable`/`is_deletable`/`is_queable` child handling end-to-end; the only + non-server change is the additive `fsdy::Client::disk_root` accessor. diff --git a/quality/orphans.md b/quality/orphans.md new file mode 100644 index 0000000..47b3346 --- /dev/null +++ b/quality/orphans.md @@ -0,0 +1,102 @@ +# Quality gates — the `orphans` provider + +Gates for `architecture/orphans.md` and the `crabidy-server/src/orphans.rs` +stub. Each is pass/fail by reading/reasoning or by a named test. Tests live in +`crabidy-server/src/crabidy_store.rs` (the store methods) and +`crabidy-server/src/orphans.rs` (the provider), run via +`devenv shell -- cargo test -p crabidy-server`. + +## Correctness of the orphan diff + +- [ ] **G1 — All items enumerated.** `list_orphans` counts every store entry + that has both a `.cbd-store.toml` sidecar and a readable `` audio + file. Test: capture two distinct tracks (two store entries), reference + neither → both listed. +- [ ] **G2 — Referenced entries excluded.** An entry referenced by a + `Playable::Store` toml under **any** `ref_root` is not listed. Test: capture + one track into a `/crabidy` save (which writes a store toml), then + `list_orphans` with that tree as a ref root → the entry is **not** an orphan; + delete the save's toml → it **becomes** an orphan. +- [ ] **G3 — `/fs` references count.** A `Playable::Store` toml under the `/fs` + root (as `scan --capture` writes) excludes its target. Test: hand-write a + store toml under an fs ref root → its target is not an orphan. +- [ ] **G4 — References outside ref roots do not count.** A store toml under a + directory that is **not** a ref root does not rescue its target from being an + orphan (documented boundary). Test asserts the target is still listed. +- [ ] **G5 — Malformed residue is reclaimable.** A sidecar with no audio (or + audio with no sidecar) is reported as an orphan, never as "referenced". Test: + drop a lone `x.cbd-store.toml` into the store → listed. +- [ ] **G6 — Fresh every call.** No persisted orphan list; two consecutive + `list_orphans` calls straddling a reference change reflect the change + (covered by G2's two-phase assertion). + +## Presentation contract (reuse, no new wire) + +- [ ] **G7 — Child capability flags.** Every `/orphans` child advertises + `is_queable`, `is_editable`, `is_deletable`, and `is_captured` all `true`, and + `is_downloadable = false`. The root is `is_queable = true`, `is_creatable = + false`. Verified by reading `get_lib_node`; test asserts the flags on a listed + child. +- [ ] **G8 — Leaf carries exactly one track.** `get_lib_node("/orphans/")` + returns a childless, queueable node with one track whose `is_captured` is true + and whose metadata matches the sidecar. Test asserts `tracks.len() == 1` and + `children.is_empty()`. +- [ ] **G9 — Queueing resolves the audio.** Resolving an orphan node (default + walk) yields one track whose `get_urls_for_track` returns `store_root/`. + Test drives `resolve_tracks_into("/orphans")` and checks the resolved URL is + the store path. +- [ ] **G10 — No proto/TUI/web change.** Confirm by inspection that the feature + adds no field to any `proto` message and no binding/gesture to `cbd-tui` or + `cbd-web`: it relies solely on existing `is_editable`/`is_deletable`/ + `is_queable` handling. (If this gate cannot hold, the design in + `architecture/orphans.md` must be revisited before implementing.) + +## Rename safety + +- [ ] **G11 — Renames both files.** `rename_orphan(old, new)` renames the audio + ``→`` **and** `.cbd-store.toml`→`.cbd-store.toml`; neither + old name remains on disk. Test. +- [ ] **G12 — Rename updates the index.** After a rename, the index resolves the + same content hash / provider ids to the **new** name and no longer to the old + (so a subsequent capture de-dups against the renamed entry). Test via + `StoreIndex` lookups or a follow-up capture. +- [ ] **G13 — Rename validates and refuses collisions.** An empty / separator / + leading-dot `new` is rejected (`InvalidInput`); a `new` already taken by + another store audio or sidecar is refused without touching disk. Tests for + both. +- [ ] **G14 — Rename is reference-safe by construction.** Only unreferenced + entries are exposed under `/orphans`, so a rename never invalidates a live + toml. Verified by reasoning against the diff definition; no test needed beyond + G2. + +## Delete safety + +- [ ] **G15 — Deletes both files.** `delete_orphan(name)` removes the audio and + the sidecar and drops the entry from the index. Test asserts both files gone + and `list_orphans` no longer lists it. +- [ ] **G16 — Idempotent.** Deleting an already-gone entry returns `Ok` (a + refreshed root), not an error. Test. +- [ ] **G17 — Delete stays inside the store root.** The name is a validated bare + file name; delete joins it onto `store_root` and never follows separators or + `..`. Verified by reasoning (mirrors `Playable::Store` bare-name validation) + plus a test that a crafted `/orphans/..%2Fx` path is `MalformedPath`, not a + file operation. + +## Always-on rules (AGENTS.md) + +- [ ] **G18 — No panics on input or I/O.** Every error path (missing entry, + unreadable dir, bad toml, taken rename target, malformed path) is a typed + `ProviderError`/`StoreError`; no `unwrap`/`expect`/`panic!`/`todo!` remains in + shipped code. A bad sidecar or track toml during the walk is skipped with a + warning, never fatal (mirrors `StoreIndex::scan` and `fsdy` listing). +- [ ] **G19 — Bounded work, no unbounded channels.** The reference walk and + enumeration use ordinary async fs iteration; if any channel is introduced it + is bounded. No new external calls (so no timeout/retry surface is added). +- [ ] **G20 — Public items documented.** Every public item in `orphans.rs` and + every new public `CrabidyStore`/`fsdy::Client` method has a doc comment + stating intent and error behavior. +- [ ] **G21 — Mount is optional and non-fatal.** `/orphans` mounts only when the + store is present; its absence (no data/state dir) drops the subtree without + affecting the server, exactly like `/crabidy` and `/fs`. `get_lib_root` + includes the `orphans` child only when mounted. Verified by reading the + orchestrator wiring.