Allow W captures of queues and bookmarks
The /queues and /bookmarks instances now advertise is_downloadable on every node (new fsdy with_downloadable_nodes option). Because such captures mix providers, the download sink skips tracks whose source cannot be captured (unresolvable streams, local file playables) with a warning instead of aborting; real download failures stay fatal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
c6e9f71cf4
commit
8032bec8e1
|
|
@ -90,8 +90,14 @@ capture must not fill the disk. Downloads run sequentially (gentle on the
|
||||||
provider, trivially bounded memory); the whole capture already runs on a
|
provider, trivially bounded memory); the whole capture already runs on a
|
||||||
spawned task, so the orchestrator keeps serving.
|
spawned task, so the orchestrator keeps serving.
|
||||||
|
|
||||||
All-or-nothing is kept: any failed download aborts the capture and removes
|
All-or-nothing is kept for real failures: any failed download (bad
|
||||||
the temp folder. A capture that lists 30 tracks *has* 30 playable files.
|
status, transport error, timeout) aborts the capture and removes the temp
|
||||||
|
folder. The one softening: a track whose source **cannot be captured at
|
||||||
|
all** — its stream fails to resolve, or resolves to a non-http(s) target
|
||||||
|
(a local file playable) — is *skipped* with a warning instead of
|
||||||
|
aborting. Queue and bookmark captures mix providers (D4), and one local
|
||||||
|
`/fs` entry must not kill the downloadable rest; the skipped track simply
|
||||||
|
has no pair in the capture.
|
||||||
|
|
||||||
### D4 — Nodes opt in via `is_downloadable`
|
### D4 — Nodes opt in via `is_downloadable`
|
||||||
|
|
||||||
|
|
@ -100,10 +106,13 @@ New proto fields `LibraryNode.is_downloadable = 8` and
|
||||||
centrally at the end of `get_lib_node`: a node is downloadable when it is
|
centrally at the end of `get_lib_node`: a node is downloadable when it is
|
||||||
**queueable or lists tracks** (the "or lists tracks" covers search-term
|
**queueable or lists tracks** (the "or lists tracks" covers search-term
|
||||||
result nodes, which are not queueable as a whole but whose track results
|
result nodes, which are not queueable as a whole but whose track results
|
||||||
are downloadable); children mirror `is_queable`. Everything else (fs,
|
are downloadable); children mirror `is_queable`. The `/queues` and
|
||||||
queues, bookmarks, captures, orchestrator roots) leaves the default
|
`/bookmarks` instances opt in wholesale
|
||||||
`false`; capturing a capture is pointless and links must not masquerade
|
(`fsdy::Client::with_downloadable_nodes`): their entries are links into
|
||||||
as downloads.
|
downloadable providers, so `W` on a saved queue or bookmark downloads
|
||||||
|
its resolvable tracks and skips the rest (D3). `/fs` and `/captures`
|
||||||
|
stay `false` — capturing a capture is pointless, and local trees have
|
||||||
|
nothing to download.
|
||||||
|
|
||||||
Tracks carry no flag: a listed track inherits its containing node's
|
Tracks carry no flag: a listed track inherits its containing node's
|
||||||
`is_downloadable` (TUI) — a Tidal album's tracks are downloadable because
|
`is_downloadable` (TUI) — a Tidal album's tracks are downloadable because
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use crabidy_core::proto::crabidy::Track;
|
use crabidy_core::proto::crabidy::Track;
|
||||||
use crabidy_core::ProviderClient;
|
use crabidy_core::ProviderClient;
|
||||||
|
use tracing::warn;
|
||||||
|
|
||||||
/// Connect timeout for download requests.
|
/// Connect timeout for download requests.
|
||||||
pub const DOWNLOAD_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
pub const DOWNLOAD_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||||
|
|
@ -141,6 +142,13 @@ impl Downloader {
|
||||||
/// The unbounded body of [`Self::download_track`]: resolve the stream
|
/// The unbounded body of [`Self::download_track`]: resolve the stream
|
||||||
/// URL, stream the response to disk against the byte budget, then
|
/// URL, stream the response to disk against the byte budget, then
|
||||||
/// write the toml.
|
/// write the toml.
|
||||||
|
///
|
||||||
|
/// A track whose source cannot be captured — the stream fails to
|
||||||
|
/// resolve, or it resolves to something other than an http(s) URL
|
||||||
|
/// (e.g. a local file playable) — is **skipped** with a warning
|
||||||
|
/// instead of aborting the capture (architecture/captures.md D3):
|
||||||
|
/// queue and bookmark captures mix providers, and one local track
|
||||||
|
/// must not kill the rest. Actual download failures stay fatal.
|
||||||
async fn fetch_track<C>(
|
async fn fetch_track<C>(
|
||||||
&self,
|
&self,
|
||||||
client: &C,
|
client: &C,
|
||||||
|
|
@ -153,13 +161,23 @@ impl Downloader {
|
||||||
C: ProviderClient + Sync,
|
C: ProviderClient + Sync,
|
||||||
{
|
{
|
||||||
let track_path = track.path.as_str();
|
let track_path = track.path.as_str();
|
||||||
let urls = client
|
let urls = match client.get_urls_for_track(track_path).await {
|
||||||
.get_urls_for_track(track_path)
|
Ok(urls) => urls,
|
||||||
.await
|
Err(err) => {
|
||||||
.map_err(|err| CaptureError::BadSource(format!("{track_path}: {err}")))?;
|
warn!(path = track_path, "skipping uncapturable track: {err}");
|
||||||
let url = urls
|
return Ok(());
|
||||||
.first()
|
}
|
||||||
.ok_or_else(|| CaptureError::BadSource(format!("{track_path}: no stream url")))?;
|
};
|
||||||
|
let Some(url) = urls.first() else {
|
||||||
|
warn!(path = track_path, "skipping track without a stream url");
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
if !(url.starts_with("http://") || url.starts_with("https://")) {
|
||||||
|
// Local file playables (fs tracks, existing captures) resolve
|
||||||
|
// to paths, not URLs — nothing to download.
|
||||||
|
warn!(path = track_path, "skipping non-http playable");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
let download_err = |err: reqwest::Error| {
|
let download_err = |err: reqwest::Error| {
|
||||||
CaptureError::Download(format!("{track_path}: {}", err.without_url()))
|
CaptureError::Download(format!("{track_path}: {}", err.without_url()))
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -413,6 +413,46 @@ mod tests {
|
||||||
assert_eq!(fs::read_dir(store.dir()).expect("store dir").count(), 0);
|
assert_eq!(fs::read_dir(store.dir()).expect("store dir").count(), 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn download_capture_skips_uncapturable_tracks() {
|
||||||
|
// A queue-like source: a downloadable fsdy instance whose folder
|
||||||
|
// mixes an http track with a local-file track — like a persisted
|
||||||
|
// queue linking tidal and fs entries.
|
||||||
|
let url = serve("200 OK", "audio/flac", b"flacbytes".to_vec()).await;
|
||||||
|
let src = TempDir::new().expect("source tempdir");
|
||||||
|
let mix = src.path().join("mix");
|
||||||
|
fs::create_dir_all(&mix).expect("mkdir");
|
||||||
|
fs::write(
|
||||||
|
mix.join("01 web.cbd-track.toml"),
|
||||||
|
format!("title = \"web\"\n[playable]\nurl = {url:?}\n"),
|
||||||
|
)
|
||||||
|
.expect("write");
|
||||||
|
fs::write(mix.join("audio.flac"), b"local").expect("write");
|
||||||
|
fs::write(
|
||||||
|
mix.join("02 local.cbd-track.toml"),
|
||||||
|
"title = \"local\"\n[playable]\nfile = \"audio.flac\"\n",
|
||||||
|
)
|
||||||
|
.expect("write");
|
||||||
|
let source = fsdy::Client::new("/queues", src.path().to_path_buf())
|
||||||
|
.expect("source instance")
|
||||||
|
.with_downloadable_nodes();
|
||||||
|
|
||||||
|
let (store, _dir) = store().await;
|
||||||
|
store
|
||||||
|
.capture(&source, "/queues/mix", "mixed")
|
||||||
|
.await
|
||||||
|
.expect("capture succeeds despite the local track");
|
||||||
|
// The http track downloaded; the local-file track was skipped
|
||||||
|
// (no pair, no abort), keeping its listing position's prefix.
|
||||||
|
assert_eq!(
|
||||||
|
visible(&store.dir().join("mixed")),
|
||||||
|
vec![
|
||||||
|
"0001 web.cbd-track.toml".to_string(),
|
||||||
|
"0001 web.flac".into()
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn capture_validates_names_and_overwrites() {
|
async fn capture_validates_names_and_overwrites() {
|
||||||
let url = serve("200 OK", "audio/flac", b"x".to_vec()).await;
|
let url = serve("200 OK", "audio/flac", b"x".to_vec()).await;
|
||||||
|
|
|
||||||
|
|
@ -274,7 +274,9 @@ impl ProviderClient for ProviderOrchestrator {
|
||||||
let queues_client = match crabidy_server::queue_store::queues_dir() {
|
let queues_client = match crabidy_server::queue_store::queues_dir() {
|
||||||
Some(dir) => match fsdy::Client::new(QUEUES_PROVIDER_ROOT, dir) {
|
Some(dir) => match fsdy::Client::new(QUEUES_PROVIDER_ROOT, dir) {
|
||||||
Ok(client) => Some(Arc::new(
|
Ok(client) => Some(Arc::new(
|
||||||
client.with_editable_top_level(&[CURRENT_QUEUE_NAME]),
|
client
|
||||||
|
.with_editable_top_level(&[CURRENT_QUEUE_NAME])
|
||||||
|
.with_downloadable_nodes(),
|
||||||
)),
|
)),
|
||||||
Err(err) => {
|
Err(err) => {
|
||||||
warn!("queues library disabled: {err}");
|
warn!("queues library disabled: {err}");
|
||||||
|
|
@ -304,7 +306,11 @@ impl ProviderClient for ProviderOrchestrator {
|
||||||
};
|
};
|
||||||
let bookmarks_client = bookmark_store.as_ref().and_then(|store| {
|
let bookmarks_client = bookmark_store.as_ref().and_then(|store| {
|
||||||
match fsdy::Client::new(BOOKMARKS_PROVIDER_ROOT, store.dir().to_path_buf()) {
|
match fsdy::Client::new(BOOKMARKS_PROVIDER_ROOT, store.dir().to_path_buf()) {
|
||||||
Ok(client) => Some(Arc::new(client.with_editable_top_level(&[]))),
|
Ok(client) => Some(Arc::new(
|
||||||
|
client
|
||||||
|
.with_editable_top_level(&[])
|
||||||
|
.with_downloadable_nodes(),
|
||||||
|
)),
|
||||||
Err(err) => {
|
Err(err) => {
|
||||||
warn!("bookmarks library disabled: {err}");
|
warn!("bookmarks library disabled: {err}");
|
||||||
None
|
None
|
||||||
|
|
|
||||||
|
|
@ -315,6 +315,12 @@ pub struct Client {
|
||||||
/// Folder names exempt from top-level editing (e.g. the auto-persisted
|
/// Folder names exempt from top-level editing (e.g. the auto-persisted
|
||||||
/// `current` queue) and rejected as rename targets.
|
/// `current` queue) and rejected as rename targets.
|
||||||
reserved: Vec<String>,
|
reserved: Vec<String>,
|
||||||
|
/// Whether this instance's nodes advertise download captures
|
||||||
|
/// (`is_downloadable`, architecture/captures.md D4) — on for
|
||||||
|
/// `/queues` and `/bookmarks`, whose entries mostly link to
|
||||||
|
/// downloadable providers; off for `/fs` and `/captures`. Tracks
|
||||||
|
/// whose source cannot be captured are skipped by the capture walk.
|
||||||
|
downloadable_nodes: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Client {
|
impl Client {
|
||||||
|
|
@ -344,6 +350,7 @@ impl Client {
|
||||||
provider_root: provider_root.to_string(),
|
provider_root: provider_root.to_string(),
|
||||||
editable_top_level: false,
|
editable_top_level: false,
|
||||||
reserved: Vec::new(),
|
reserved: Vec::new(),
|
||||||
|
downloadable_nodes: false,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -359,6 +366,16 @@ impl Client {
|
||||||
self
|
self
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Marks every node of this instance downloadable (`is_downloadable`
|
||||||
|
/// — `W` captures the subtree, see architecture/captures.md D4). Used
|
||||||
|
/// by `/queues` and `/bookmarks`, whose entries are links into
|
||||||
|
/// downloadable providers; the capture walk skips tracks whose
|
||||||
|
/// source cannot be captured (local files, unresolvable links).
|
||||||
|
pub fn with_downloadable_nodes(mut self) -> Self {
|
||||||
|
self.downloadable_nodes = true;
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
/// The provider root this instance owns (e.g. `/fs`), as configured
|
/// The provider root this instance owns (e.g. `/fs`), as configured
|
||||||
/// through [`Self::new`].
|
/// through [`Self::new`].
|
||||||
pub fn provider_root(&self) -> &str {
|
pub fn provider_root(&self) -> &str {
|
||||||
|
|
@ -478,6 +495,7 @@ impl Client {
|
||||||
let mut child = LibraryNodeChild::new(child_lib, name, true);
|
let mut child = LibraryNodeChild::new(child_lib, name, true);
|
||||||
child.is_editable = editable;
|
child.is_editable = editable;
|
||||||
child.is_deletable = editable;
|
child.is_deletable = editable;
|
||||||
|
child.is_downloadable = self.downloadable_nodes;
|
||||||
child
|
child
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
@ -498,7 +516,7 @@ impl Client {
|
||||||
tracks,
|
tracks,
|
||||||
is_queable: true,
|
is_queable: true,
|
||||||
is_creatable: false,
|
is_creatable: false,
|
||||||
is_downloadable: false,
|
is_downloadable: self.downloadable_nodes,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -646,7 +664,7 @@ impl ProviderClient for Client {
|
||||||
tracks: Vec::new(),
|
tracks: Vec::new(),
|
||||||
is_queable: true,
|
is_queable: true,
|
||||||
is_creatable: false,
|
is_creatable: false,
|
||||||
is_downloadable: false,
|
is_downloadable: self.downloadable_nodes,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1237,6 +1255,26 @@ mod tests {
|
||||||
assert_eq!(reparsed.to_track("/queues/x.cbd-track.toml"), track);
|
assert_eq!(reparsed.to_track("/queues/x.cbd-track.toml"), track);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn downloadable_instances_flag_every_node() {
|
||||||
|
let dir = TempDir::new().expect("tempdir");
|
||||||
|
fs::create_dir_all(dir.path().join("mix")).expect("mkdir");
|
||||||
|
// Off by default: /fs and /captures stay uncapturable.
|
||||||
|
let plain = Client::new("/fs", dir.path().to_path_buf()).expect("instance");
|
||||||
|
let node = plain.get_lib_node("/fs").await.expect("list");
|
||||||
|
assert!(!node.is_downloadable);
|
||||||
|
assert!(!node.children[0].is_downloadable);
|
||||||
|
// Opted in (/queues, /bookmarks): nodes and children advertise W.
|
||||||
|
let blessed = Client::new("/queues", dir.path().to_path_buf())
|
||||||
|
.expect("instance")
|
||||||
|
.with_downloadable_nodes();
|
||||||
|
let node = blessed.get_lib_node("/queues").await.expect("list");
|
||||||
|
assert!(node.is_downloadable);
|
||||||
|
assert!(node.children[0].is_downloadable);
|
||||||
|
let nested = blessed.get_lib_node("/queues/mix").await.expect("list");
|
||||||
|
assert!(nested.is_downloadable);
|
||||||
|
}
|
||||||
|
|
||||||
// ---- mutable top level (bookmarks / saved queues) ---------------------
|
// ---- mutable top level (bookmarks / saved queues) ---------------------
|
||||||
|
|
||||||
/// An editable instance over a root with two folders (one reserved) and
|
/// An editable instance over a root with two folders (one reserved) and
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,20 @@
|
||||||
# Implementation summaries
|
# Implementation summaries
|
||||||
|
|
||||||
|
## captures follow-up: W on queues and bookmarks (2026-07-21)
|
||||||
|
|
||||||
|
Small fix on top of the captures feature: `/queues` and `/bookmarks`
|
||||||
|
nodes are now `W`-capturable. `fsdy::Client` gained
|
||||||
|
`with_downloadable_nodes()` (instance-wide `is_downloadable`, applied to
|
||||||
|
the queues and bookmarks mounts; `/fs` and `/captures` stay off), and
|
||||||
|
the download sink softens all-or-nothing for exactly one case: a track
|
||||||
|
whose source cannot be captured — stream resolution fails, or resolves
|
||||||
|
to a non-http(s) target like a local file playable — is skipped with a
|
||||||
|
warning instead of aborting, since queue/bookmark captures mix
|
||||||
|
providers. Real download failures (bad status, transport, timeout) stay
|
||||||
|
fatal. `architecture/captures.md` D3/D4 reconciled; new tests
|
||||||
|
`downloadable_instances_flag_every_node` (fsdy) and
|
||||||
|
`download_capture_skips_uncapturable_tracks` (capture store).
|
||||||
|
|
||||||
## youtube-provider (2026-07-21)
|
## youtube-provider (2026-07-21)
|
||||||
|
|
||||||
Built per `plan/youtube-provider.md`: a new workspace crate **`ytdy`**
|
Built per `plan/youtube-provider.md`: a new workspace crate **`ytdy`**
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue