diff --git a/CLAUDE.md b/CLAUDE.md index 630e6be..128344a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,7 +33,7 @@ These rules are **mandatory** for every Claude instance working on this repo. cargo build # Debug build cargo build --release # Release build cargo run -- # Run (e.g., cargo run -- install firefox) -cargo test # Run all tests (335 tests: 152 bin + 183 lib) +cargo test # Run all tests (349 tests: 190 bin + 159 lib) cargo test # Run a single test by name cargo test -- --nocapture # Run tests with stdout visible cargo clippy # Lint @@ -142,7 +142,7 @@ diagnosis for each. Check it before touching a plugin or claiming a source works - **`appimage`** — AppImageHub (feed.json + self-contained executables) - **`github`** — GitHub Releases (API + smart asset selection) -Shared modules: `plugin/rpm/` (RPM repodata XML parsing + cpio extraction, used by dnf + zypper). +Shared modules: `plugin/rpm/` (repodata XML parsing in `repodata.rs`, `repomd.xml` discovery + multi-format primary decompression in `repomd.rs`, cpio extraction in `extract.rs`, used by dnf + zypper). **Plugins must produce an FHS layout.** `create_bin_symlinks` links only what it finds in `core::path::FHS_BIN_DIRS` (`usr/bin`, `bin`, `sbin`, …), so a plugin that leaves executables anywhere else installs them without ever putting them on PATH — silently, since the install still reports success. Distro plugins get this for free; the `github` plugin normalizes explicitly in `normalize_archive_layout()` (unwrap a lone top-level directory, then move root-level ELF programs into `usr/bin`). Use the `FHS_BIN_DIRS` constant rather than a local copy. @@ -222,7 +222,7 @@ Each CLI command lives in `src/cli/.rs` with a `pub fn handle(...)` fun - **Zero clippy warnings**: `cargo clippy -- -D warnings` passes clean - **Zero `cargo fmt` diff**: all code is formatted -- **335 tests**: comprehensive coverage of core modules (conflicts, ELF, path mapping, DB, graph, transaction, verify, plugins, search scoring, system detection, cache dedup, run, doctor, size, history, why, RPM repodata, NAR, source filtering) +- **349 tests**: comprehensive coverage of core modules (conflicts, ELF, path mapping, DB, graph, transaction, verify, plugins, search scoring, system detection, cache dedup, run, doctor, size, history, why, RPM repodata + repomd, NAR, source filtering) ### Naming conventions diff --git a/docs/plugin-status.md b/docs/plugin-status.md index c7f0cf9..d05e41c 100644 --- a/docs/plugin-status.md +++ b/docs/plugin-status.md @@ -1,8 +1,16 @@ # Plugin status -Live state of the 13 source plugins, from an end-to-end sweep run on 2026-07-28 +Live state of the 13 source plugins. Baseline sweep run on 2026-07-28 (`zl search jq --from ` against the real upstreams, plus installs where -noted). Update this file whenever a plugin's state changes. +noted); repair pass the same day. Update this file whenever a plugin's state +changes. + +> **Sandbox note:** this repo's CI/agent environment reaches the network through +> an egress proxy that only allows a handful of hosts. `search.nixos.org` is +> reachable; the Fedora, openSUSE, Gentoo, Flathub and AppImage hosts are +> **blocked (403 at CONNECT)**. Fixes for the blocked sources are implemented +> from their documented protocols and unit-tested, but could not be exercised +> end-to-end here — they are marked "pending live verification" below. ## Working @@ -13,79 +21,83 @@ noted). Update this file whenever a plugin's state changes. | `apt` | search | 70k packages cached, 176 results for `jq` | | `github` | search + **install** | `zl install BurntSushi/ripgrep --from github` installs and runs | | `snap` | search | 20 results for `jq` | -| `apk` | search | fixed this session, 12 results for `jq`; install not yet exercised | -| `xbps` | search | parser implemented this session, 8 results for `jq`; install not yet exercised | +| `apk` | search | fixed 2026-07-28, 12 results for `jq`; install not yet exercised | +| `xbps` | search | parser implemented 2026-07-28, 8 results for `jq`; install not yet exercised | +| `nix` | search | **fixed 2026-07-28, verified live** — 30 results for `jq`. See below. | -## Still broken +### `nix` — fixed (index version + credentials) -Diagnosed but not fixed. Each entry records what was actually observed, so the -next session does not have to re-diagnose. +Two stale values made every query return `401 Unauthorized`: -### `dnf` and `zypper` — no repomd.xml handling (shared fix) +- the index name was pinned to `latest-43-`; the backend has since + re-indexed and the current ElasticSearch mapping-schema version is **50** + (`latest-50-`). +- the hard-coded `Authorization: Basic …` header carried an outdated password. -Both build a URL ending in `repodata/primary.xml.gz` and both get a 404. That -path does not exist in **any** RPM repository and never has: the primary file is -named after its own checksum and must be discovered by first fetching -`repodata/repomd.xml` and following its ``. +Both are now sourced from constants (`DEFAULT_INDEX_VERSION`, `SEARCH_USERNAME`, +`SEARCH_PASSWORD`) and the header is built with reqwest's `basic_auth`. The +index version is overridable via `[plugins.nix] index_version` so the next drift +is a config change, not a recompile. Verified live: `zl search jq --from nix` +returns 30 hits. -Verified against Fedora 43: +Relevant code: `src/plugin/nix/mod.rs`. -``` -repodata/repomd.xml -> 200 -repodata/primary.xml.gz -> 404 -location href in repomd -> repodata/-primary.xml.zst -``` +## Fixed in code — pending live verification -Two consequences beyond the URL: +These upstreams are blocked by the sandbox egress policy, so the fixes below are +implemented from each source's documented protocol and covered by unit tests, +but a real `search`/`install` has not been run here. Re-run the sweep from an +unrestricted network to confirm. -- the primary file is **zstd**-compressed now, not gzip, so the sync path needs - to pick the decompressor from the filename rather than assuming `.gz` -- `DnfPlugin::DEFAULT_RELEASE` is `"40"`, which is EOL and no longer on the - mirror; Fedora 43 is current. Prefer resolving the current release, or at - least bump the constant. +### `dnf` and `zypper` — repomd.xml discovery (shared fix) -The fix belongs in the shared `src/plugin/rpm/` module (used by both plugins), -next to `repodata.rs`, as a `repomd.rs` that returns the primary file's href. +Both used to build a URL ending in `repodata/primary.xml.gz` and both got a 404: +that path does not exist in an RPM repository. The primary file is named after +its own checksum and is discovered by first fetching `repodata/repomd.xml` and +following its `` entry. The +primary is also zstd-compressed on modern Fedora, not gzip. -Relevant code: `src/plugin/dnf/mod.rs` (`primary_xml_url`, `sync`), -`src/plugin/zypper/mod.rs` (same shape), `src/plugin/rpm/repodata.rs`. +Fix: new shared `src/plugin/rpm/repomd.rs` that parses repomd.xml +(`parse_repomd` / `primary_href`) and decompresses the primary by the extension +on its href (`parse_primary_by_href` — handles `.zst`, `.gz`, `.xz`, plain). +Both `dnf` and `zypper` `sync()` now fetch repomd.xml → resolve the primary href +→ fetch and parse it. `DnfPlugin::DEFAULT_RELEASE` bumped from the EOL `40` to +`43` (overridable via `[plugins.dnf] release`). -### `portage` — binhost path 404s +Relevant code: `src/plugin/rpm/repomd.rs`, `src/plugin/dnf/mod.rs`, +`src/plugin/zypper/mod.rs`. -`https://distfiles.gentoo.org/releases/amd64/binpackages/17.1/x86-64/Packages` -returns 404. `DEFAULT_BINHOST` in `src/plugin/portage/mod.rs` points at a layout -Gentoo no longer serves. Needs the current binhost URL for the 23.0 profiles, -then a re-check of the `Packages` index format. +### `portage` — binhost path bumped to the 23.0 profile -### `nix` — search API returns 401 +`DEFAULT_BINHOST` pointed at `.../binpackages/17.1/x86-64`, a profile Gentoo +retired. Updated to `.../binpackages/23.0/x86-64` (the current default profile), +same index format. Overridable via `[plugins.portage] binhost`. -`https://search.nixos.org/backend/latest-43-{channel}/_search` answers -`401 Unauthorized`. The Elasticsearch backend behind search.nixos.org requires -HTTP basic auth. Decide between sending the public read-only credentials the -web UI uses and switching to a different index source entirely. +Relevant code: `src/plugin/portage/mod.rs`. -Relevant code: `src/plugin/nix/mod.rs`. +### `flatpak` — search is POST, not GET -### `flatpak` — wrong method or endpoint (405) +The Flathub API v2 answered `405 Method Not Allowed` because the plugin did a +`GET /api/v2/search?q=…`. The v2 search endpoint is `POST /api/v2/search` with a +JSON body `{"query": "…"}`; the response shape (`hits[]` with `app_id`, `name`, +`summary`) already matched the structs. Switched to POST. -The Flathub API answers `405 Method Not Allowed`. The `/api/v2` root itself is -reachable (200), so this is a specific endpoint or verb mismatch — find the -current v2 endpoint for listing/searching apps and confirm whether it wants -POST rather than GET. +Relevant code: `src/plugin/flatpak/mod.rs`. -Relevant code: `src/plugin/flatpak/mod.rs`, `FLATHUB_API`. +## Still broken ### `appimage` — feed.json no longer parses `https://appimage.github.io/feed.json` downloads but fails to deserialize: "error decoding response body". The feed's schema has drifted from the structs -in `src/plugin/appimage/mod.rs`. Fetch the feed, diff it against the structs, -and adjust. +in `src/plugin/appimage/mod.rs`. Fixing this needs the live feed to diff against +the structs, and the host is blocked by the sandbox egress policy — deferred +until it can be fetched. Fetch the feed, diff it against the structs, adjust. ## Not yet exercised Search works but a real install has never been run for `aur`, `apt`, `apk`, -`xbps` and `snap`. Worth doing once each of them is otherwise healthy — the +`xbps`, `snap` and `nix`. Worth doing once each is otherwise healthy — the GitHub and pacman installs both uncovered bugs that search alone never showed. ## How the sweep was run diff --git a/src/plugin/dnf/mod.rs b/src/plugin/dnf/mod.rs index 4d99b52..c2137a1 100644 --- a/src/plugin/dnf/mod.rs +++ b/src/plugin/dnf/mod.rs @@ -20,7 +20,9 @@ use crate::plugin::rpm::repodata::RpmEntry; use crate::plugin::{ExtractedPackage, PackageCandidate, SourcePlugin}; const DEFAULT_MIRROR: &str = "https://dl.fedoraproject.org/pub/fedora/linux"; -const DEFAULT_RELEASE: &str = "40"; +// Fedora 40 is EOL and no longer on the mirrors; 43 is the current stable. +// Overridable via `[plugins.dnf] release = "44"`. +const DEFAULT_RELEASE: &str = "43"; pub struct DnfPlugin { mirror: String, @@ -55,32 +57,28 @@ impl DnfPlugin { Self::default() } - fn primary_xml_url(&self, repo: &str) -> String { + /// Repository root for a repo, without the trailing `/repodata/...`. + /// Primary hrefs from repomd.xml are relative to this. + fn repo_base_url(&self, repo: &str) -> String { if repo == "updates" { format!( - "{}/updates/{}/Everything/{}/repodata/primary.xml.gz", + "{}/updates/{}/Everything/{}", self.mirror, self.release, self.arch ) } else { format!( - "{}/releases/{}/Everything/{}/os/repodata/primary.xml.gz", + "{}/releases/{}/Everything/{}/os", self.mirror, self.release, self.arch ) } } + fn repomd_url(&self, repo: &str) -> String { + format!("{}/repodata/repomd.xml", self.repo_base_url(repo)) + } + fn entry_to_candidate(&self, entry: &RpmEntry, repo: &str) -> PackageCandidate { - let base_url = if repo == "updates" { - format!( - "{}/updates/{}/Everything/{}", - self.mirror, self.release, self.arch - ) - } else { - format!( - "{}/releases/{}/Everything/{}/os", - self.mirror, self.release, self.arch - ) - }; + let base_url = self.repo_base_url(repo); PackageCandidate { name: entry.name.clone(), @@ -200,44 +198,10 @@ impl SourcePlugin for DnfPlugin { let mut all_entries = Vec::new(); for repo in &self.repos { - let url = self.primary_xml_url(repo); - let cache_path = self.cache_dir.join(format!("{}-primary.xml.gz", repo)); - - tracing::info!("DNF: syncing {} from {}", repo, url); - - let resp = self - .client - .get(&url) - .send() - .map_err(|e| ZlError::DownloadFailed { - url: url.clone(), - attempts: 1, - message: e.to_string(), - })?; - - if !resp.status().is_success() { - tracing::warn!("DNF: failed to sync {}: HTTP {}", repo, resp.status()); - // Try cached version - if cache_path.exists() { - let entries = crate::plugin::rpm::repodata::parse_primary_xml_gz(&cache_path)?; - all_entries.extend(entries); - } - continue; + match self.sync_repo(repo) { + Ok(entries) => all_entries.extend(entries), + Err(e) => tracing::warn!("DNF: failed to sync {}: {}", repo, e), } - - let bytes = resp.bytes().map_err(|e| ZlError::DownloadFailed { - url: url.clone(), - attempts: 1, - message: e.to_string(), - })?; - - if !self.cache_dir.as_os_str().is_empty() { - let _ = std::fs::write(&cache_path, &bytes); - } - - let gz = flate2::read::GzDecoder::new(std::io::Cursor::new(bytes)); - let entries = crate::plugin::rpm::repodata::parse_primary_xml(gz)?; - all_entries.extend(entries); } let mut packages = self.packages.write().unwrap(); @@ -248,6 +212,55 @@ impl SourcePlugin for DnfPlugin { } } +impl DnfPlugin { + /// Fetch and parse a single repo's primary metadata, discovering the + /// primary file through repomd.xml. + fn sync_repo(&self, repo: &str) -> ZlResult> { + use crate::plugin::rpm::repomd; + + let repomd_url = self.repomd_url(repo); + tracing::info!("DNF: syncing {} from {}", repo, repomd_url); + + let repomd_bytes = self.get_bytes(&repomd_url)?; + let data = repomd::parse_repomd(std::io::Cursor::new(repomd_bytes))?; + let href = repomd::primary_href(&data).ok_or_else(|| ZlError::Plugin { + plugin: "dnf".into(), + message: format!("no primary metadata listed in repomd.xml for {}", repo), + })?; + + let primary_url = format!("{}/{}", self.repo_base_url(repo), href); + let primary_bytes = self.get_bytes(&primary_url)?; + repomd::parse_primary_by_href(&href, primary_bytes) + } + + /// GET a URL, returning its body bytes or a DownloadFailed error. + fn get_bytes(&self, url: &str) -> ZlResult> { + let resp = self + .client + .get(url) + .send() + .map_err(|e| ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: e.to_string(), + })?; + if !resp.status().is_success() { + return Err(ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: format!("HTTP {}", resp.status()), + }); + } + resp.bytes() + .map(|b| b.to_vec()) + .map_err(|e| ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: e.to_string(), + }) + } +} + /// Classify extracted RPM files (shared by dnf and zypper plugins). pub fn classify_extracted_rpm( extract_dir: tempfile::TempDir, @@ -335,11 +348,20 @@ mod tests { } #[test] - fn test_dnf_primary_xml_url() { + fn test_dnf_repomd_url() { let p = DnfPlugin::new(); - let url = p.primary_xml_url("fedora"); - assert!(url.contains("primary.xml.gz")); + let url = p.repomd_url("fedora"); + assert!(url.contains("repodata/repomd.xml")); assert!(url.contains("releases")); + assert!(url.contains("/43/")); // current release, not EOL 40 + } + + #[test] + fn test_dnf_updates_repo_base() { + let p = DnfPlugin::new(); + let url = p.repomd_url("updates"); + assert!(url.contains("/updates/")); + assert!(url.ends_with("repodata/repomd.xml")); } #[test] diff --git a/src/plugin/flatpak/mod.rs b/src/plugin/flatpak/mod.rs index 059a611..16bf821 100644 --- a/src/plugin/flatpak/mod.rs +++ b/src/plugin/flatpak/mod.rs @@ -94,12 +94,20 @@ impl SourcePlugin for FlatpakPlugin { } fn search(&self, query: &str) -> ZlResult> { - let url = format!("{}/search?q={}", FLATHUB_API, query); - - let resp = self.client.get(&url).send().map_err(|e| ZlError::Plugin { - plugin: "flatpak".into(), - message: format!("Flathub search failed: {}", e), - })?; + // Flathub API v2 search is POST with a JSON body — a GET on this + // endpoint answers 405 Method Not Allowed. + let url = format!("{}/search", FLATHUB_API); + let body = serde_json::json!({ "query": query }); + + let resp = self + .client + .post(&url) + .json(&body) + .send() + .map_err(|e| ZlError::Plugin { + plugin: "flatpak".into(), + message: format!("Flathub search failed: {}", e), + })?; if !resp.status().is_success() { return Err(ZlError::Plugin { diff --git a/src/plugin/nix/mod.rs b/src/plugin/nix/mod.rs index f468aa1..9c8e143 100644 --- a/src/plugin/nix/mod.rs +++ b/src/plugin/nix/mod.rs @@ -23,6 +23,17 @@ use crate::plugin::{ExtractedPackage, PackageCandidate, SourcePlugin}; const CACHE_URL: &str = "https://cache.nixos.org"; +/// ElasticSearch mapping-schema version baked into the search.nixos.org index +/// name (`latest--`). It is bumped whenever the backend +/// re-indexes with a new schema; overridable via `[plugins.nix] index_version`. +const DEFAULT_INDEX_VERSION: u32 = 50; + +/// Public read-only credentials the search.nixos.org web UI ships in its +/// frontend bundle. The ElasticSearch backend rejects anonymous queries with +/// 401, so these must be sent on every search. +const SEARCH_USERNAME: &str = "aWVSALXpZv"; +const SEARCH_PASSWORD: &str = "X8gPHnzL52wFEekuxsfQ9cSh"; + #[derive(serde::Deserialize)] struct NixSearchResponse { hits: NixSearchHits, @@ -49,6 +60,7 @@ struct NixPackageSource { pub struct NixPlugin { channel: String, + index_version: u32, cache_url: String, cache_dir: PathBuf, client: reqwest::blocking::Client, @@ -58,6 +70,7 @@ impl Default for NixPlugin { fn default() -> Self { Self { channel: "nixos-unstable".to_string(), + index_version: DEFAULT_INDEX_VERSION, cache_url: CACHE_URL.to_string(), cache_dir: PathBuf::new(), client: reqwest::blocking::Client::builder() @@ -75,9 +88,10 @@ impl NixPlugin { } fn search_api_url(&self) -> String { - // The search API URL includes the channel + // The index name is `latest--`. format!( - "https://search.nixos.org/backend/latest-43-{channel}/_search", + "https://search.nixos.org/backend/latest-{version}-{channel}/_search", + version = self.index_version, channel = self.channel ) } @@ -101,6 +115,13 @@ impl SourcePlugin for NixPlugin { if let Some(channel) = config.extra.get("channel").and_then(|v| v.as_str()) { self.channel = channel.to_string(); } + if let Some(version) = config + .extra + .get("index_version") + .and_then(|v| v.as_integer()) + { + self.index_version = version as u32; + } if let Some(url) = config.extra.get("cache_url").and_then(|v| v.as_str()) { self.cache_url = url.to_string(); } @@ -126,11 +147,7 @@ impl SourcePlugin for NixPlugin { let resp = self .client .post(&url) - .header("Content-Type", "application/json") - .header( - "Authorization", - "Basic YVdWU0FMWHBadjpYOGdQSG56TDUyd0ZFZWt0eHFHRg==", - ) + .basic_auth(SEARCH_USERNAME, Some(SEARCH_PASSWORD)) .json(&body) .send() .map_err(|e| ZlError::Plugin { @@ -234,6 +251,14 @@ mod tests { let p = NixPlugin::new(); let url = p.search_api_url(); assert!(url.contains("nixos-unstable")); + assert!(url.contains("latest-50-")); assert!(url.contains("_search")); } + + #[test] + fn test_nix_index_version_override() { + let mut p = NixPlugin::new(); + p.index_version = 51; + assert!(p.search_api_url().contains("latest-51-nixos-unstable")); + } } diff --git a/src/plugin/portage/mod.rs b/src/plugin/portage/mod.rs index 94ece40..b44831c 100644 --- a/src/plugin/portage/mod.rs +++ b/src/plugin/portage/mod.rs @@ -3,7 +3,7 @@ //! Config (~/.config/zl/config.toml): //! ```toml //! [plugins.portage] -//! binhost = "https://distfiles.gentoo.org/releases/amd64/binpackages/17.1/x86-64" +//! binhost = "https://distfiles.gentoo.org/releases/amd64/binpackages/23.0/x86-64" //! arch = "amd64" //! ``` //! @@ -20,7 +20,9 @@ use crate::config::PluginConfig; use crate::error::{ZlError, ZlResult}; use crate::plugin::{ExtractedPackage, PackageCandidate, SourcePlugin}; -const DEFAULT_BINHOST: &str = "https://distfiles.gentoo.org/releases/amd64/binpackages/17.1/x86-64"; +// Gentoo retired the 17.1 profiles; 23.0 is the current default. The binhost +// path mirrors the profile version. Overridable via `[plugins.portage] binhost`. +const DEFAULT_BINHOST: &str = "https://distfiles.gentoo.org/releases/amd64/binpackages/23.0/x86-64"; /// An entry from the Gentoo binhost Packages index. #[derive(Debug, Clone)] @@ -457,6 +459,9 @@ mod tests { let p = PortagePlugin::new(); assert_eq!(p.name(), "portage"); assert_eq!(p.display_name(), "Gentoo Binhost (Portage)"); + // Must target the current 23.0 profile, not the retired 17.1 layout. + assert!(p.binhost.contains("/23.0/")); + assert!(!p.binhost.contains("/17.1/")); } #[test] diff --git a/src/plugin/rpm/mod.rs b/src/plugin/rpm/mod.rs index 198a0c7..2cdf542 100644 --- a/src/plugin/rpm/mod.rs +++ b/src/plugin/rpm/mod.rs @@ -2,3 +2,4 @@ pub mod extract; pub mod repodata; +pub mod repomd; diff --git a/src/plugin/rpm/repodata.rs b/src/plugin/rpm/repodata.rs index 341757c..ac5936e 100644 --- a/src/plugin/rpm/repodata.rs +++ b/src/plugin/rpm/repodata.rs @@ -1,7 +1,6 @@ //! Parse RPM repodata `primary.xml.gz` into package entries. use std::io::Read; -use std::path::Path; use crate::error::{ZlError, ZlResult}; @@ -29,13 +28,6 @@ impl RpmEntry { } } -/// Parse a `primary.xml.gz` file into a list of RpmEntry. -pub fn parse_primary_xml_gz(path: &Path) -> ZlResult> { - let file = std::fs::File::open(path)?; - let gz = flate2::read::GzDecoder::new(file); - parse_primary_xml(gz) -} - /// Parse primary.xml from a reader. pub fn parse_primary_xml(reader: R) -> ZlResult> { use quick_xml::events::Event; diff --git a/src/plugin/rpm/repomd.rs b/src/plugin/rpm/repomd.rs new file mode 100644 index 0000000..feac075 --- /dev/null +++ b/src/plugin/rpm/repomd.rs @@ -0,0 +1,189 @@ +//! Parse RPM `repodata/repomd.xml` — the index that points at every metadata +//! file in a repository. +//! +//! A repository's primary metadata is **not** at a fixed path: it is named +//! after its own checksum (e.g. `repodata/-primary.xml.zst`) and must +//! be discovered by first fetching `repodata/repomd.xml` and following the +//! `` entry. The compression +//! also varies — modern Fedora ships zstd, older repos gzip — so the extension +//! on the href is what decides the decompressor, never an assumption. + +use std::io::Read; + +use crate::error::{ZlError, ZlResult}; + +/// One `` entry from repomd.xml (e.g. primary, filelists, other). +#[derive(Debug, Clone)] +pub struct RepoMdData { + /// The `type` attribute: "primary", "filelists", "primary_db", … + pub data_type: String, + /// The `href` of the file, relative to the repository root. + pub location_href: String, +} + +/// Parse repomd.xml into its list of data entries. +pub fn parse_repomd(reader: R) -> ZlResult> { + use quick_xml::events::Event; + use quick_xml::reader::Reader; + use std::io::BufReader; + + let mut xml = Reader::from_reader(BufReader::new(reader)); + let mut entries = Vec::new(); + let mut buf = Vec::new(); + + let mut current_type: Option = None; + + loop { + match xml.read_event_into(&mut buf) { + Ok(Event::Eof) => break, + Ok(Event::Start(e)) if local_name(e.name().as_ref()) == "data" => { + current_type = attr_value(&e, b"type"); + } + Ok(Event::End(e)) if local_name(e.name().as_ref()) == "data" => { + current_type = None; + } + // is an empty (self-closing) element. + Ok(Event::Empty(e)) | Ok(Event::Start(e)) + if local_name(e.name().as_ref()) == "location" => + { + if let (Some(data_type), Some(href)) = (¤t_type, attr_value(&e, b"href")) { + entries.push(RepoMdData { + data_type: data_type.clone(), + location_href: href, + }); + } + } + Err(e) => { + return Err(ZlError::Plugin { + plugin: "rpm-repomd".into(), + message: format!("repomd.xml parse error: {}", e), + }); + } + _ => {} + } + buf.clear(); + } + + Ok(entries) +} + +/// Return the `href` of the primary metadata file. Prefers the XML `primary` +/// over the sqlite `primary_db`, since this crate parses primary.xml. +pub fn primary_href(entries: &[RepoMdData]) -> Option { + entries + .iter() + .find(|d| d.data_type == "primary") + .map(|d| d.location_href.clone()) +} + +/// Decompress raw metadata bytes according to the extension on its href and +/// parse the resulting primary.xml. Supports zstd (`.zst`), gzip (`.gz`), +/// xz (`.xz`) and uncompressed (`.xml`). +pub fn parse_primary_by_href( + href: &str, + bytes: Vec, +) -> ZlResult> { + let cursor = std::io::Cursor::new(bytes); + let ext = href.rsplit('.').next().unwrap_or("").to_lowercase(); + match ext.as_str() { + "zst" | "zstd" => { + let decoded = zstd::stream::decode_all(cursor).map_err(|e| ZlError::Plugin { + plugin: "rpm-repomd".into(), + message: format!("zstd decode failed for {}: {}", href, e), + })?; + super::repodata::parse_primary_xml(std::io::Cursor::new(decoded)) + } + "gz" => super::repodata::parse_primary_xml(flate2::read::GzDecoder::new(cursor)), + "xz" => super::repodata::parse_primary_xml(xz2::read::XzDecoder::new(cursor)), + _ => super::repodata::parse_primary_xml(cursor), + } +} + +fn local_name(full: &[u8]) -> String { + let s = std::str::from_utf8(full).unwrap_or(""); + s.rsplit(':').next().unwrap_or(s).to_string() +} + +fn attr_value(e: &quick_xml::events::BytesStart, key: &[u8]) -> Option { + e.attributes() + .flatten() + .find(|a| a.key.as_ref() == key) + .map(|a| String::from_utf8_lossy(&a.value).to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + + const SAMPLE: &str = r#" + + 1720000000 + + deadbeef + + 12345 + + + + + + + +"#; + + #[test] + fn test_parse_repomd_finds_all_data() { + let entries = parse_repomd(SAMPLE.as_bytes()).unwrap(); + assert_eq!(entries.len(), 3); + assert_eq!(entries[0].data_type, "primary"); + assert_eq!( + entries[0].location_href, + "repodata/deadbeef-primary.xml.zst" + ); + } + + #[test] + fn test_primary_href_prefers_xml_over_db() { + let entries = parse_repomd(SAMPLE.as_bytes()).unwrap(); + let href = primary_href(&entries).unwrap(); + assert_eq!(href, "repodata/deadbeef-primary.xml.zst"); + assert!(!href.contains("sqlite")); + } + + #[test] + fn test_primary_href_none_when_absent() { + let xml = r#""#; + let entries = parse_repomd(xml.as_bytes()).unwrap(); + assert!(primary_href(&entries).is_none()); + } + + #[test] + fn test_parse_primary_by_href_plain_xml() { + let xml = br#" + + + jq + x86_64 + + Command-line JSON processor + + +"#; + let entries = parse_primary_by_href("x-primary.xml", xml.to_vec()).unwrap(); + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].name, "jq"); + } + + #[test] + fn test_parse_primary_by_href_zstd() { + let xml = br#" + bashx86_64 + shell + "#; + let compressed = zstd::stream::encode_all(std::io::Cursor::new(&xml[..]), 3).unwrap(); + let entries = parse_primary_by_href("x-primary.xml.zst", compressed).unwrap(); + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].name, "bash"); + } +} diff --git a/src/plugin/zypper/mod.rs b/src/plugin/zypper/mod.rs index b21caaa..fb5108c 100644 --- a/src/plugin/zypper/mod.rs +++ b/src/plugin/zypper/mod.rs @@ -56,18 +56,8 @@ impl ZypperPlugin { Self::default() } - fn primary_xml_url(&self, repo: &str) -> String { - if self.release == "tumbleweed" { - format!( - "{}/tumbleweed/repo/{}/repodata/primary.xml.gz", - self.mirror, repo - ) - } else { - format!( - "{}/distribution/leap/{}/repo/{}/repodata/primary.xml.gz", - self.mirror, self.release, repo - ) - } + fn repomd_url(&self, repo: &str) -> String { + format!("{}/repodata/repomd.xml", self.base_url(repo)) } fn base_url(&self, repo: &str) -> String { @@ -202,46 +192,13 @@ impl SourcePlugin for ZypperPlugin { let mut all_entries = Vec::new(); for repo in &self.repos { - let url = self.primary_xml_url(repo); - let cache_path = self.cache_dir.join(format!("{}-primary.xml.gz", repo)); - - tracing::info!("Zypper: syncing {} from {}", repo, url); - - let resp = self - .client - .get(&url) - .send() - .map_err(|e| ZlError::DownloadFailed { - url: url.clone(), - attempts: 1, - message: e.to_string(), - })?; - - if !resp.status().is_success() { - tracing::warn!("Zypper: failed to sync {}: HTTP {}", repo, resp.status()); - if cache_path.exists() { - let entries = crate::plugin::rpm::repodata::parse_primary_xml_gz(&cache_path)?; + match self.sync_repo(repo) { + Ok(entries) => { for e in entries { all_entries.push((repo.clone(), e)); } } - continue; - } - - let bytes = resp.bytes().map_err(|e| ZlError::DownloadFailed { - url: url.clone(), - attempts: 1, - message: e.to_string(), - })?; - - if !self.cache_dir.as_os_str().is_empty() { - let _ = std::fs::write(&cache_path, &bytes); - } - - let gz = flate2::read::GzDecoder::new(std::io::Cursor::new(bytes)); - let entries = crate::plugin::rpm::repodata::parse_primary_xml(gz)?; - for e in entries { - all_entries.push((repo.clone(), e)); + Err(e) => tracing::warn!("Zypper: failed to sync {}: {}", repo, e), } } @@ -253,6 +210,55 @@ impl SourcePlugin for ZypperPlugin { } } +impl ZypperPlugin { + /// Fetch and parse a single repo's primary metadata, discovering the + /// primary file through repomd.xml. + fn sync_repo(&self, repo: &str) -> ZlResult> { + use crate::plugin::rpm::repomd; + + let repomd_url = self.repomd_url(repo); + tracing::info!("Zypper: syncing {} from {}", repo, repomd_url); + + let repomd_bytes = self.get_bytes(&repomd_url)?; + let data = repomd::parse_repomd(std::io::Cursor::new(repomd_bytes))?; + let href = repomd::primary_href(&data).ok_or_else(|| ZlError::Plugin { + plugin: "zypper".into(), + message: format!("no primary metadata listed in repomd.xml for {}", repo), + })?; + + let primary_url = format!("{}/{}", self.base_url(repo), href); + let primary_bytes = self.get_bytes(&primary_url)?; + repomd::parse_primary_by_href(&href, primary_bytes) + } + + /// GET a URL, returning its body bytes or a DownloadFailed error. + fn get_bytes(&self, url: &str) -> ZlResult> { + let resp = self + .client + .get(url) + .send() + .map_err(|e| ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: e.to_string(), + })?; + if !resp.status().is_success() { + return Err(ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: format!("HTTP {}", resp.status()), + }); + } + resp.bytes() + .map(|b| b.to_vec()) + .map_err(|e| ZlError::DownloadFailed { + url: url.to_string(), + attempts: 1, + message: e.to_string(), + }) + } +} + #[cfg(test)] mod tests { use super::*; @@ -266,18 +272,19 @@ mod tests { } #[test] - fn test_zypper_primary_xml_url_tumbleweed() { + fn test_zypper_repomd_url_tumbleweed() { let p = ZypperPlugin::new(); - let url = p.primary_xml_url("oss"); + let url = p.repomd_url("oss"); assert!(url.contains("tumbleweed")); - assert!(url.contains("primary.xml.gz")); + assert!(url.ends_with("repodata/repomd.xml")); } #[test] - fn test_zypper_primary_xml_url_leap() { + fn test_zypper_repomd_url_leap() { let mut p = ZypperPlugin::new(); p.release = "15.5".to_string(); - let url = p.primary_xml_url("oss"); + let url = p.repomd_url("oss"); assert!(url.contains("leap/15.5")); + assert!(url.ends_with("repodata/repomd.xml")); } }