Update for iprox downloads.

docs/usage.md

@@ -239,7 +239,74 @@ pridepy download-px-raw-files \
 | --- | --- | --- |
 | `-a, --accession` | ProteomeXchange accession (e.g. `PXD039236`). `--px` is a deprecated alias | required |
 | `-o, --output-folder` | Destination directory | required |
+| `-p, --protocol` | Transfer protocol: `ftp`, `aspera`, `globus`, `s3` (FTP-first with fallback) | `ftp` |
+| `-w, --parallel-files` | Download 1–32 files concurrently (across-file concurrency) | `1` |
+| `-t, --threads` | Parallel HTTP Range threads per file (1–32) for fast per-file downloads | `1` |
 | `--skip-if-downloaded-already` | Skip files already present locally | off |
+| `--preserve-structure` | Recreate the dataset's subdirectory layout under the output folder | off |
+| `--iprox-user` | Your registered iProX username (required with `--protocol aspera`; env fallback: `IPROX_USER`) | — |
+| `--aspera-key` | Path to an Aspera private key for iProX (optional with `--protocol aspera`; env fallback: `IPROX_ASPERA_KEY`) | — |
+
+iProX Aspera defaults to password authentication (your iProX account
+password), supplied via the `IPROX_ASPERA_PASSWORD` env var or an
+interactive hidden prompt. Pass `--aspera-key` instead if you have a
+registered Aspera private key. Exactly one credential is required —
+`pridepy` fails fast rather than letting `ascp` block on an interactive
+prompt, so batch/sbatch jobs must set `IPROX_ASPERA_PASSWORD` (or
+`--aspera-key`) up front.
+
+### Fast downloads: parallel files and per-file segments
+
+Combine `-w` (files in parallel) and `-t` (Range segments per file) for fast bulk downloads.
+The total concurrent connections is approximately `parallel_files × threads`.
+
+**Parallel across files (recommended for most users, no account required):**
+
+```bash
+# Download up to 8 files concurrently from ProteomeXchange
+pridepy download-px-raw-files \
+  -a PXD077178 \
+  -o ./PXD077178 \
+  -w 8
+```
+
+**Combine parallel files with per-file segments:**
+
+```bash
+# Download 8 files in parallel, each split into 4 Range segments
+pridepy download-px-raw-files \
+  -a PXD077178 \
+  -o ./out \
+  -w 8 \
+  -t 4
+```
+
+### Fast downloads with iProX Aspera (account required)
+
+iProX offers Aspera for very large bulk transfers. Aspera is faster than HTTP
+on high-bandwidth connections but requires an iProX account. Combine
+`--protocol aspera` with `--iprox-user` and a credential — either the
+`IPROX_ASPERA_PASSWORD` env var (password auth, the default) or
+`--aspera-key` (a registered Aspera private key):
+
+```bash
+# password auth (env var — safe for sbatch; no prompt/hang)
+IPROX_ASPERA_PASSWORD=... pridepy download-px-raw-files -a PXD077178 -o ./out --protocol aspera --iprox-user <user>
+
+# or with a registered key
+pridepy download-px-raw-files -a PXD077178 -o ./out --protocol aspera --iprox-user <user> --aspera-key /path/to/key
+```
+
+If run interactively without `IPROX_ASPERA_PASSWORD` or `--aspera-key`,
+`pridepy` prompts for the password (hidden input). In non-interactive/batch
+contexts (e.g. `sbatch`) you must set `IPROX_ASPERA_PASSWORD` or pass
+`--aspera-key` up front — otherwise the command errors immediately instead
+of hanging on a prompt.
+
+Aspera transfers a single batched session per dataset (`ascp --file-list`)
+and always preserves the iProX `IPX.../IPX.../` source directory tree under
+the output folder — `--preserve-structure` / flattening does not apply to
+the Aspera path.
 
 ### Go directly to the hosting repository (native MassIVE / JPOST / iProX accessions)
 
@@ -268,7 +335,7 @@ How each repository is enumerated:
 
 - **MassIVE** walks the FTPS tree at `massive-ftp.ucsd.edu` (the server requires TLS). MassIVE distributes datasets across versioned root directories (`/v01`–`/vNN`); `pridepy` discovers the correct root automatically. If FTP/FTPS is blocked by the network, `pridepy` falls back to HTTPS: it lists the dataset from the GNPS2 file index (`datasetcache.gnps2.org`) and downloads each file from the ProteoSAFe endpoint at `massive.ucsd.edu` (byte-identical to the FTPS copy).
 - **JPOST** lists files through the JSON PROXI endpoint at `https://repository.jpostdb.org/proxi/datasets/<JPSTxxxxxx>` and downloads from `ftp.jpostdb.org` over plain FTP. The PROXI listing avoids the source-IP connection limit JPOST enforces on FTP.
-- **iProX** fetches the dataset's ProteomeXchange XML from `http://download.iprox.org/<accession>/PX_<accession>.xml`, then downloads each referenced file from the same host over anonymous HTTP (with `Range` support for resume). iProX also exposes Aspera (`faspe://`) with username/password for very large bulk transfers; `pridepy` uses the public HTTP endpoint so no iProX credentials are required.
+- **iProX** fetches the dataset's ProteomeXchange XML from `http://download.iprox.org/<accession>/PX_<accession>.xml`, then downloads each referenced file from the same host over anonymous HTTP (with `Range` support for resume). iProX also exposes Aspera with username/password for very large bulk transfers; `pridepy` uses the public HTTP endpoint so no iProX credentials are required.
 
 `download-all-public-raw-files` retrieves the files stored under the dataset's
 `raw/` collection. These direct downloads support resume (REST for FTP,

pridepy/download/client.py

@@ -318,8 +318,23 @@ def download_px_raw_files(
         output_folder: str,
         skip_if_downloaded_already: bool = True,
         flatten: bool = True,
+        parallel_files: int = 1,
+        download_threads: int = 1,
+        protocol: str = "ftp",
+        iprox_user: Optional[str] = None,
+        aspera_key: Optional[str] = None,
+        aspera_password: Optional[str] = None,
     ) -> None:
         """Delegate to :meth:`ProteomeXchangeProvider.download_from_accession_or_url`."""
         return ProteomeXchangeProvider().download_from_accession_or_url(
-            px_id_or_url, output_folder, skip_if_downloaded_already, flatten=flatten
+            px_id_or_url,
+            output_folder,
+            skip_if_downloaded_already,
+            flatten=flatten,
+            parallel_files=parallel_files,
+            download_threads=download_threads,
+            protocol=protocol,
+            iprox_user=iprox_user,
+            aspera_key=aspera_key,
+            aspera_password=aspera_password,
         )

pridepy/download/iprox.py

@@ -14,6 +14,8 @@
 import logging
 import os
 import re
+import subprocess
+import tempfile
 import defusedxml.ElementTree as ET
 from typing import ClassVar, Dict, List, Optional
 from urllib.parse import urlparse
@@ -34,6 +36,8 @@ class IproxProvider(Provider):
     PX_XML_URL_TEMPLATE: ClassVar[str] = (
         "http://download.iprox.org/{accession}/PX_{accession}.xml"
     )
+    ASPERA_HOST: ClassVar[str] = "download.iprox.org"
+    ASPERA_PORT: ClassVar[str] = "33001"
     # iProX PX XML uses the same PSI-MS cvParam "name" values as JPOST PROXI,
     # so we reuse JpostProvider's category map.
     PX_CATEGORY_MAP: ClassVar[Dict[str, str]] = JpostProvider.PROXI_CATEGORY_MAP
@@ -45,6 +49,84 @@ def matches(accession: str) -> bool:
             return False
         return bool(re.fullmatch(r"IPX\d{7,10}", accession.upper()))
 
+    @staticmethod
+    def _ascp_binary() -> str:
+        # Reuse PRIDE's bundled ascp binary resolution.
+        from pridepy.download.pride import PrideProvider
+        return PrideProvider.get_ascp_binary()
+
+    @classmethod
+    def aspera_download(
+        cls,
+        urls: List[str],
+        output_folder: str,
+        user: Optional[str],
+        key_path: Optional[str] = None,
+        password: Optional[str] = None,
+        maximum_bandwidth: str = "500M",
+    ) -> None:
+        """Download iProX-hosted URLs in one batched ``ascp --file-list`` session.
+
+        Auth resolution is fail-fast and never lets ``ascp`` fall back to an
+        interactive ``Password:`` prompt (which would hang a batch/sbatch
+        job): pass ``key_path`` for key-based auth (``-i <key_path>``), or
+        ``password`` for password auth (via the ``ASPERA_SCP_PASS``
+        env var — iProX's own account password, not a key). Exactly one of
+        the two must be supplied by the caller. ``ascp --mode recv
+        --file-list`` always recreates the remote ``/IPX.../IPX.../`` source
+        tree under ``output_folder`` — this transfer path does not support
+        flattening.
+        """
+        if not user:
+            raise ValueError(
+                "iProX Aspera requires --iprox-user (your registered iProX "
+                "username)."
+            )
+        env = dict(os.environ)
+        if key_path:
+            if not os.path.isfile(key_path):
+                raise ValueError(
+                    f"iProX Aspera key not found: {key_path}. Pass "
+                    "--aspera-key <path> to your registered Aspera private "
+                    "key, or use the default HTTP transport."
+                )
+        elif password:
+            env["ASPERA_SCP_PASS"] = password
+        else:
+            raise ValueError(
+                "iProX Aspera needs a credential: set IPROX_ASPERA_PASSWORD "
+                "(your iProX account password) or pass --aspera-key <path>. "
+                "HTTP is the default alternative."
+            )
+        ascp = cls._ascp_binary()
+        remote_paths = [urlparse(u).path for u in urls]
+        os.makedirs(output_folder, exist_ok=True)
+        fd, list_path = tempfile.mkstemp(prefix="iprox_aspera_", suffix=".txt", text=True)
+        try:
+            with os.fdopen(fd, "w") as fh:
+                for path in remote_paths:
+                    fh.write(path + "\n")
+            argv = [
+                ascp, "-T", "-l", maximum_bandwidth, "-P", cls.ASPERA_PORT,
+                "-k", "1",
+            ] + (["-i", key_path] if key_path else []) + [
+                "--mode", "recv",
+                "--host", cls.ASPERA_HOST, "--file-list", list_path,
+                "--user", user, output_folder,
+            ]
+            logging.info(
+                "iProX Aspera: transferring %d file(s) via ascp --file-list",
+                len(remote_paths),
+            )
+            try:
+                subprocess.run(argv, check=True, env=env, stdin=subprocess.DEVNULL)
+            except subprocess.CalledProcessError as e:
+                raise RuntimeError(
+                    f"iProX Aspera transfer failed (exit {e.returncode})"
+                ) from e
+        finally:
+            os.remove(list_path)
+
     @staticmethod
     def _get_public_root(accession: str) -> str:
         return f"/{accession.upper()}"

pridepy/download/pride.py

@@ -53,6 +53,14 @@ class PrideProvider(Provider):
     ARCHIVE_FTP_URL_PREFIX: ClassVar[str] = "ftp://ftp.pride.ebi.ac.uk/"
     ARCHIVE_HTTPS_URL_PREFIX: ClassVar[str] = "https://ftp.pride.ebi.ac.uk/"
     S3_URL: ClassVar[str] = "https://hh.fire.sdo.ebi.ac.uk"
+    # EBI-internal FIRE S3 endpoint. Only reachable from within EBI
+    # infrastructure (compute/login nodes), where it is markedly faster and
+    # more reliable than the public FTP/HTTPS/Globus paths — which is the
+    # whole point of the ``fire`` protocol. Overridable via the
+    # ``PRIDEPY_FIRE_ENDPOINT`` env var for sites with a different host.
+    FIRE_S3_URL: ClassVar[str] = os.environ.get(
+        "PRIDEPY_FIRE_ENDPOINT", "https://hl.fire.sdo.ebi.ac.uk"
+    )
     S3_BUCKET: ClassVar[str] = "pride-public"
     PROTOCOL_ORDER: ClassVar[List[str]] = ["aspera", "s3", "ftp", "globus"]
 
@@ -138,7 +146,16 @@ def get_submitted_file_path_prefix(self, accession):
     def _protocol_sequence(protocol: str) -> List[str]:
         """
         Build the ordered list of protocols to try for a requested download mode.
+
+        ``fire`` (the EBI-internal FIRE S3 endpoint) is only ever tried when it
+        is explicitly requested — it is never folded into another protocol's
+        fallback chain, because it is unreachable outside EBI and would just
+        waste retries there. When requested, it is tried first and then falls
+        back to the public protocols so a run started outside EBI still
+        completes.
         """
+        if protocol == "fire":
+            return ["fire"] + PrideProvider.PROTOCOL_ORDER
         if protocol not in PrideProvider.PROTOCOL_ORDER:
             return []
         return [protocol] + [p for p in PrideProvider.PROTOCOL_ORDER if p != protocol]
@@ -218,7 +235,9 @@ def _get_download_url(file_record: Dict, protocol: str) -> str:
                 PrideProvider.ARCHIVE_HTTPS_URL_PREFIX,
                 1,
             )
-        if protocol == "s3":
+        if protocol in ("s3", "fire"):
+            # Both S3 modes derive the object key from the FTP path; they
+            # differ only in the FIRE endpoint host (external vs EBI-internal).
             return ftp_url
         raise ValueError(f"Unsupported protocol: {protocol}")
 
@@ -511,17 +530,26 @@ def download_files_from_globus(
 
     @staticmethod
     def download_files_from_s3(
-        file_list_json: List[Dict], output_folder: str, skip_if_downloaded_already
+        file_list_json: List[Dict],
+        output_folder: str,
+        skip_if_downloaded_already,
+        endpoint_url: Optional[str] = None,
     ):
         """
-        Download files using S3 transfer URL with a progress bar and retry logic.
+        Download files from a FIRE S3 endpoint with a progress bar and retry logic.
+
         :param file_list_json: file list in JSON format
         :param output_folder: folder to download the files
         :param skip_if_downloaded_already: Boolean value to skip the download if the file has already been downloaded.
+        :param endpoint_url: FIRE S3 endpoint. Defaults to the public
+            :attr:`S3_URL` (``hh.fire``); pass :attr:`FIRE_S3_URL` (``hl.fire``)
+            for the EBI-internal ``fire`` protocol.
         """
         if not os.path.isdir(output_folder):
             os.makedirs(output_folder, exist_ok=True)
 
+        endpoint_url = endpoint_url or PrideProvider.S3_URL
+
         # Retry and timeout config
         retry_config = Config(
             retries={"max_attempts": 5, "mode": "standard"},
@@ -533,19 +561,21 @@ def download_files_from_s3(
         s3_resource = boto3.resource(
             "s3",
             config=retry_config,
-            endpoint_url=PrideProvider.S3_URL,
+            endpoint_url=endpoint_url,
         )
         bucket = s3_resource.Bucket(PrideProvider.S3_BUCKET)
+        logging.info(
+            "Downloading %d file(s) from FIRE S3 endpoint %s (bucket %s)",
+            len(file_list_json), endpoint_url, PrideProvider.S3_BUCKET,
+        )
 
         failed: List[str] = []
         for file in file_list_json:
             try:
-                # Determine S3 or FTP path
-                download_url = (
-                    file["publicFileLocations"][0]["value"]
-                    if file["publicFileLocations"][0]["name"] == "FTP Protocol"
-                    else file["publicFileLocations"][1]["value"]
-                )
+                # Resolve the canonical FTP URL, then map it to the S3 object
+                # key: pride-public mirrors the archive layout, so the key is
+                # the archive-relative path (YYYY/MM/ACCESSION/filename).
+                download_url = PrideProvider._get_download_url(file, "ftp")
 
                 ftp_base_url = "ftp://ftp.pride.ebi.ac.uk/pride/data/archive/"
                 s3_path = download_url.replace(ftp_base_url, "")
@@ -731,11 +761,14 @@ def _batch_download_by_protocol(
                 download_threads=download_threads,
             )
             return
-        if protocol == "s3":
+        if protocol in ("s3", "fire"):
             PrideProvider.download_files_from_s3(
                 file_list,
                 output_folder,
                 skip_if_downloaded_already=skip_if_downloaded_already,
+                endpoint_url=(
+                    PrideProvider.FIRE_S3_URL if protocol == "fire" else PrideProvider.S3_URL
+                ),
             )
             return
         raise ValueError(f"Unsupported protocol: {protocol}")
@@ -925,9 +958,9 @@ def _download_files_batch(
         :param aspera_maximum_bandwidth: parameter in Aspera sets the maximum bandwidth for the transfer.
         :param skip_if_downloaded_already: Boolean value to skip the download if the file has already been downloaded.
         """
-        protocols_supported = ["ftp", "aspera", "globus", "s3"]
+        protocols_supported = ["ftp", "aspera", "globus", "s3", "fire"]
         if protocol not in protocols_supported:
-            logging.error("Protocol should be one of ftp, aspera, globus, s3")
+            logging.error("Protocol should be one of ftp, aspera, globus, s3, fire")
             return
 
         os.makedirs(output_folder, exist_ok=True)
@@ -943,8 +976,13 @@ def _download_files_batch(
 
         protocol_sequence = PrideProvider._protocol_sequence(protocol)
         primary_protocol = protocol_sequence[0]
-        # Retry with the primary protocol first, then fall back to others
-        fallback_sequence = protocol_sequence
+        # Retry with the primary protocol first, then fall back to others.
+        # ``fire`` is excluded from the per-file fallback: a FIRE failure is an
+        # endpoint-level condition (the EBI-internal host is unreachable), so it
+        # fails identically for every file. Re-attempting it per file in Phase 2
+        # would just burn retries on connections that cannot succeed — send those
+        # files straight to the public fallback protocols instead.
+        fallback_sequence = [p for p in protocol_sequence if p != "fire"]
 
         # Phase 1: batch download with the requested protocol. Reuses a single
         # FTP/S3 connection for all files (the previous behaviour) instead of