Add WorkflowHub importer with per-tool workflow ID outputs

[paulzierep]

Add WorkflowHub importer with per-tool workflow ID outputs

Creates a new workflowhub-import/ importer that fetches all workflows from the WorkflowHub API and produces per-tool workflow ID files for the RSEc content repo.

What it does:

• Paginates through all workflows on WorkflowHub (~1500)
• Fetches detailed metadata for each workflow (title, creators, tags, EDAM annotations, license, DOI)
• Extracts tool names from two sources: curated bio.tools annotations and Galaxy internals.steps descriptions
• Maps tool names to data/{biotool_id}/ directories using galaxy_codex Suite ID → bio.tool ID mapping, falling back to raw tool name
• Writes data/{biotool_id}/{biotool_id}.workflowhub.json containing only a list of workflow IDs
• Saves full workflow entries to imports/workflowhub/{id}.workflowhub.json
• Supports --test [N] flag to limit workflows for testing
• Outputs stats: workflow count, tool distribution, and workflow engine breakdown (Galaxy vs Nextflow vs Snakemake etc.)

Usage:

# Full run (all workflows)
python3 workflowhub-import/workflowhub-import.py
python3 workflowhub-import/workflowhub-import.py --test 100   # Test with 100

# In the RSEc content repo
# The content repo has data/ with per-tool directories; the importer
# creates *.workflowhub.json files alongside existing *.biotools.json etc.

Format of per-tool files (e.g. data/fastp/fastp.workflowhub.json):

["1738", "1739", "1740", "1741"]

imports/workflowhub/1104.workflowhub.json (workflow entry with mapped_tools):

{
    "content_blobs": [],
    "create_time": "2026-07-08",
    "creators": [
        "Amirhossein Naghsh Nilchi",
        "Pavankumar Videm",
        "Bj\u00f6rn Gr\u00fcning"
    ],
    "description": "A comprehensive Galaxy workflow for the end-to-end analysis of spatial data, featuring automated QC, clustering, spatial statistics, cell type annotation, and cell-cell communication.",
    "discussion_links": [],
    "doi": "10.48546/workflowhub.workflow.2174.5",
    "edam_operation": [
        "Clustering",
        "Dimensionality reduction",
        "Expression correlation analysis"
    ],
    "edam_topic": [
        "Single-cell sequencing"
    ],
    "id": "2174",
    "latest_version": "5",
    "license": "MIT",
    "link": "https://workflowhub.eu/workflows/2174?version=5",
    "mapped_tools": [
        "anndata",
        "celltypist",
        "liana",
        "scanpy",
        "spatialdata",
        "squidpy"
    ],
    "name": "Galaxy EISTA spatialdata + updated squidpy",
    "number_of_steps": 80,
    "projects": [],
    "source": "WorkflowHub",
    "tags": [
        "bioinformatics",
        "galaxy",
        "single cell clustering",
        "transcriptomics",
        "single-cell",
        "spatial transcriptomics"
    ],
    "tools": [
        "scanpy_inspect",
        "param_value_from_file",
        "liana",
        "scanpy_filter",
        "addvalue",
        "add_a_column1",
        "liana+",
        "compose_text_param",
        "celltypist",
        "tp_replace_in_line",
        "spatialdata_operation",
        "pick_value",
        "collection_column_join",
        "anndata",
        "scanpy",
        "set the value to a text input\nparam_value_from_file",
        "split_file_to_collection",
        "select the resolution\ncut1",
        "tp_awk_tool",
        "tp_cut_tool",
        "anndata_inspect",
        "scanpy_cluster_reduce_dimension",
        "table_compute",
        "tp_sort_header_tool",
        "scanpy_remove_confounders",
        "scanpy_normalize",
        "tp_find_and_replace",
        "scanpy_plot",
        "datamash_transpose",
        "anndata_manipulate",
        "squidpy",
        "spatialdata",
        "squidpy_graph",
        "liana_methods",
        "map_param_value"
    ],
    "update_time": "2026-07-08",
    "versions": "5",
    "workflow_class": "Galaxy"
}

Unfortunatly most nextflow / snakemake workflows do not have tools available via workflowHUB api, I made some tests to get it via ro-crate or the associated github, but this is messy. I would opt to fix this rather on the workflow hub side.

[paulzierep]

linter fails due to other scripts alread merged !

[mihai-sysbio]

@rioualen @supernord just want to make sure you are aware of this, it would be great if you could consider reviewing

[paulzierep]

Same as for tess: @mihai-sysbio imo this is ready for review, maybe @hmenager can check if the logic makes sense (ids in data folder, full metadata in dataset folder), or I can present in the next meeting, toimorrow :) @arash77 maybe you can check the code logic, worked for me locally - the question would be how to process it downstream for the atlas, so that we can see for each tool, which workflow is using it, the workflow should include at least the link, version and name I guess

[rioualen]

Thank you @mihai-sysbio for drawing my attention to this, I'll look into it!

Talking about EDAM annotations and the Codex, it seems tools and workflows are annotated with EDAM labels rather than URIs, which is not ideal. I opened an issue to discuss this in the Codex repository.

[arash77]

Same as for tess: @mihai-sysbio imo this is ready for review, maybe @hmenager can check if the logic makes sense (ids in data folder, full metadata in dataset folder), or I can present in the next meeting, toimorrow :) @arash77 maybe you can check the code logic, worked for me locally - the question would be how to process it downstream for the atlas, so that we can see for each tool, which workflow is using it, the workflow should include at least the link, version and name I guess

Checked the logic, works locally.

On your question: link, version and name are all there. That's exactly the shape the Atlas frontend already expects for workflows, so imports/workflowhub/ records are directly usable downstream.

Matches the repo conventions. CI is red from pre-existing lint/format issues on main (bioschemas-gen/, bioconductor-to-biotools/, a notebook), not this PR.

Minor: requirements.txt lists boltons but it's unused — matches the other importers, harmless.

[paulzierep]

@agnesbrnb can you check if there is some metadata missing that is created here https://github.com/research-software-ecosystem/micoreca/blob/main/bin/extract_workflowhub.py ?

[agnesbrnb]

The extraction of the complete JSON file from workflowhub is what MiCoReCa needs for filtering.
This is ok for me :)

[albangaignard]

Thanks @paulzierep for this PR ! We have been working with FAIR-checker to extract JSON-LD semantic markup, following the Bioschemas Computational Workflow profile. I do not lanage to get references to EDAM terms, but I see that you manage to do it ? How do you proceed ? Would it be possible to document it for the workflowHub dev team so that they expose more complete Bioschemas markup on each web page ?

[paulzierep]

Thanks @paulzierep for this PR ! We have been working with FAIR-checker to extract JSON-LD semantic markup, following the Bioschemas Computational Workflow profile. I do not lanage to get references to EDAM terms, but I see that you manage to do it ? How do you proceed ? Would it be possible to document it for the workflowHub dev team so that they expose more complete Bioschemas markup on each web page ?

Actually, the EDAM terms come directly form workflowHUB. The API response at /workflows/{id}.json includes them as topic_annotations and operation_annotations arrays with labels.

[agnesbrnb]

@paulzierep For MiCoReCa (research-software-ecosystem/micoreca#49) I need the extraction of the workflow type (Galaxy, Nextflow, Snakemake..) under the attribute « type ».
This info can be found under the attribute « workflow_class » > « title »

[rioualen]

This PR looks good to me, maybe the final approval of @hmenager would be useful, provided he has some availability. As soon as it is merged, I'll work on a better RDF dump to incorporate into the SPARQL endpoint.