External module loading broken on Ursa

[AlexanderHrabski-NOAA]

Description

When trying to install a new environment, the intel-oneapi-compilers/2025.3.1 compiler module won't load, leading to a failure at install time. The following message is generated:

==> Installing mct-2.11.0-rm7thr67fctoy5dgqauvmegcmmf5czql [36/334]
==> Error: ModuleLoadError: Module 'intel-oneapi-compilers/2025.3.1' could not be loaded.

To Reproduce

On Ursa:

git clone https://github.com/JCSDA/spack-stack.git --recurse-submodules
cd spack-stack/
source setup.sh
spack stack create env --name=mct-test --site=ursa --compiler=oneapi-2025.3.1
cd envs/mct-test/
spacktivate -p .
spack add mct
spack concretize --force --fresh
spack install

Expected behavior

The environment should build.

System, compiler, code, ...

Ursa, with the intel-oneapi-compilers/2025.3.1 compilers. I have not tried it on other systems or with other compilers.

[rickgrubin-noaa]

Apologies for the length of this comment; there are a few things I hope to point out in addition to a suggested fix for this issue (and for future stack builds).

---

This may boil down to, I think, a difference in spack versions (different commits more specifically):

Cloning branch release/2.1:

$ spack --version
1.2.0.dev0 (324bf79a31704dbc68fafc9e4a053aa4c60019ea)

mct will successfully build per the set of steps noted in the description:

[...]
==> mct: Successfully installed mct-2.11.0-nnpypqkr4y2h2pu352yg2bexdfhzx727
  Stage: 0.88s.  Autoreconf: 0.00s.  Configure: 7.17s.  Build: 15.27s.  Install: 1.08s.  Post-install: 0.22s.  Total: 24.70s
[+] /path/to/spack-stack-release-2.1/envs/mct-test/install/intel-oneapi-compilers/2025.3.1/mct-2.11.0-nnpypqk
==> Installing mct-2.11.0-nnpypqkr4y2h2pu352yg2bexdfhzx727 [9/9]

---

Cloning branch develop (as the issue description does):

$ spack --version
1.2.0.dev0 (0a2c2f21d52d70692f52ed48c3ab4b5bbdcb2bde)

Capturing the output from spack concretize will yield something similar to the following :

$ spack concretize --force --fresh 2>&1 | tee log.concretize
==> Warning: /path/to/develop/spack-stack/envs/mct-test/site/packages_oneapi-2025.3.1.yaml:4: 'packages: all: require: ["%intel-oneapi-compilers@=2025.3.1"]' applies a dependency constraint to all packages. This often leads to concretization errors. This was likely intended as a requirement for a provider of a (language) virtual. Consider instead:
# For each language virtual (c, cxx, fortran, ...):
packages:
  c:
    require:
    - intel-oneapi-compilers@=2025.3.1
==> Warning: /path/to/develop/spack-stack/envs/mct-test/site/packages_oneapi-2025.3.1.yaml:6: 'packages: all: require: ["%gcc@=12.4.0"]' applies a dependency constraint to all packages. This often leads to concretization errors. This was likely intended as a requirement for a provider of a (language) virtual. Consider instead:
# For each language virtual (c, cxx, fortran, ...):
packages:
  c:
    require:
    - gcc@=12.4.0

which is informative in that for spack-stack@develop, the spack version (commit) is at a point in time where core languages are treated as virtual packages and can be satisfied for oneAPI envs (are there better [likely!] ways to specify?) by:

packages:
  c:
    prefer:
      - intel-oneapi-compilers@=2025.3.1
      - gcc@=12.4.0
  cxx:
    prefer:
      - intel-oneapi-compilers@=2025.3.1
      - gcc@=12.4.0
  fortran:
    prefer:
      - intel-oneapi-compilers@=2025.3.1
      - gcc@=12.4.0

With respect to an install error, I see something similar but different:

==> No binary for mct-2.11.0-rm7thr67fctoy5dgqauvmegcmmf5czql found: installing from source
==> Error: ModuleLoadError: Module 'intel-oneapi-mpi/2021.17.1' could not be loaded.
==> Error: Terminating after first install failure: ModuleLoadError: Module 'intel-oneapi-mpi/2021.17.1' could not be loaded.

despite seeing the following in captured output from spack install:

==> intel-oneapi-compilers@2025.3.1 : has external module in ['intel-oneapi-compilers/2025.3.1']
[+] /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2025.3.1-yqgrg5v3jonwexcjwd3xu3rzvwcp4ejk (external intel-oneapi-compilers-2025.3.1-2idvs7suqd2pwe5lihdgiwjoizpefdw7)
==> intel-oneapi-mpi@2021.17 : has external module in ['intel-oneapi-mpi/2021.17.1']
[+] /apps/spack-2024-12/linux-rocky9-x86_64/oneapi-2025.3.1/intel-oneapi-mpi-2021.17.1-emmz2zv6lx5ypzmlc3csouqffu7mwnwr (external intel-oneapi-mpi-2021.17-246mwvwyxjxlamh6vms4sbii6k7ct5iu)

and on ursa

$ module spider intel-oneapi-mpi/2021.17.1

--------------------------------------------------------------------------------------------------
  intel-oneapi-mpi: intel-oneapi-mpi/2021.17.1
--------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "intel-oneapi-mpi/2021.17.1" module is available to load.

      intel-oneapi-compilers/2025.3.1

Removing the modules block for each of the intel-oneapi-compilers and intel-oneapi-mpi blocks in envs/mct-test/site/packages_oneapi-2025.3.1.yaml and relying solely on the prefix pathing results in a successful build.

Perhaps this serves to point out that the path forward, so to speak, for future releases is to use prefix: instead of modules: and not rely on module load for concretization or building of packages.

[AlexanderHrabski-NOAA]

@rickgrubin-noaa Thanks for taking a detailed look at this. I think my error message in the description must have been from the terminal where I first caught this bug. When I run the MRE it indeed produces the exact error you received:

==> Installing mct-2.11.0-rm7thr67fctoy5dgqauvmegcmmf5czql [9/9]
==> Error: ModuleLoadError: Module 'intel-oneapi-mpi/2021.17.1' could not be loaded.

My mistake!

It sounds like there is a path forward using prefix:, and it's great to have a fix! Despite the updates to spack, the ability to load modules rather than specify a prefix manually is still a documented feature. Something clearly must have changed, but our usage of module: looks correct, and it certainly worked before, as you demonstrated.

[rickgrubin-noaa]

It sounds like there is a path forward using prefix:, and it's great to have a fix! Despite the updates to spack, the ability to load modules rather than specify a prefix manually is still a documented feature. Something clearly must have changed, but our usage of module: looks correct, and it certainly worked before, as you demonstrated.

intel-oneapi-compilers/2025.3.1 is a parent module for intel-oneapi-mpi/2021.17.1, so the following is suggested / should work:

  intel-oneapi-mpi:
    buildable: False
    externals:
    - spec: intel-oneapi-mpi@2021.17
      prefix: /apps/spack-2024-12/linux-rocky9-x86_64/oneapi-2025.3.1/intel-oneapi-mpi-2021.17.1-emmz2zv6lx5ypzmlc3csouqffu7mwnwr
      modules:
      - intel-oneapi-compilers/2025.3.1
      - intel-oneapi-mpi/2021.17.1

however that errors with

==> No binary for mct-2.11.0-2mgdcy6y4z7jyjlwnmv7ewym7a7ymnoj found: installing from source
==> Error: ModuleLoadError: Module 'intel-oneapi-compilers/2025.3.1' could not be loaded.
==> Error: Terminating after first install failure: ModuleLoadError: Module 'intel-oneapi-compilers/2025.3.1' could not be loaded.

This is puzzling; something seems to be inadvertently mucking up MODULEPATH.

Some experimentation:

• load intel-oneapi-compilers/2025.3.1 / concretize / install -- FAIL
• explicitly modify MODULEPATH in packages_oneapi-2025.3.1.yaml -- FAIL

      modules:
      - intel-oneapi-compilers/2025.3.1
      - intel-oneapi-mpi/2021.17.1
      extra_attributes:
        environment:
          prepend_path:
            MODULEPATH: /apps/spack-2024-12/modules/linux-rocky9-x86_64/oneapi/2025.3.1

So far, not specifying modules and relying on prefix: is the only idiom that's worked.

[rickgrubin-noaa]

@AlexanderHrabski-NOAA the differences in spack versions (different commits) are from JCSDA/spack, and

commit 0a2c2f21d52d70692f52ed48c3ab4b5bbdcb2bde is a very large update (your PR):

Update spack-stack-dev from spack develop 2026/05/20 (JCSDA/spack#577)
This PR updates spack-stack-dev with the latest developments from spack/spack:develop

A little more experimentation -- explicitly specifying the following in spack.yaml

  config:
    installer: old

results in the errors seen previously. Specifying

  config:
    installer: new

generates the same failure, but with a stack trace:

Traceback (most recent call last):
  File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/new_installer.py", line 758, in _install
    env_mods = spack.build_environment.setup_package(pkg, dirty=dirty)
  File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/build_environment.py", line 820, in setup_package
    load_external_modules(setup_context)
  File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/build_environment.py", line 1120, in load_external_modules
    spack.util.module_cmd.load_module(external_module)
  File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/util/module_cmd.py", line 118, in load_module
    raise ModuleLoadError(mod)
spack.util.module_cmd.ModuleLoadError: Module 'intel-oneapi-compilers/2025.3.1' could not be loaded.
[x] 2qg3m33 mct@2.11.0 failed: /scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/cache/build_stage/spack-stage-mct-2.11.0-2qg3m33r7v6p6pmctpj7bbwfnmjrwyy3-kiiw_cjt.log (4s)
-- lines 1 to 10 --
  Traceback (most recent call last):
    File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/new_installer.py", line 758, in _install
      env_mods = spack.build_environment.setup_package(pkg, dirty=dirty)
    File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/build_environment.py", line 820, in setup_package
      load_external_modules(setup_context)
    File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/build_environment.py", line 1120, in load_external_modules
      spack.util.module_cmd.load_module(external_module)
    File "/scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/spack/lib/spack/spack/util/module_cmd.py", line 118, in load_module
      raise ModuleLoadError(mod)
  spack.util.module_cmd.ModuleLoadError: Module 'intel-oneapi-compilers/2025.3.1' could not be loaded.
==> Error: The following packages failed to install:
mct@2.11.0/2qg3m33r7v6p6pmctpj7bbwfnmjrwyy3: /scratch4/NCEPDEV/nems/Richard.Grubin/tmp/develop/spack-stack/cache/build_stage/spack-stage-mct-2.11.0-2qg3m33r7v6p6pmctpj7bbwfnmjrwyy3-kiiw_cjt.log
==> Error: Expected InstallError to include the associated package

This seems like a regression from upstream.

[AlexanderHrabski-NOAA]

@rickgrubin-noaa Adding the module dependencies to a spack external's module list has fixed similar issues for me in the past! This must be a different issue. I agree that some change from the upstream included in my PR must be messing up the build time environment. Thanks again for the detailed look into this. In the next day or two I'll try to get an MRE assembled based only on the upstream spack:develop and I'll open an issue upstream if I'm successful. Otherwise, I'll try a git bisect. Either way, I'll report back here!

[AlexanderHrabski-NOAA]

@rickgrubin-noaa I was not able to create an MRE using the upstream. Including the following as ~/.spack/packages.yaml

packages:
  mpi:
    buildable: False
    require:
    - intel-oneapi-mpi@2021.17
  intel-oneapi-compilers:
    buildable: False
    externals:
    - spec: intel-oneapi-compilers@2025.3.1
      prefix: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2025.3.1-yqgrg5v3jonwexcjwd3xu3rzvwcp4ejk
      modules:
      - intel-oneapi-compilers/2025.3.1
      extra_attributes:
        compilers:
          c: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2025.3.1-yqgrg5v3jonwexcjwd3xu3rzvwcp4ejk/compiler/2025.3/bin/icx
          cxx: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2025.3.1-yqgrg5v3jonwexcjwd3xu3rzvwcp4ejk/compiler/2025.3/bin/icpx
          fortran: /apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2025.3.1-yqgrg5v3jonwexcjwd3xu3rzvwcp4ejk/compiler/2025.3/bin/ifx
        environment:
          prepend_path:
            LD_LIBRARY_PATH: '/usr/lib:/usr/lib64'
  intel-oneapi-mpi:
    buildable: False
    externals:
    - spec: intel-oneapi-mpi@2021.17
      prefix: /apps/spack-2024-12/linux-rocky9-x86_64/oneapi-2025.3.1/intel-oneapi-mpi-2021.17.1-emmz2zv6lx5ypzmlc3csouqffu7mwnwr
      modules:
      - intel-oneapi-mpi/2021.17.1

led to a concretization error when trying to build an environment with spack add mct %intel-oneapi-compilers@2025.3.1. Everything above looks good to me, so I was surprised this didn't work. The error was due to the compiler not being buildable, but also not being satisfied by an external package.

I also tried a git bisect, but I ran into issues with the spack stack command not being defined. I'll have to set this down for a little while!