feat: parallelize viscosity calculation by ltalirz · Pull Request #388 · glasagent/amorphouspy

ltalirz · 2026-06-12T11:08:55Z

Run 3 different temperatures in parallel.

codecov · 2026-06-12T11:10:46Z

Codecov Report

❌ Patch coverage is 67.89474% with 61 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
amorphouspy/src/amorphouspy/pipelines/viscosity.py	9.09%	30 Missing ⚠️
...morphouspy_api/src/amorphouspy_api/routers/jobs.py	23.52%	13 Missing ⚠️
...py_api/src/amorphouspy_api/routers/jobs_helpers.py	65.00%	7 Missing ⚠️
amorphouspy_api/src/amorphouspy_api/pipeline.py	87.23%	6 Missing ⚠️
amorphouspy/src/amorphouspy/lammps/runner.py	44.44%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

ltalirz · 2026-06-12T14:01:18Z

@Atilaac one question: in the current "master" workflow, we run the viscosity after the melt quench.
I guess that doesn't really make sense and we should start them in parallel to the melt quench that goes to room temperature (from the same random input structure), right?

Atilaac · 2026-06-12T14:45:48Z

Since we do liquid viscosity, you can run them in parallel. Just start from the same random structure and cool it using the same cooling rate to the temperature of interest, so you can claim that this viscosity is from the same trajectory as the glass.
You only need to wait for the glass to be made for the properties that require a glass structure, such as CTE and Elastic moduli

Squashed: parallelize viscosity, import cleanup, workflow special-case, test fixes.

Viscosity tasks now start from the freshly generated random structure and depend only on structure_generation, so they fan out in parallel with the main melt-quench instead of waiting for it. Each task still performs its own melt-quench cooling to its target temperature.

ltalirz · 2026-07-02T08:26:02Z

@Atilaac / @Gitdowski I think I'm making a mistake in the LAMMPS settings in the viscosity run here - can you spot it?

"Command 'mpiexec -n 3 --oversubscribe lmp_mpi -in lmp.in' returned non-zero exit status 1.
LAMMPS stdout:
t with -dE/dr.
WARNING:  Should only be flagged at inflection points (src/pair_table.cpp:466)
WARNING: New thermo_style command, previous thermo_modify settings will be lost (src/output.cpp:904)\n\nCITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE\n\nYour simulation uses code contributions which should be cited:\n- Type Label Framework: https://doi.org/10.1021/acs.jpcb.3c08419\nThe log file lists these citations in BibTeX format.\n\nCITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE\n
Generated 0 of 10 mixed pair_coeff terms from geometric mixing rule
WARNING: One or more atoms are time integrated more than once (src/modify.cpp:296)
...

ERROR: Lost atoms: original 6000 current 5142 (src/thermo.cpp:494)
Last command: run 10000

ltalirz · 2026-07-02T08:27:35Z

The relevant code should be in amorphouspy/src/amorphouspy/pipelines/viscosity.py

Atilaac · 2026-07-02T08:50:20Z

I will have a look

Atilaac · 2026-07-02T11:00:26Z

I think I managed to find the problem and a fix. Now, should I push it here or in a separate PR?

ltalirz · 2026-07-02T11:01:30Z

Thanks a lot - feel free to push directly here

ltalirz · 2026-07-03T08:00:43Z

Just checking whether you forgot to push or whether you found something else to check

Previously a failed pipeline blanket-marked every step 'failed' and stored the error under a generic 'pipeline' key. Now the failed branch probes the per-step executorlib caches to attribute the failure to the step that actually raised, falling back to the old behaviour when it cannot be localised.

Atilaac · 2026-07-03T08:12:01Z

I did not forget to push, I'm still working on it.

fix: stop dangling SHIK melt block from corrupting downstream MD stages _viscosity_simulation, elastic_simulation, and md_simulation each stripped the melt pre-equilibration block using a substring pattern from the original block ("fix langevin ... 5000 5000"). The fixes were renamed before to langevinnve/ensemblenve and changed the temperature to 4000 K, but these three strip sites were never updated, so a dangling Langevin thermostat + nve/limit integrator survived into every NPT/NVT stage, fighting the real ensemble and exploding the simulation. Consolidate the melt block into a single source of truth (lammps/potentials/_melt_block.py: melt_block_lines, strip_melt_block, set_melt_block_temperature), used by all six potential generators, the six melt-quench protocols, and the three downstream MD entry points. Also: - melt_quench_simulation now retunes the block to the caller's temperature_high instead of the generator's hardcoded default. - SHIK viscosity equilibration runs at 0.1 GPa (matching the melt-quench protocol convention) instead of 0 GPa. - melt_quench_simulation raises a clear ValueError when the resolved temperature_high equals temperature_low, instead of silently sending 0 heating/cooling steps to LAMMPS (which failed deep inside with an unrelated "Invalid dump frequency 0" error). Fixes the "sample explodes" report for SHIK viscosity runs.

Atilaac · 2026-07-03T11:53:23Z

Can you check now if it is working on your sample? I tried many glasses with Li2O, mainly Li2O:20, SiO2:80 (the one you suggested), and Li2O:15, Na2O:18, SiO2:67. Both the viscosity and elastic moduli workflows are working locally for me.

ltalirz · 2026-07-03T13:14:38Z

Thanks a lot @Atilaac , will do!
I'll also have a close look at your commit to understand what was the problem here

Atilaac · 2026-07-03T13:21:44Z

Check the commit message; I have a description there. Some other changes are more about reducing code duplication and have nothing to do with the problem directly.

ltalirz · 2026-07-03T13:47:04Z

btw, I notice you included a pixi.lock update in your commit. I will revert this since it led to problems on my side. Feel free to open a separate PR for this.

Caused problems locally

On a partial rerun, executorlib reads a cached parent step's SLURM queue_id back out of its _o.h5 and injects it as an 'afterok' dependency for the resubmitted child. That id is long purged from the scheduler, so sbatch rejects the submission with 'Job dependency problem' (or emits afterok:None,None,None when no live id exists). _clear_executor_cache(failed_only=True) now strips the queue_id from every successful cache file it keeps, so executorlib omits the dependency entirely (the parent result is already on disk). Adds test_rerun_cache.py.

ltalirz · 2026-07-06T07:49:30Z

Thanks @Atilaac ! Indeed, we no longer appear to run into missing atoms here.

I have two follow-up questions

The bug was caused by stripping out text in the input file, which is inherently a brittle operation. Is this really necessary or can we create it in an additive way so we don't need to strip out stuff? I would strongly prefer that.
I also computed the CTE in my test and it was completely off - perhaps there is still something to fix there as well?

Atilaac · 2026-07-06T07:58:55Z

Hi, the bug was caused by not using nve/limit at the beginning of the melt run for SHIK potential. So, when using it to prepare the melt or with another simulation that takes a pre-equilibrated structure as input by pre-equilibrated, I mean any non-random structure, this nve/limit + langevin should not be used. Thus, stripping those lines is necessary for the correctness of the simulations. I can not think of an alternative for now, but maybe in the future we might find a better way.

Regarding the CTE, I can not comment on it as I do not know what you mean by off: is it off compared to a previous simulation of the same composition using the same model, or off for a random composition with respect to experiments?

Gitdowski · 2026-07-06T09:18:19Z

The CTE in this example simulation was much too high (roughly 3 orders of magnitude). If there was a problem with SHIK when using pre-equilibrated structures this might already fix the issue (?). However, then I wonder why the elastic constants did not struggle with it in the same way...

Add a shared simulation_working_directory context manager in runner.py and use it in all three _run_lammps_md variants (runner, meltquench, cte). When tmp_working_directory is None the run directory is created in the OS temp location and auto-removed on exit (unchanged default). When a directory is provided, a uniquely-named sub-directory is created inside it and left in place afterwards: the caller now owns it and is responsible for cleanup, so run artefacts such as log.lammps remain available for inspection.

ltalirz · 2026-07-06T10:54:06Z

re cte: I simply noticed that Achraf's commit 66edc81#diff-382680fd60dc33c7501699a76ead0bcafc3ac1f005f55ae52211192445a9c131 contained changes to elastic (using your strip_melt_block function), but not to CTE

Atilaac · 2026-07-06T11:35:25Z

you need to strip those line only when the melt keyword is set to True in the potential generation. This keyword must always be set to True for any config starting from a random structure. if the CTE starts from a pre equilibrated sample you can keep that keyword as False and it should work normally. It all depends on what you are doing.

Gitdowski · 2026-07-06T12:11:13Z

I see... maybe "hiding" this in the potential is not the best way then. How about adding this explicitly to the meltquench_protocol for the potentials that require this? Or do we expect that somebody will start with a random structure for any other analysis/workflow?

Atilaac · 2026-07-06T12:14:53Z

I prefer to keep it in the potential section because it can be useful if someone decides to develop their own melt-quench protocol rather than the one imposed by the potential paper/amorphouspy.

Gitdowski · 2026-07-06T12:23:38Z

I am not fully convinced yet, but let's follow your approach. That means that most of the simulation/workflow need the strip_melt_block() at the beginning. That's an easy fix.

Atilaac · 2026-07-06T12:28:08Z

They need that only if the melt keyword is set to True; if set to False, you do not need to strip anything because there is nothing to strip. So now, if we wanna add it to everything, it will serve as a safeguard.

FYI: here, by default the melt is set to False

generate_potential(atoms_dict: dict, potential_type: str = 'pmmcs', *, melt: bool = False, electrostatics: InteractionConfig | None = None)

Gitdowski · 2026-07-06T12:49:04Z

Yes, for the notebook examples where we provide starting structures this is not an issue because of melt=False as default. But we need such a safeguard for the full workflow, where the starting point is always a random structure (melt=True for the respective potentials), because the potential is handed to all sub-workflows 👍

Atilaac · 2026-07-06T12:57:26Z

Yes, that right! thanks for fixing it!

Strong-scaling benchmarks (soda-lime silicate, 3k/10k atoms) show per-step efficiency holds >=~90% down to ~500 atoms/core before falling off sharply, and the knee is essentially potential-independent. Lower the per-potential MIN_ATOMS_PER_CORE and the DEFAULT_MIN_ATOMS_PER_CORE fallback from 2000 to 500 so LAMMPS runs use more cores while staying in the efficient regime. --- Here are results for a 3k-atom Na-Ca-Si-O system on a 128-core Zen3 AMD CPU using simply the LAMMPS binary from conda-forgea On this CPU, we are still at >85% computational efficiency on 8 cores here (375 atoms) and <80% on 12 cores (250 atoms). I ran the same benchmark with a 10k-atom structure with similar results, here we got >82% computational efficiency for 417 atoms. This tells me that on this CPU, 500 atoms is even on the conservative side - anyhow, let's go with 500. 3001 atoms CaNaOSi (source: benchmark_cores_results.json) per-step integration time [ms] (LAMMPS Loop time / n_steps) potential dim 1 2 3 4 6 8 12 16 24 -------------------------------------------------------------------------------------------------------- pmmcs 4t/4p 9.58 4.86 3.30 2.53 1.74 1.37 1.11 0.95 0.70 bmp-harmonic 4t/6p 11.88 6.03 4.07 3.13 2.15 1.65 1.35 1.15 0.85 bmp-screened-harmonic 4t/6p 12.42 6.31 4.28 3.29 2.26 1.74 1.40 1.18 0.85 shik 4t/9p 19.93 10.40 7.06 5.30 3.69 2.81 2.33 1.83 1.34 du_teter 4t/4p 19.66 10.36 7.02 5.29 3.64 2.79 2.33 1.84 1.36 yang2026 4t/4p 30.65 15.95 10.87 8.19 5.55 4.23 3.50 2.74 1.96 throughput [ns/day] (LAMMPS Performance line) potential dim 1 2 3 4 6 8 12 16 24 -------------------------------------------------------------------------------------------------------- pmmcs 4t/4p 9.02 17.76 26.15 34.12 49.53 63.01 78.14 91.41 123.88 bmp-harmonic 4t/6p 7.28 14.34 21.24 27.60 40.19 52.37 63.98 75.29 101.58 bmp-screened-harmonic 4t/6p 6.96 13.69 20.18 26.30 38.24 49.51 61.50 73.13 101.39 shik 4t/9p 4.33 8.31 12.24 16.30 23.41 30.70 37.09 47.10 64.48 du_teter 4t/4p 4.39 8.34 12.31 16.32 23.76 31.00 37.05 47.05 63.75 yang2026 4t/4p 2.82 5.42 7.95 10.55 15.56 20.41 24.70 31.57 44.08 parallel speedup and efficiency (baseline = each potential's 1-core run) pmmcs (4 types, 4 pair-terms) cores ms/step ns/day speedup eff% reps 1 9.58 9.02 1.00x 100.0 1 2 4.86 17.76 1.97x 98.4 4 3 3.30 26.15 2.90x 96.6 6 4 2.53 34.12 3.78x 94.6 6 6 1.74 49.53 5.49x 91.5 6 8 1.37 63.01 6.98x 87.3 6 12 1.11 78.14 8.66x 72.2 6 16 0.95 91.41 10.13x 63.3 6 24 0.70 123.88 13.73x 57.2 6 <- noisy bmp-harmonic (4 types, 6 pair-terms) cores ms/step ns/day speedup eff% reps 1 11.88 7.28 1.00x 100.0 1 2 6.03 14.34 1.97x 98.6 3 3 4.07 21.24 2.92x 97.3 5 4 3.13 27.60 3.79x 94.8 6 6 2.15 40.19 5.52x 92.1 6 8 1.65 52.37 7.20x 90.0 6 12 1.35 63.98 8.79x 73.3 6 16 1.15 75.29 10.35x 64.7 6 24 0.85 101.58 13.96x 58.2 6 <- noisy bmp-screened-harmonic (4 types, 6 pair-terms) cores ms/step ns/day speedup eff% reps 1 12.42 6.96 1.00x 100.0 1 2 6.31 13.69 1.97x 98.4 3 3 4.28 20.18 2.90x 96.7 5 4 3.29 26.30 3.78x 94.5 6 6 2.26 38.24 5.50x 91.6 6 8 1.74 49.51 7.12x 89.0 6 12 1.40 61.50 8.84x 73.7 6 16 1.18 73.13 10.51x 65.7 6 24 0.85 101.39 14.57x 60.7 6 <- noisy shik (4 types, 9 pair-terms) cores ms/step ns/day speedup eff% reps 1 19.93 4.33 1.00x 100.0 1 2 10.40 8.31 1.92x 95.8 2 3 7.06 12.24 2.82x 94.1 3 4 5.30 16.30 3.76x 94.0 4 6 3.69 23.41 5.40x 90.0 5 8 2.81 30.70 7.08x 88.5 6 12 2.33 37.09 8.55x 71.3 6 16 1.83 47.10 10.87x 67.9 6 24 1.34 64.48 14.87x 62.0 6 <- noisy du_teter (4 types, 4 pair-terms) cores ms/step ns/day speedup eff% reps 1 19.66 4.39 1.00x 100.0 1 2 10.36 8.34 1.90x 94.9 2 3 7.02 12.31 2.80x 93.4 3 4 5.29 16.32 3.71x 92.9 4 6 3.64 23.76 5.41x 90.1 5 8 2.79 31.00 7.06x 88.2 6 12 2.33 37.05 8.43x 70.3 6 16 1.84 47.05 10.71x 66.9 6 24 1.36 63.75 14.51x 60.4 6 <- noisy yang2026 (4 types, 4 pair-terms) cores ms/step ns/day speedup eff% reps 1 30.65 2.82 1.00x 100.0 1 2 15.95 5.42 1.92x 96.1 2 3 10.87 7.95 2.82x 94.0 2 4 8.19 10.55 3.74x 93.6 2 6 5.55 15.56 5.52x 92.0 4 8 4.23 20.41 7.24x 90.5 5 12 3.50 24.70 8.76x 73.0 6 <- noisy 16 2.74 31.57 11.20x 70.0 6 24 1.96 44.08 15.64x 65.2 6 <- noisy

github-actions Bot added the type: feature Changelog: new feature or performance improvement → bumps minor version label Jun 12, 2026

ltalirz force-pushed the feat/parallelize-visc-2 branch from 872e464 to a72d00f Compare June 12, 2026 11:15

ltalirz marked this pull request as draft June 12, 2026 12:10

ltalirz mentioned this pull request Jun 12, 2026

feat: parallelize viscosity calculation #174

Closed

ltalirz added the integration Tag a PR with this to run integration tests label Jun 12, 2026

feat: parallelize viscosity calculation

aca1283

Squashed: parallelize viscosity, import cleanup, workflow special-case, test fixes.

ltalirz force-pushed the feat/parallelize-visc-2 branch from bf2e6bf to aca1283 Compare July 1, 2026 09:18

ltalirz force-pushed the feat/parallelize-visc-2 branch from a895385 to 6657c2f Compare July 1, 2026 09:30

ltalirz added 4 commits July 1, 2026 11:45

chore: drop unused function

3dc5c38

chore: simplify further

28e4992

fix: respect potential-dependent melt temperature

96a0f94

chore: introduce step registry

ca5e6ee

ltalirz marked this pull request as ready for review July 1, 2026 10:43

ltalirz added 2 commits July 3, 2026 15:47

chore: revert change to pixi.lock

1525406

Caused problems locally

add potential langevin-nve-safeguard to cte

51166a0

fix tests

207dd40

ltalirz linked an issue Jul 6, 2026 that may be closed by this pull request

Parallelization strategy for LAMMPS with classical potentials #180

Open

Uh oh!

Conversation

ltalirz commented Jun 12, 2026

Uh oh!

codecov Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ltalirz commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Atilaac commented Jun 12, 2026

Uh oh!

ltalirz commented Jul 2, 2026

Uh oh!

ltalirz commented Jul 2, 2026

Uh oh!

Atilaac commented Jul 2, 2026

Uh oh!

Atilaac commented Jul 2, 2026

Uh oh!

ltalirz commented Jul 2, 2026

Uh oh!

ltalirz commented Jul 3, 2026

Uh oh!

Atilaac commented Jul 3, 2026

Uh oh!

Atilaac commented Jul 3, 2026

Uh oh!

ltalirz commented Jul 3, 2026

Uh oh!

Atilaac commented Jul 3, 2026

Uh oh!

ltalirz commented Jul 3, 2026

Uh oh!

ltalirz commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Atilaac commented Jul 6, 2026

Uh oh!

Gitdowski commented Jul 6, 2026

Uh oh!

ltalirz commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Atilaac commented Jul 6, 2026

Uh oh!

Gitdowski commented Jul 6, 2026

Uh oh!

Atilaac commented Jul 6, 2026

Uh oh!

Gitdowski commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Atilaac commented Jul 6, 2026

Uh oh!

Gitdowski commented Jul 6, 2026

Uh oh!

Atilaac commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jun 12, 2026 •

edited

Loading

ltalirz commented Jun 12, 2026 •

edited

Loading

ltalirz commented Jul 6, 2026 •

edited

Loading

ltalirz commented Jul 6, 2026 •

edited

Loading

Gitdowski commented Jul 6, 2026 •

edited

Loading