Skip to content

HPC executor reports success when job fails #8

Description

@iparask

I run a HPC executor example and it reported successful while the actual batch job failed.

(colonies) ~/tutorials/colonyos_eval/03_mpi$ python submit_mpi.py
--- case e: 2 nodes x 4 ranks ---
kwargs: {"docker-image": "local/mpihello:1.0", "cmd": "/bin/sh", "args": ["-c", "echo a && echo b > /tmp/probe.txt"]}
env:    {}
processid: 20690d201057ddb4777402613db6a4d319e5d20c9ad35aeaed2129dd90577691

state:  SUCCESSFUL
output: []
errors: []
(colonies) ~/tutorials/colonyos_eval/03_mpi$   colonies log get --processid 20690d201057ddb4777402613db6a4d319e5d20c9ad35aeaed2129dd90577691
DEBU[0000] Starting a Colonies HTTP client               Insecure=true ServerHost=localhost ServerPort=50080
ColonyOS: Ignoring downloading snapshots and syncing dirs as mount not definied
ColonyOS: Creating singularity container: local/mpihello:1.0 to /scratch/gpfs/SIMONSOBS/users/ip8725/colonyos/images
ColonyOS: Image already exists: local/mpihello:1.0
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
/usr/bin/sh: 1: Syntax error: "&" unexpected
srun: error: tiger-g04c1n1: tasks 0-3: Exited with exit code 2
srun: Terminating StepId=3716659.0
srun: error: tiger-g04c1n3: tasks 4-7: Exited with exit code 2
ColonyOS: Ignoring cleaning up snapshots as mount not definied
ColonyOS: Ignoring syncing dirs as mount not definied

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions