Submitting a run
A production run goes through the queue, and two scripts write the batch file for it: submit,
the shell script of the parameter-file route, and py/pysubmit.py, the Python script introduced
with release 26. Both create <case>.slurm next to the case, hand it to sbatch, write stdout
to <case>-out.txt and stderr to <case>-err.txt, and append the error file to the output file
when the job ends. For an interactive run without the queue see
Driving a case from Python and the invocation tables of the
manual.
The parameter-file route: submit
submit sits in $PICMCPATH/sh and takes the parameter file; the .par suffix may be left
off.
submit -picmc 8 -time 48:00:00 -seed 4711 case.par
| option | meaning |
|---|---|
-picmc <n> | number of picmc workers |
-bem <n> | number of bem workers, for a magnetic field computation |
-time <hh:mm:ss> | wall time, an integer is read as hours |
-mem <MB> | memory per core |
-partition <name> | partition, default epyc |
-N <n> | number of nodes |
-exclusive | exclusive node access |
-seed <n> | fixed random seed, exported as PICMC_SEED |
-collisionModel legacy|SK07 | collision model |
-after <jobid> | start after that job finished (--dependency=afterok) |
-valgrind | run under valgrind |
-help | print the usage |
The queue defaults - partition, memory, wall time, MPI path - are constants at the top of the
script and have to be edited there for a new cluster. submit reads PICMCPATH from the
calling shell, so the exports of the installation have
to be in place.
The Python route: py/pysubmit.py
py/pysubmit.py submits a Python case and takes the queue defaults from py/<cluster>.json
instead of from the script:
python3 $PICMCPATH/py/pysubmit.py case.py -n 8 -c IST_EPYC -t 48:00:00 -s 4711
| option | meaning |
|---|---|
-n <n> | number of workers; the job gets one task more for the Python rank |
-c <cluster> | configuration file py/<cluster>.json, a copy in the working directory wins |
| -b | run the bem binary instead of picmc, mandatory for a BEM script |
-t <hh:mm:ss> | wall time, default 336:00:00 |
-s <n> | random seed |
-m <MB> | memory per core |
-p <name> | partition, has to be listed in allowed_partitions |
-N <n> | number of nodes |
-l <list> | nodelist |
-e | exclusive node access |
-a <jobid> | start after that job finished (--dependency=afterok) |
Name the configuration with -c: without it the helper falls back to its built-in default
IST_INTEL, which on another cluster carries the wrong partition, memory size and MPI path.
What the file holds and how the two processes of a case cooperate is described in
Driving a case from Python.
Chaining runs
Both scripts pass a dependency to SLURM, so a chain of runs is queued in one go - the second job waits until the first one has finished successfully:
submit -picmc 8 plasma.par
python3 $PICMCPATH/py/pysubmit.py transport.py -n 8 -c IST_EPYC -a <job id printed by the first submit>
A chain that has to survive a wall time limit is cut at a checkpoint instead: the second job continues the first one from its state, described in Checkpoints and continuation.