Skip to main content

Submitting a run

A production run goes through the queue, and two scripts write the batch file for it: submit, the shell script of the parameter-file route, and py/pysubmit.py, the Python script introduced with release 26. Both create <case>.slurm next to the case, hand it to sbatch, write stdout to <case>-out.txt and stderr to <case>-err.txt, and append the error file to the output file when the job ends. For an interactive run without the queue see Driving a case from Python and the invocation tables of the manual.

The parameter-file route: submit​

submit sits in $PICMCPATH/sh and takes the parameter file; the .par suffix may be left off.

submit -picmc 8 -time 48:00:00 -seed 4711 case.par
optionmeaning
-picmc <n>number of picmc workers
-bem <n>number of bem workers, for a magnetic field computation
-time <hh:mm:ss>wall time, an integer is read as hours
-mem <MB>memory per core
-partition <name>partition, default epyc
-N <n>number of nodes
-exclusiveexclusive node access
-seed <n>fixed random seed, exported as PICMC_SEED
-collisionModel legacy|SK07collision model
-after <jobid>start after that job finished (--dependency=afterok)
-valgrindrun under valgrind
-helpprint the usage

The queue defaults - partition, memory, wall time, MPI path - are constants at the top of the script and have to be edited there for a new cluster. submit reads PICMCPATH from the calling shell, so the exports of the installation have to be in place.

The Python route: py/pysubmit.py​

py/pysubmit.py submits a Python case and takes the queue defaults from py/<cluster>.json instead of from the script:

python3 $PICMCPATH/py/pysubmit.py case.py -n 8 -c IST_EPYC -t 48:00:00 -s 4711
optionmeaning
-n <n>number of workers; the job gets one task more for the Python rank
-c <cluster>configuration file py/<cluster>.json, a copy in the working directory wins
-brun the bem binary instead of picmc, mandatory for a BEM script
-t <hh:mm:ss>wall time, default 336:00:00
-s <n>random seed
-m <MB>memory per core
-p <name>partition, has to be listed in allowed_partitions
-N <n>number of nodes
-l <list>nodelist
-eexclusive node access
-a <jobid>start after that job finished (--dependency=afterok)

Name the configuration with -c: without it the helper falls back to its built-in default IST_INTEL, which on another cluster carries the wrong partition, memory size and MPI path. What the file holds and how the two processes of a case cooperate is described in Driving a case from Python.

Chaining runs​

Both scripts pass a dependency to SLURM, so a chain of runs is queued in one go - the second job waits until the first one has finished successfully:

submit -picmc 8 plasma.par
python3 $PICMCPATH/py/pysubmit.py transport.py -n 8 -c IST_EPYC -a <job id printed by the first submit>

A chain that has to survive a wall time limit is cut at a checkpoint instead: the second job continues the first one from its state, described in Checkpoints and continuation.