Skip to main content

Checkpoints and continuation

A plasma stage of a real coater runs for hours to days. A checkpoint is the state of the run at one time step - particles, averages, charges, surface coverages and the voltage control - so the run can be stopped and continued later, or continued after a change downstream. Both routes write the same kind of files; they differ in who triggers the write and who reads it back.

What is written​

filecontentnotes
checkpoint/particles_<species>.bincoordinates, velocities and weights of the particles of that speciesalways
checkpoint/averages_<species>.bintime averaged density, velocity and squared velocity per cellonly with TAVG
checkpoint/charges.bincharge per grid nodeonly for WORKMODE = 1
checkpoint/material_<material>.bincoverages and deposition rates per surface cellparameter file route only
last_shutdown.defiteration step, time step, plot interval and electrode voltagealways

The binary layout of the files is described under checkpoints in postprocessing and checkpoint file format. The parameter file route writes last_shutdown.def into the case folder, the Python route into <case>/checkpoint/ - see the note at the end of this page.

How often: NDUMPS​

NDUMPS is the number of checkpoints distributed evenly over the run, so one is written every TSIM/DT/NDUMPS steps. The default 1 means the checkpoint is written when the run reaches its end. A new checkpoint overwrites the previous one, there is no rotation - raise the value if an unexpected crash would cost days.

NDUMPS = 4; # four checkpoints over the run

In Python NDUMPS has to be set before sim.init(). A value of zero or below turns the dumps off, and sim.exit() does not write one either: such a run has no checkpoint at all.

Continuing a run​

The parameter file route continues on its own: if <case>/last_shutdown.def exists, the master reads the checkpoints and starts at the stored step instead of generating a new particle distribution. To stop a run cleanly, drop a file named shutdown into the case folder; the master finishes the remaining PAR.NTESTCYCLES (default 200) profiling cycles, writes the checkpoint and ends. A file named kill ends the run without a checkpoint.

touch case/shutdown # stop after the profiling cycles, write the checkpoint
submit -picmc 8 case.par # continue - the same case, nothing to change in the parameter file

The Python route decides this in the script:

sim.init(reload_if_possible = True) # continue at the stored step if a state is present

With reload_if_possible = True and no checkpoint the init prints a note and starts a fresh distribution; with False an existing checkpoint is ignored. Python has no shutdown file - the run ends at the last step or when the job is stopped, so stopping between two checkpoints loses everything since the last one.

parameter filePython
stop into a checkpointtouch <case>/shutdown, the run ends by itselfstop at a checkpoint step, nothing else
continuesubmit the same case againsim.init(reload_if_possible = True)
write a checkpoint by handS.write_checkpoint(...)not offered
coverages and deposition statewritten and read backnot written

Both routes rescale the step counter with the new time step: the stored step is multiplied by DT_OLD/DT, so DT may change between two runs of the same case. The geometry may not - a checkpoint carries the grid of the run that wrote it, see troubleshooting.

Writing a checkpoint by hand​

The parameter file can place a checkpoint wherever it makes sense, for instance in a user block after the plasma has settled or once a membrane flux has been sampled:

S.write_checkpoint("case/checkpoint/particles_Ar.bin", "particles", "Ar");
S.write_checkpoint("case/checkpoint/averages_Ar.bin", "averages", "Ar");
S.write_checkpoint("case/checkpoint/charges.bin", "charges", 0);
S.write_checkpoint("case/checkpoint/material_TiO2.bin", "material", "TiO2");

The third argument is the species, the material name, or a placeholder for charges. Reading works the same way with S.read_checkpoint(...); as type, material and coverages are both accepted. The combined DSMC and PIC-MC example uses this to start the plasma from the particle state of a preceding DSMC run.

Do not mix the two routes in one case folder

The parameter file route looks for <case>/last_shutdown.def, the Python route for <case>/checkpoint/last_shutdown.def, and the Python route writes no coverage state. A case written by one route is therefore not continued by the other - use separate folders when both routes are in play. Checkpoints from release 2.5 and older cannot be continued at all, the particle record changed when the per-particle weights were introduced.