Checkpoints and continuation
A plasma stage of a real coater runs for hours to days. A checkpoint is the state of the run at one time step - particles, averages, charges, surface coverages and the voltage control - so the run can be stopped and continued later, or continued after a change downstream. Both routes write the same kind of files; they differ in who triggers the write and who reads it back.
What is written
| file | content | notes |
|---|---|---|
checkpoint/particles_<species>.bin | coordinates, velocities and weights of the particles of that species | always |
checkpoint/averages_<species>.bin | time averaged density, velocity and squared velocity per cell | only with TAVG |
checkpoint/charges.bin | charge per grid node | only for WORKMODE = 1 |
checkpoint/material_<material>.bin | coverages and deposition rates per surface cell | parameter file route only |
last_shutdown.def | iteration step, time step, plot interval and electrode voltage | always |
The binary layout of the files is described under
checkpoints in postprocessing and
checkpoint file format. The parameter file route writes
last_shutdown.def into the case folder, the Python route into <case>/checkpoint/ - see the
note at the end of this page.
How often: NDUMPS
NDUMPS is the number of checkpoints distributed evenly over the run, so one is written every
TSIM/DT/NDUMPS steps. The default 1 means the checkpoint is written when the run reaches its
end. A new checkpoint overwrites the previous one, there is no rotation - raise the value if an
unexpected crash would cost days.
- RVM
- Python
NDUMPS = 4; # four checkpoints over the run
sim = PICMC.Sim(
...
NDUMPS = 4 # four checkpoints over the run
)
In Python NDUMPS has to be set before sim.init(). A value of zero or below turns the dumps
off, and sim.exit() does not write one either: such a run has no checkpoint at all.
Continuing a run
The parameter file route continues on its own: if <case>/last_shutdown.def exists, the master
reads the checkpoints and starts at the stored step instead of generating a new particle
distribution. To stop a run cleanly, drop a file named shutdown into the case folder; the
master finishes the remaining PAR.NTESTCYCLES (default 200) profiling cycles, writes the
checkpoint and ends. A file named kill ends the run without a checkpoint.
touch case/shutdown # stop after the profiling cycles, write the checkpoint
submit -picmc 8 case.par # continue - the same case, nothing to change in the parameter file
The Python route decides this in the script:
sim.init(reload_if_possible = True) # continue at the stored step if a state is present
With reload_if_possible = True and no checkpoint the init prints a note and starts a fresh
distribution; with False an existing checkpoint is ignored. Python has no shutdown file - the
run ends at the last step or when the job is stopped, so stopping between two checkpoints loses
everything since the last one.
| parameter file | Python | |
|---|---|---|
| stop into a checkpoint | touch <case>/shutdown, the run ends by itself | stop at a checkpoint step, nothing else |
| continue | submit the same case again | sim.init(reload_if_possible = True) |
| write a checkpoint by hand | S.write_checkpoint(...) | not offered |
| coverages and deposition state | written and read back | not written |
Both routes rescale the step counter with the new time step: the stored step is multiplied by
DT_OLD/DT, so DT may change between two runs of the same case. The geometry may not - a
checkpoint carries the grid of the run that wrote it, see
troubleshooting.
Writing a checkpoint by hand
The parameter file can place a checkpoint wherever it makes sense, for instance in a user block after the plasma has settled or once a membrane flux has been sampled:
- RVM
- Python
S.write_checkpoint("case/checkpoint/particles_Ar.bin", "particles", "Ar");
S.write_checkpoint("case/checkpoint/averages_Ar.bin", "averages", "Ar");
S.write_checkpoint("case/checkpoint/charges.bin", "charges", 0);
S.write_checkpoint("case/checkpoint/material_TiO2.bin", "material", "TiO2");
# There is no callable form. The run writes its checkpoint at the steps given by NDUMPS,
# so a state at a chosen step is taken by setting NDUMPS and stopping the job there.
The third argument is the species, the material name, or a placeholder for charges. Reading
works the same way with S.read_checkpoint(...); as type, material and coverages are both
accepted. The combined DSMC and PIC-MC example uses
this to start the plasma from the particle state of a preceding DSMC run.
The parameter file route looks for <case>/last_shutdown.def, the Python route for
<case>/checkpoint/last_shutdown.def, and the Python route writes no coverage state. A case
written by one route is therefore not continued by the other - use separate folders when both
routes are in play. Checkpoints from release 2.5 and older cannot be continued at all, the
particle record changed when the per-particle weights were introduced.