DMTCP checkpoint and restart
DMTCP saves the state of a running Linux application and restores it in a later Slurm job. This example uses a single-process CPU application on one Komondor compute node. MPI and GPU workloads require separate compatibility testing.
Load the module
module avail dmtcp
module load dmtcp/4.2.0
Use the same DMTCP version and application environment for both jobs.
Keep the executable, input files, working directory and checkpoint files
accessible to both jobs. Use a private directory on shared storage.
/scratch/<project_ID> is temporary. For longer retention, use an
appropriate directory on /project/<project_ID>. See Storage Overview.
Try checkpoint and restart
Create a fresh, private working directory on shared storage and save this
example as counter.py. It appends one number per second to progress.log.
import time
for step in range(1, 91):
with open("progress.log", "a") as log:
print(step, file=log)
time.sleep(1)
Start the job
Save the following as checkpoint.sbatch in the working directory.
Replace YOUR_PROJECT with your Slurm account.
#!/bin/bash -l
# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-checkpoint
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:01:00
#SBATCH --output=checkpoint-%j.out
# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000
# Run the counter and save a checkpoint every 10 seconds
dmtcp_launch --new-coordinator --coord-port 0 --interval 10 \
python3 counter.py
Submit from the directory containing counter.py and the batch script.
Slurm uses this as the job’s working directory.
sbatch checkpoint.sbatch
--new-coordinator --coord-port 0 starts a separate DMTCP coordinator
with an automatically selected port. ulimit -s 10000 limits the stack
size to avoid unnecessarily large thread stacks during checkpointing.
DMTCP saves a checkpoint about every 10 seconds. Slurm stops this job at
the one-minute time limit, before the counter reaches 90. Wait for
TIMEOUT in sacct -j JOBID and check that a ckpt_*.dmtcp file exists.
Resume from the checkpoint
Save the following as restart.sbatch in the same directory and replace
YOUR_PROJECT with your Slurm account.
#!/bin/bash -l
# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-restart
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:02:00
#SBATCH --output=restart-%j.out
# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000
# Restore the saved counter and let it finish
dmtcp_restart --new-coordinator --coord-port 0 \
ckpt_*.dmtcp
After the first job has stopped, submit from the same working directory.
sbatch restart.sbatch
After the restart job completes, tail progress.log should end at 90.
Some numbers can appear twice because the program replays work done after
the last checkpoint. For your own application, account for this when
writing output and leave enough time for each checkpoint to finish.
Request a signal before the time limit
For longer runs, add this directive to a Bash batch script to request
USR1 about two minutes before the time limit. Slurm can send it up to
one minute earlier than requested.
#SBATCH --signal=B:USR1@120
B: sends the signal to the batch shell. Set a trap for USR1
that uses dmtcp_command --checkpoint with the job’s coordinator address
to request a final checkpoint. Run the application in the background and
use Bash’s wait so the trap can run while the application is active.
Allow enough lead time for the checkpoint to finish writing before stopping
or requeueing the job. Keep periodic checkpoints for unexpected stops.
Slurm requeue reruns the batch script. To resume application progress, the script must explicitly restore a completed checkpoint.
See the Slurm sbatch reference for signal timing and the NERSC DMTCP guide for an example that uses NERSC’s requeue helpers. The DMTCP Quick Start covers basic checkpoint and restart commands.