DMTCP checkpoint and restart

DMTCP saves the state of a running Linux application and restores it in a later Slurm job. This example uses a single-process CPU application on one Komondor compute node. MPI and GPU workloads require separate compatibility testing.

Load the module

module avail dmtcp
module load dmtcp/4.2.0

Use the same DMTCP version and application environment for both jobs. Keep the executable, input files, working directory and checkpoint files accessible to both jobs. Use a private directory on shared storage. /scratch/<project_ID> is temporary. For longer retention, use an appropriate directory on /project/<project_ID>. See Storage Overview.

Try checkpoint and restart

Create a fresh, private working directory on shared storage and save this example as counter.py. It appends one number per second to progress.log.

import time

for step in range(1, 91):
    with open("progress.log", "a") as log:
        print(step, file=log)
    time.sleep(1)

Start the job

Save the following as checkpoint.sbatch in the working directory. Replace YOUR_PROJECT with your Slurm account.

#!/bin/bash -l

# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-checkpoint
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:01:00
#SBATCH --output=checkpoint-%j.out

# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000

# Run the counter and save a checkpoint every 10 seconds
dmtcp_launch --new-coordinator --coord-port 0 --interval 10 \
    python3 counter.py

Submit from the directory containing counter.py and the batch script. Slurm uses this as the job’s working directory.

sbatch checkpoint.sbatch

--new-coordinator --coord-port 0 starts a separate DMTCP coordinator with an automatically selected port. ulimit -s 10000 limits the stack size to avoid unnecessarily large thread stacks during checkpointing.

DMTCP saves a checkpoint about every 10 seconds. Slurm stops this job at the one-minute time limit, before the counter reaches 90. Wait for TIMEOUT in sacct -j JOBID and check that a ckpt_*.dmtcp file exists.

Resume from the checkpoint

Save the following as restart.sbatch in the same directory and replace YOUR_PROJECT with your Slurm account.

#!/bin/bash -l

# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-restart
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:02:00
#SBATCH --output=restart-%j.out

# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000

# Restore the saved counter and let it finish
dmtcp_restart --new-coordinator --coord-port 0 \
    ckpt_*.dmtcp

After the first job has stopped, submit from the same working directory.

sbatch restart.sbatch

After the restart job completes, tail progress.log should end at 90. Some numbers can appear twice because the program replays work done after the last checkpoint. For your own application, account for this when writing output and leave enough time for each checkpoint to finish.

Request a signal before the time limit

For longer runs, add this directive to a Bash batch script to request USR1 about two minutes before the time limit. Slurm can send it up to one minute earlier than requested.

#SBATCH --signal=B:USR1@120

B: sends the signal to the batch shell. Set a trap for USR1 that uses dmtcp_command --checkpoint with the job’s coordinator address to request a final checkpoint. Run the application in the background and use Bash’s wait so the trap can run while the application is active. Allow enough lead time for the checkpoint to finish writing before stopping or requeueing the job. Keep periodic checkpoints for unexpected stops.

Slurm requeue reruns the batch script. To resume application progress, the script must explicitly restore a completed checkpoint.

See the Slurm sbatch reference for signal timing and the NERSC DMTCP guide for an example that uses NERSC’s requeue helpers. The DMTCP Quick Start covers basic checkpoint and restart commands.