# DMTCP checkpoint and restart

DMTCP saves the state of a running Linux application and restores it in a
later Slurm job. This example uses a single-process CPU application on one
Komondor compute node. MPI and GPU workloads require separate compatibility
testing.

## Load the module

```bash
module avail dmtcp
module load dmtcp/4.2.0
```

Use the same DMTCP version and application environment for both jobs.
Keep the executable, input files, working directory and checkpoint files
accessible to both jobs. Use a private directory on shared storage.
`/scratch/<project_ID>` is temporary. For longer retention, use an
appropriate directory on `/project/<project_ID>`. See [Storage Overview](../storage/overview.md).

## Try checkpoint and restart

Create a fresh, private working directory on shared storage and save this
example as `counter.py`. It appends one number per second to `progress.log`.

```python
import time

for step in range(1, 91):
    with open("progress.log", "a") as log:
        print(step, file=log)
    time.sleep(1)
```

### Start the job

Save the following as `checkpoint.sbatch` in the working directory.
Replace `YOUR_PROJECT` with your Slurm account.

```bash
#!/bin/bash -l

# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-checkpoint
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:01:00
#SBATCH --output=checkpoint-%j.out

# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000

# Run the counter and save a checkpoint every 10 seconds
dmtcp_launch --new-coordinator --coord-port 0 --interval 10 \
    python3 counter.py
```

Submit from the directory containing `counter.py` and the batch script.
Slurm uses this as the job’s working directory.

```bash
sbatch checkpoint.sbatch
```

`--new-coordinator --coord-port 0` starts a separate DMTCP coordinator
with an automatically selected port. `ulimit -s 10000` limits the stack
size to avoid unnecessarily large thread stacks during checkpointing.

DMTCP saves a checkpoint about every 10 seconds. Slurm stops this job at
the one-minute time limit, before the counter reaches 90. Wait for
`TIMEOUT` in `sacct -j JOBID` and check that a `ckpt_*.dmtcp` file exists.

### Resume from the checkpoint

Save the following as `restart.sbatch` in the same directory and replace
`YOUR_PROJECT` with your Slurm account.

```bash
#!/bin/bash -l

# Slurm resources and output
#SBATCH --account=YOUR_PROJECT
#SBATCH --partition=cpu
#SBATCH --job-name=dmtcp-restart
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:02:00
#SBATCH --output=restart-%j.out

# Prepare the environment
set -e
module load dmtcp/4.2.0
ulimit -s 10000

# Restore the saved counter and let it finish
dmtcp_restart --new-coordinator --coord-port 0 \
    ckpt_*.dmtcp
```

After the first job has stopped, submit from the same working directory.

```bash
sbatch restart.sbatch
```

After the restart job completes, `tail progress.log` should end at 90.
Some numbers can appear twice because the program replays work done after
the last checkpoint. For your own application, account for this when
writing output and leave enough time for each checkpoint to finish.

## Request a signal before the time limit

For longer runs, add this directive to a Bash batch script to request
`USR1` about two minutes before the time limit. Slurm can send it up to
one minute earlier than requested.

```bash
#SBATCH --signal=B:USR1@120
```

`B:` sends the signal to the batch shell. Set a `trap` for `USR1`
that uses `dmtcp_command --checkpoint` with the job’s coordinator address
to request a final checkpoint. Run the application in the background and
use Bash’s `wait` so the trap can run while the application is active.
Allow enough lead time for the checkpoint to finish writing before stopping
or requeueing the job. Keep periodic checkpoints for unexpected stops.

Slurm requeue reruns the batch script. To resume application progress, the
script must explicitly restore a completed checkpoint.

See the [Slurm sbatch reference](https://slurm.schedmd.com/sbatch.html#OPT_signal)
for signal timing and the [NERSC DMTCP guide](https://docs.nersc.gov/development/checkpoint-restart/dmtcp/)
for an example that uses NERSC’s requeue helpers. The
[DMTCP Quick Start](https://dmtcp.github.io/quick-start.html) covers basic
checkpoint and restart commands.
