OpenFOAM

OpenFOAM is a Finite Volume Method solver framework to solve CFD problems. Two important forks of OpenFOAM are supported:

The before mentioned OpenFOAM versions are compiled for various types of number representations.

Module use of OpenFOAM

Various OpenFOAM versions can be accessed by using module system. To access OpenFOAM modules you need to use their folder (to not bother users not interested).

module use /opt/software/packages/openfoam/modules

It is useful to place it in your ~/.bashrc.

You can list available versions:

module avail OpenFOAM

Example output:

---------------------------------------------------------------------------------- /opt/software/packages/openfoam/modules ----------------------------------------------------------------------------------
   OpenFOAM/v2306-DP32-Gcc-CRAY-MPICH      OpenFOAM/v2406-DP64-Amd-CRAY-MPICH      OpenFOAM/v2412-DP64-Cray-CRAY-MPICH      OpenFOAM/v2506-DP64-Gcc-CRAY-MPICH    OpenFOAM/12-DP32-Gcc-CRAY-MPICH
   OpenFOAM/v2312-DP32-Gcc-CRAY-MPICH      OpenFOAM/v2406-DP64-Cray-CRAY-MPICH     OpenFOAM/v2412-DP64-Gcc-CRAY-MPICH       OpenFOAM/9-DP32-Gcc-CRAY-MPICH        OpenFOAM/12-DP64-Gcc-CRAY-MPICH
   OpenFOAM/v2312-DP64-Gcc-CRAY-MPICH      OpenFOAM/v2406-DP64-Gcc-CRAY-MPICH      OpenFOAM/v2412-SPDP32-Cray-CRAY-MPICH    OpenFOAM/10-DP32-Gcc-CRAY-MPICH       OpenFOAM/13-DP32-Clang-CRAY-MPICH
   OpenFOAM/v2312-SP32-Gcc-CRAY-MPICH      OpenFOAM/v2406-SPDP32-Amd-CRAY-MPICH    OpenFOAM/v2412-SPDP32-Gcc-CRAY-MPICH     OpenFOAM/11-DP32-Gcc-CRAY-MPICH       OpenFOAM/13-DP32-Gcc-CRAY-MPICH
   OpenFOAM/v2312-SP64-Gcc-CRAY-MPICH      OpenFOAM/v2406-SPDP32-Gcc-CRAY-MPICH    OpenFOAM/v2412-SPDP64-Cray-CRAY-MPICH    OpenFOAM/11-DP64-Gcc-CRAY-MPICH       OpenFOAM/13-DP64-Clang-CRAY-MPICH
   OpenFOAM/v2312-SPDP32-Gcc-CRAY-MPICH    OpenFOAM/v2406-SPDP64-Amd-CRAY-MPICH    OpenFOAM/v2412-SPDP64-Gcc-CRAY-MPICH     OpenFOAM/11-LP32-Gcc-CRAY-MPICH       OpenFOAM/13-DP64-Gcc-CRAY-MPICH   (D)
   OpenFOAM/v2312-SPDP64-Gcc-CRAY-MPICH    OpenFOAM/v2406-SPDP64-Gcc-CRAY-MPICH    OpenFOAM/v2506-DP32-Clang-CRAY-MPICH     OpenFOAM/11-LP64-Gcc-CRAY-MPICH
   OpenFOAM/v2406-DP32-Amd-CRAY-MPICH      OpenFOAM/v2412-DP32-Cray-CRAY-MPICH     OpenFOAM/v2506-DP32-Gcc-CRAY-MPICH       OpenFOAM/11-SP32-Gcc-CRAY-MPICH

The name contains following information:

OpenFOAM/AA-FFII-CC-MM

AA: means the version, eg. 13, or v2506
FF: means the floating point representation e.g. DP
II: means the integer representation e.g. 32
CC: means the compiler used e.g. Gcc
MM: means the MPI version

You can use it in the following way:

module load OpenFOAM/12-DP32-Gcc-CRAY-MPICH
source $FOAM_BASH

The module load is actually setting the $FOAM_BASH variables including the:

WM_COMPILER=<CC>
WM_LABEL_SIZE=<II>
WM_PRECISION_OPTION=<FF>
WM_MPLIB=CRAY-MPICH

specifications.

This specifications are important for the maintenance team to collect specific usage statistics.

The second line (source $FOAM_BASH) is corresponding to the “well known” sourcing of OpenFOAM/etc/bashrc

Advantage of using module supplied OpenFOAM:

The background module dependencies are loaded appropriately as the code was compiled.
You can access easily the release patches (which are common and very useful), since the background folder can be updated without affecting you.

You can access the actual commit number e.g. by:

blockMesh --help

Version specifications

Floating number precision:

SP single precision is usually not enough for convergence, DP double is the optimal choice. SPDP only exists for .com versions.

Integer number precision:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separately defined mesh elements (e.g. points, faces, cells)

Home folder:

It is also useful to specify your working directory, since libraries will be compiled to that location.

WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-<AA>/

<projectname> is the project name @ Komondor.

Special HPC settings

Parallel I/O

Since the number of files are strongly limited in your quota, it is strongly recommended to use the collated (org, com) fileHandler. On Komondor this is configured as default. If you have any anticipated problems, simple deactivate it for your actual case in system/controlDict:

OptimisationSwitches
{
    fileHandler uncollated;
}

Problems were found for some application (snappyHexMesh, createNonConformalCouple, createPatch) using the collated method. In this case -fileHandler uncollated should be used for these steps of the simulation and re-decompose it for the actual simulation.

IO grouping

For big cases running on multiple nodes when IO matters, it can be beneficial to group IO using IORANKS specified as

export FOAM_IORANKS='(0 G 2G 3G ....  NG)'

where G is the size of the group (e.g. 32) and NG (N*G) is the number of all processors used. You need to set up this before any OpenFOAM command execution.

Multi-node simulation

When running on high number of nodes the FI_CXI_RX_MATCH_MODE can significantly (factor of 2-10) influence your simulation speed.

export FI_CXI_RX_MATCH_MODE=hardware|hybrid|software

Specification of complex `srun` commands

There could be many reasons to introduce bash variables to be used in the slurm script instead of the srun command.

Specify special flags
Specify command for serial run
Prepare you script for various HPC environment where e.g. mpirun should be used (define variable as a function of e.g. hostname)

if [[ "$(hostname)" == "vn01" ]]; then
 echo "We are on Komondor"
 export Srun='srun --ntasks=1'
 export Prun='srun --cpu-bind=verbose --ntasks-per-node=23 '
else
   echo "We are elsewhere"
 export Srun=''
 export Prun='mpirun  '
fi



$Srun blockMesh &>log.blockMesh

$Prun snappyHexMesh -parallel &>log.snappyHexMesh

$Prun foamRun -parallel &>log.foamRun

Memory Requirement Estimation

OpenFOAM currently runs on CPUs (GPU support is under development and not available on Komondor). Therefore, you should use the cpu partition.

By default, each CPU core on Komondor provides 2 GB (2000M) of memory. In most cases, you can decompose your simulation into enough subdomains so that this per-core memory is sufficient.

However, in some cases—such as simulations with highly elongated geometries—the memory requirement per processor cannot be reduced simply by increasing the number of domains. If your case falls into this category, you might think of using Slurm’s --mem-per-cpu option to request more memory per core. This will not work as expected, because Slurm will adjust the CPU allocation in unexpected ways (e.g., doubling the number of cores).

The correct way to request more memory is to specify the total memory for the job using --mem and control the node allocation:

#SBATCH --mem=X*2000M
#SBATCH --nodes=Y

For example:

#SBATCH --mem=112000M   # X = 56 cores * 2000M
#SBATCH --nodes=Y

This means you allocate 56 CPU cores per node. When launching the job, you can control how many cores you actually use:

srun --ntasks-per-node=Z --nodes=Y foamRun -parallel

Here, the job will run on Z * Y cores, and each core will effectively have (X / Z) * 2000M memory available.

Why does this matter?

If Z is much smaller than X, you are wasting CPU capacity. Therefore, you need to determine the minimum X / Z ratio that satisfies your memory requirements.

Standard tools like jobstats are not sufficient for this, because they sample memory usage infrequently and cannot capture short-lived memory peaks. For accurate monitoring (e.g., every 0.1 seconds), use the PMrun script provided on Komondor.

Memory usage measurement

PMrun — Per-node Memory Sampler for Slurm/OpenFOAM

The script is available on Komondor at:

/opt/software/packages/openfoam/bin/PMrun

This directory also contains other useful helper scripts. To make them easily accessible, add it to your PATH in ~/.bashrc:

export PATH=/opt/software/packages/openfoam/bin:$PATH

PMrun launches a lightweight per-node memory monitor in parallel with your OpenFOAM jobstep under Slurm. The $Prun command can be replaced for the case you are investigating memory usage. It records process-level resident set size (RSS) on each node while your solver runs, then writes one log file per node:

memlog_<command>_<node>.txt

This is useful for detecting peak memory usage and temporal patterns across nodes/ranks during large-scale CFD runs.

Quick Start

# From within an sbatch allocation (or sbatch --wrap)
PMrun.sh <ntasks-per-node> [monitor_max_sec] [sample_dt] -- <OpenFOAM-command> [args...]

Examples

# Sample every 0.2 s for up to 300 s while createNonConformalCouples runs
PMrun.sh 34 300 0.2 -- createNonConformalCouples -overwrite -parallel

# Sample every 0.5 s for up to 600 s while simpleFoam runs
PMrun.sh 25 600 0.5 -- simpleFoam -parallel

Command-line arguments

<ntasks-per-node> (required)
[monitor_max_sec] (optional, default: 600)
[sample_dt] (optional, default: 0.1)
-- (separator)
<OpenFOAM-command> [args...] (required)

What the script does

Starts a lightweight monitor with one task per node using srun --overlap.
Logs one line per matching process at each sample: - UNIX epoch time - Sample index - PID - RSS in kB - Executable name
Runs the primary OpenFOAM job.
Stops the monitor when the command disappears or time limit is reached.

Output files

log.<command>: stdout/stderr of the primary run.
memlog_<command>_<node>.txt: per-node memory samples.

Each log line has:

<epoch_utc_s> <sample_idx> <pid> <rss_kB> <comm>

Memory Aggregation and Plotting Tool

A second script to analyze files generated by PMrun is plot_mem.sh. This script aggregates per-process RSS memory logs and produces per-node time series and plots for total memory and utilization.

Inputs

The script expects one or more files in the current directory named:

memlog_<COMMAND>_<node>.txt

Each file may have either 3 or 4 whitespace-separated columns:

4-column format: epoch sample_idx pid rss_kB
3-column format: sample_idx pid rss_kB

Lines starting with # are treated as headers and ignored.

Usage

./plot_mem.sh COMMAND [sample_interval_s]

COMMAND: label used to select input files and to name outputs (e.g., foamMultiRun, simpleFoam, etc.).
sample_interval_s: sampling interval used by the logger; default: 0.2 s.

Environment Variables

ALLOC_UNIT (default: MiB)
Allowed values: MiB (×1024) or MB (×1000).
ALLOC_PER_PROC (default: 2000)
Memory allocation per process, as a number in the unit given by ALLOC_UNIT. Used to compute utilization. The default value is inline with actual cpu partition configuration.
NPROC_NODE (default: 0)
If > 0, force the number of processes per node to this value when computing allocation (more stable than counting lines per sample). If 0, the script uses the observed count per sample.

Examples

plot_mem.sh foamMultiRun 0.2

ALLOC_PER_PROC=4000 ALLOC_UNIT=MB plot_mem.sh simpleFoam 0.5
NPROC_NODE=64 plot_mem.sh foamMultiRun 0.2

Outputs

Data files (per node):

memsum_<COMMAND>_<node>.dat Columns: time_s total_GiB — total RSS aggregated over processes.
memutil_<COMMAND>_<node>.dat Columns: time_s used_GiB alloc_GiB util_% nproc_seen

Plots (PNG):

mem_total_per_node_<COMMAND>.png Total memory curves per node.
mem_util_per_node_<COMMAND>.png Utilization curves per node (used / allocated x 100%).

Method

For each memlog_<COMMAND>_<node>.txt file: - Extract sample_idx and rss_kB (header is skipped; supports both 3- and 4-column formats). - Sort by sample index.
Aggregate per sample index: - Total memory: sum of rss_kB across processes, converted to GiB. - Utilization:
- used_GiB = sum of rss_kB / 2^20
- alloc_GiB = (nproc × ALLOC_PER_PROC in kB) / 2^20
- util_% = used / alloc × 100 (0 if alloc == 0)
- nproc is either the observed process count or NPROC_NODE if set.
Time coordinate assumes zero-based start: time_s = (sample_idx - 1) × sample_interval_s.

Notes

If no matching log files are found, the script exits with an error.
Y-axis label units: - Memory in GiB (binary gigabytes). - Utilization in % (0–130% shown by default to visualize over-allocation).

Configuration examples

Basic slurm settings

Simulation can be submitted using the following slurm script:

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --mail-type=END
#SBATCH --mail-user=your.name@mailserver.com

#SBATCH --ntasks=256
#SBATCH -J motorBikeSteady_256
# SBATCH --exclusive
#SBATCH --mem-per-cpu=2000M

module load OpenFOAM/12-DP32-Gcc-CRAY-MPICH
source $FOAM_BASH  WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-<AA>/

# To make number of cores passed to the domain decomposition
sed -i 's~numberOfSubdomains.*~numberOfSubdomains ${SLURM_NTASKS};~g'  system/decomposeParDict

export FI_CXI_RX_MATCH_MODE=hybrid

./Allrun

Advanced slurm settings

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --mail-type=END
#SBATCH --mail-user=your.name@mailserver.com
#SBATCH -J motorBikeSteady_4nodes
#SBATCH --nodes=4
#SBATCH --mem=120000M

module load OpenFOAM/12-DP32-Gcc-CRAY-MPICH
source $FOAM_BASH  WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-<AA>/

if [[ "$(hostname)" == "vn01" ]]; then
 echo "We are on Komondor"
 export Srun='srun --ntasks=1'
 export Prun='srun --cpu-bind=verbose --ntasks-per-node=50 '
else
   echo "We are elsewhere"
 export Srun=''
 export Prun='mpirun  '
fi

# To make number of cores passed to the domain decomposition
sed -i 's~numberOfSubdomains.*~numberOfSubdomains 200;~g'  system/decomposeParDict

export FI_CXI_RX_MATCH_MODE=hybrid

## For apriory memory investigation
# PMrun 50 600 0.1 -- foamRun --parallel
$Prun  foamRun --parallel &>log.foamRun

First run a short simulation using PMrun providing high amount of memory than analyze detailed memory usage (plot_mem.sh) and reduce your memory requirement to the real need.

Storage considerations

You should use the /project/ filesystem to store your case setup, than copy it to the /scratch/ filesystem and run it from there. When simulation converged, you should move your converged result back to your /project/ folder.

It is highly suggested to script this process!

Basic TODOs

It is useful to create your user directory:

mkdir -p /project/${project}/OpenFOAM/${USER}-<CC>/run

Practical tools

Convergence plotting

Use gnuplot for convergence plotting, it is an efficient tool for continuous monitoring.

set term qt size 1366, 768
# set term x11 size 1366, 768

delta_v(x) = ( vD = x - old_v, old_v = x, vD)
old_v = NaN

while (1) {
   set multiplot layout 3,1
      set log y
      set autoscale y

      plot "< cat log.foamRun | grep 'Solving for Ux' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "Ux", \
      "< cat log.foamRun | grep 'Solving for Uy' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "Uy", \
      "< cat log.foamRun | grep 'Solving for Uz' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "Uz", \
      "< cat log.foamRun | grep 'Solving for p' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "p", \
      "< cat log.foamRun | grep 'Solving for omega' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "omega", \
      "< cat log.foamRun | grep 'Solving for k' | cut -d' ' -f9 | tr ',' ' ' " every 1::0 t "k"

      unset log y

      set yrange [0:2]
      plot "< cat log.foamRun | grep 'ExecutionTime =' | cut -d' ' -f3 " using 0:(delta_v($1)) t "ExecutionTime"

      plot "< cat log.foamRun | grep 'Cl    =' | cut -d'=' -f2  " every 1::0 t "Cl", \
      "< cat log.foamRun | grep 'Cd    =' | cut -d'=' -f2  " every 1::0 t "Cd"

   unset multiplot

pause 10

}

Processing the simulation output (e.g.: log.foamRun) is useful especially if they are appended for continuation of interrupted jobs (foamRun -parallel &>>log.foamRun).
For HPC environment it is especially useful to precess the Execution Time of each iteration, it can easily highlight many type of user error or HPC environment effects.
Use qt or x11 for plotting each having its advantages and disadvantages.

Editing

Use Visual Studio Code remote development technique for any longer editing.

OpenFOAM extension is useful for configuration file editing.

you should specify your username

username@komondor.hpc.kifu.hu

and also edit your ssh config file.

~/.ssh/config

Download results

It is efficient to use VS Code RightClick Download... option.

Terminal emulation

MobaXterm is a handy tool for terminal emulation.

Pre-processing tools

Salome-Platform

Salome-Platform is an opensource CAD and meshing framework. You can use it for automated/scripted geometry and mesh generation, in your OpenFOAM workflow.

Your generated salome script can be run using -t flag without GUI:

module load SALOME/15
salome -t script.py

if the above fails you can try the more tested version:

module load SALOME/12
salome -t script.py

Author:

Máté LOHÁSZ

Version:

Oct. 2025.