Developer Notes

This chapter collects practical guidance for contributing to hypredrive with a focus on:

  • Continuous Integration (CI)

  • Static code analysis (cppcheck, clang-tidy)

  • Code coverage (gcov/gcovr, CTest)

  • Fuzzing replay and live fuzz campaigns

It explains how the CI is structured, how to reproduce checks locally, and what options and targets are available in CMake to enable these workflows. New contributors should read this once before opening their first PR.

Library API Ownership Semantics

For public setter APIs that accept optional external hypre objects:

  • HYPREDRV_LinearSystemSetInitialGuess(h, vec)

  • HYPREDRV_LinearSystemSetReferenceSolution(h, vec)

  • HYPREDRV_LinearSystemSetPrecMatrix(h, mat)

the NULL argument keeps the existing file/default behavior from parsed input args. When the argument is non-NULL, hypredrive uses the supplied object directly.

Ownership depends on library mode:

  • HYPREDRV_SetLibraryMode enabled: supplied objects are borrowed and never destroyed by hypredrive.

  • HYPREDRV_SetLibraryMode disabled: ownership is transferred to hypredrive, and the objects can be destroyed on replacement or object teardown.

When documenting or reviewing changes around these APIs, keep this object-lifetime contract explicit in both code comments and user docs.

CI Overview

Workflows

Hypredrive uses GitHub Actions with the following workflows (see .github/workflows):

  • ci.yml: main build-and-test matrix (Ubuntu + macOS; compilers: gcc/clang; build type: Debug)

  • format.yml: code style checks (clang-format, private/public naming prefix validation, binary symbol prefix validation)

  • docs.yml: builds documentation (Sphinx and Doxygen jobs)

  • coverage.yml: builds with coverage instrumentation and generates HTML/XML reports

  • analysis.yml: code analysis (static: cppcheck and clang-tidy; dynamic: ASan/UBSan with gcc/clang)

To reduce duplication, CI uses composite actions in .github/actions:

  • setup-ubuntu: installs compilers, MPI, and tools

  • build-hypre: builds and caches HYPRE from source and exposes its install prefix via the hypre_prefix output

HYPRE reuse and caching

All jobs that need HYPRE call the build-hypre action and pass:

  • hypre_version (default: master)

  • compiler (gcc or clang)

  • build_type (Debug)

  • hypre_shared_libs (ON)

  • os (ubuntu-latest or macos-latest)

The action caches HYPRE under ${{ runner.tool_cache }}/hypre/<version> with a descriptive key that includes OS, compiler, build type, and whether libraries are shared (shared/static).

Automatic HYPRE build

When building locally, hypredrive can automatically fetch and build HYPRE from source using CMake’s FetchContent if HYPRE_ROOT is not specified. This feature:

  • Automatically downloads HYPRE from GitHub (default: master branch)

  • Inherits all CMake arguments from the parent project (build type, compilers, TPLs, etc.)

  • Builds HYPRE in the same build tree with unified library and include directories

  • Supports specifying HYPRE version via -DHYPRE_VERSION=<version>

To use a pre-built HYPRE instead, specify -DHYPRE_ROOT=<path>.

MPI configuration

  • MPI implementation: OpenMPI (default in CI). MPICH is also supported.

  • C compiler for CMake: -DCMAKE_C_COMPILER=mpicc.

  • On Ubuntu, the action adds -DMPI_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/openmpi/include.

Local reproduction of CI

On Ubuntu (gcc example):

sudo apt-get update
sudo apt-get install -y cmake ninja-build libopenmpi-dev openmpi-bin ccache clang-format gcc
# Build HYPRE once and set CMAKE_PREFIX_PATH
git clone --depth 1 --branch master https://github.com/hypre-space/hypre.git
cmake -S hypre/src -B hypre/build -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED_LIBS=ON \
  -DHYPRE_BUILD_TESTS=OFF -DHYPRE_BUILD_EXAMPLES=OFF \
  -DCMAKE_C_COMPILER=mpicc -DMPI_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/openmpi/include \
  -DCMAKE_INSTALL_PREFIX=$HOME/.local/hypre/master
cmake --build hypre/build --parallel
cmake --install hypre/build

export HYPRE_PREFIX=$HOME/.local/hypre/master
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug \
  -DHYPREDRV_ENABLE_TESTING=ON -DHYPREDRV_ENABLE_EXAMPLES=ON \
  -DHYPRE_ROOT=$HYPRE_PREFIX \
  -DBUILD_SHARED_LIBS=ON -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_C_COMPILER_LAUNCHER=ccache \
  -DCMAKE_PREFIX_PATH=${HYPRE_PREFIX}
cmake --build build --parallel
ctest --test-dir build --output-on-failure

On macOS (Apple clang):

brew update && brew install cmake ninja mpich hypre clang-format
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug \
  -DHYPREDRV_ENABLE_TESTING=ON -DHYPREDRV_ENABLE_EXAMPLES=ON \
  -DBUILD_SHARED_LIBS=ON -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_PREFIX_PATH=$(brew --prefix hypre)
cmake --build build --parallel
ctest --test-dir build --output-on-failure

Code Analysis

Overview

Code analysis runs in analysis.yml and includes both static and dynamic analysis:

Static analysis: - cppcheck (C99 rules, exhaustive checks) - clang-tidy (driven by compile_commands.json)

Dynamic analysis: - sanitizers (AddressSanitizer and UndefinedBehaviorSanitizer with gcc/clang)

CMake options and targets

  • Enable analysis flags and targets: -DHYPREDRV_ENABLE_ANALYSIS=ON.

  • Configure with export of compile commands for clang-tidy: -DCMAKE_EXPORT_COMPILE_COMMANDS=ON.

  • Static analysis targets: - cppcheck: runs analysis on src/ only, with includes to include/ and the build dir. - clang-tidy: runs clang-tidy (non-fix) across the project using the compile database.

  • Dynamic analysis: When HYPREDRV_ENABLE_ANALYSIS=ON, sanitizers (ASan/UBSan) are automatically enabled for all targets. Run tests with ctest to exercise the sanitizers.

cppcheck configuration

  • Scope narrowed to src/ to keep signal high and run time reasonable.

  • HYPRE includes are added so macros/types resolve.

  • Threads: cppcheck uses the same number as CMake parallel level when available.

  • Known suppressions: irrelevant warnings for mixed precision or specific HYPRE headers can be ignored (e.g., _hypre_IJ_mv.h, _hypre_utilities.h).

clang-tidy usage

  • Use the clang-tidy target for read-only reports.

  • Avoid running aggressive automatic fixes over the entire tree; they can mangle identifiers in macro-heavy code. If you must use clang-tidy -fix, constrain to specific files and review diffs carefully. Prefer incremental, human-reviewed edits.

  • If you add checks, ensure they don’t fight our established style or C idioms in this project.

Local runs

Static analysis:

cmake -S . -B build-analysis -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug \
  -DHYPREDRV_ENABLE_ANALYSIS=ON \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_PREFIX_PATH=${HYPRE_PREFIX}
cmake --build build-analysis --parallel
# cppcheck
cmake --build build-analysis --target cppcheck
# clang-tidy
cmake --build build-analysis --target clang-tidy

Dynamic analysis (sanitizers):

cmake -S . -B build-sanitizers -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug \
  -DHYPREDRV_ENABLE_TESTING=ON -DHYPREDRV_ENABLE_EXAMPLES=ON \
  -DHYPREDRV_ENABLE_ANALYSIS=ON \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_PREFIX_PATH=${HYPRE_PREFIX}
cmake --build build-sanitizers --parallel
cmake --build build-sanitizers --target data
# Run tests with sanitizers enabled
export ASAN_OPTIONS="detect_leaks=1:abort_on_error=1:print_stacktrace=1"
export UBSAN_OPTIONS="print_stacktrace=1:abort_on_error=1"
ctest --test-dir build-sanitizers --output-on-failure

Fuzzing

Overview

Fuzzing is opt-in and is enabled with -DHYPREDRV_ENABLE_FUZZING=ON. The fuzzing tree lives under tests/fuzz/ and uses a single harness source, tests/fuzz/harness.c, compiled into mode-specific targets:

  • hypredrv-fuzz-parse: parses YAML and CLI-style argument fragments

  • hypredrv-fuzz-solve: parses YAML and exercises a small one-rank solve path

  • hypredrv-fuzz-lsseq: reads compressed linear-system sequence files when HYPREDRV_ENABLE_COMPRESSION=ON

  • hypredrv-fuzz-matrix: reads IJ matrix files and multipart matrix sequences

  • hypredrv-fuzz-vector: reads IJ vector files and multipart vector sequences

The supported engines are:

  • replay: deterministic CTest replay of seeds and regressions

  • libfuzzer: live in-process fuzzing with Clang’s libFuzzer

  • afl: AFL++ fuzzing with afl-clang-fast or afl-clang-lto

Enabling fuzzing also enables testing and analysis, disables shared libraries, and rejects coverage builds. Coverage and sanitizer/fuzzing instrumentation should stay in separate build trees.

CMake options and targets

  • Enable fuzzing: -DHYPREDRV_ENABLE_FUZZING=ON.

  • Select an engine: -DHYPREDRV_FUZZ_ENGINE=replay|libfuzzer|afl. The default is replay.

  • Enable MemorySanitizer for fuzzing builds: -DHYPREDRV_FUZZ_MSAN=ON. This requires Clang and a compatible dependency stack.

  • Replay tests are registered under the fuzz-replay CTest label.

  • The build targets are hypredrv-fuzz-parse, hypredrv-fuzz-solve, hypredrv-fuzz-lsseq, hypredrv-fuzz-matrix, and hypredrv-fuzz-vector.

The solve replay tests require HYPRE APIs guarded by HYPREDRV_HAVE_HYPRE_21900_DEV0. When those APIs are unavailable, CMake prints a status message and skips solve replay registration.

Replay builds

Replay is the CI-friendly mode. It builds normal executables that accept one or more input files and registers every seed and regression as a CTest test.

cmake -S . -B build-fuzz -G Ninja \
  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
  -DHYPREDRV_ENABLE_FUZZING=ON \
  -DHYPREDRV_ENABLE_TESTING=ON
cmake --build build-fuzz --parallel
ctest --test-dir build-fuzz -L fuzz-replay --output-on-failure

To replay one mode:

ctest --test-dir build-fuzz -L fuzz-replay -R "matrix" --output-on-failure

Replay binaries are compiled with a mode-specific FUZZ_MODE definition. For manual debugging only, HYPREDRV_FUZZ_MODE=<mode> can override that baked-in mode in replay builds.

Because parse replay uses examples/*.yml directly, checked-in example YAML files are part of the fuzz replay corpus. New examples should be valid parser inputs unless the fuzz CMake registration is updated to exclude or gate them explicitly.

Live libFuzzer runs

Use a separate Clang build tree for live libFuzzer campaigns:

cmake -S . -B build-fuzz -G Ninja \
  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
  -DHYPREDRV_ENABLE_FUZZING=ON \
  -DHYPREDRV_FUZZ_ENGINE=libfuzzer \
  -DHYPREDRV_ENABLE_TESTING=ON
cmake --build build-fuzz --target hypredrv-fuzz-parse --parallel
tests/fuzz/fuzzing.sh parse 300 libfuzzer

The helper script accepts parse, solve, lsseq, matrix, or vector as the first argument and the run duration in seconds as the second argument:

tests/fuzz/fuzzing.sh matrix 600 libfuzzer
tests/fuzz/fuzzing.sh vector 600 libfuzzer

Set HYPREDRV_FUZZ_BUILD_DIR if the fuzz build tree is not build-fuzz.

AFL++ runs

Configure AFL++ builds with the AFL compiler wrapper:

CC=afl-clang-fast cmake -S . -B build-fuzz-afl -G Ninja \
  -DHYPREDRV_ENABLE_FUZZING=ON \
  -DHYPREDRV_FUZZ_ENGINE=afl \
  -DHYPREDRV_ENABLE_TESTING=ON
cmake --build build-fuzz-afl --target hypredrv-fuzz-matrix --parallel
HYPREDRV_FUZZ_BUILD_DIR=$PWD/build-fuzz-afl tests/fuzz/fuzzing.sh matrix 600 afl

The script chooses a default AFL timeout per mode. Override it with HYPREDRV_FUZZ_AFL_TIMEOUT_MS when investigating slower paths, especially solve. For parse and solve modes, the harness runs a small warmup input before starting the AFL forkserver. This intentionally pays setup cost once so HYPRE and hypredrive global state are initialized before AFL forks persistent children.

Corpora, dictionaries, and generated seeds

The versioned fuzz inputs are intentionally small and deterministic:

  • tests/fuzz/seeds/<mode>/ contains mode-specific seed inputs that are not already represented elsewhere in the repository.

  • tests/fuzz/regressions/<mode>/ contains minimized inputs for fixed bugs.

  • tests/fuzz/dicts/ contains dictionaries for YAML/CLI fragments and IJ-like binary formats.

  • tests/fuzz/tools/ contains seed generation helpers for IJ matrix, IJ vector, and linear-system sequence inputs.

Do not duplicate existing example YAML files under tests/fuzz/seeds. Parse replay uses examples/*.yml directly, and solve replay uses examples/ex1.yml, examples/ex2.yml, and examples/ex7.yml by path before adding the fuzz-specific solve seeds.

The live fuzzing helper creates per-run corpora under build-fuzz/fuzz-run/<mode>-<time> and copies in the checked-in seeds, regressions, and reused example YAML inputs. Those run directories, generated corpora, crashes, hangs, and minimization artifacts are local outputs and should not be committed.

Regression workflow

When a live campaign finds a crash, leak, timeout, or sanitizer report:

  • Minimize the input with the engine that found it.

  • Add the minimized reproducer to tests/fuzz/regressions/<mode>/ with a descriptive filename.

  • Confirm the replay test fails before the code fix.

  • Fix the owning module rather than weakening the harness.

  • Re-run the relevant replay subset, then the full fuzz-replay label.

Typical replay commands:

ctest --test-dir build-fuzz -L fuzz-replay -R "solve" --output-on-failure
ctest --test-dir build-fuzz -L fuzz-replay --output-on-failure

Mode ownership is a useful starting point for triage:

  • parse maps primarily to src/yaml.c and src/args.c.

  • solve also reaches src/linsys.c, src/solver.c, src/precon.c, and related setup paths.

  • matrix maps to src/matrix.c.

  • vector maps to vector input paths.

  • lsseq maps to src/lsseq.c and compression-dependent sequence handling.

CI fuzzing

The fuzzing workflow has three tiers:

  • Pull requests run replay tests for deterministic coverage of the checked-in corpus and regressions.

  • Labeled pull requests can run short libFuzzer smoke jobs.

  • Scheduled runs execute longer nightly libFuzzer campaigns and upload run artifacts.

Replay failures should be treated like normal test failures. Live fuzz failures should be minimized and promoted into tests/fuzz/regressions/<mode>/ so future CI catches the same bug deterministically.

Code Coverage

Overview

Coverage is generated in coverage.yml using gcovr (HTML and XML artifacts). The build uses -O0 -g --coverage and runs unit tests via CTest. We also expose a simple percentage summary in the job output.

CMake options and targets

  • Enable instrumentation: -DHYPREDRV_ENABLE_COVERAGE=ON (Debug builds).

  • Build normally, then run: ctest and cmake --build <build> --target coverage.

  • Artifacts: coverage.html and coverage.xml in the build directory.

Local reproduction

pip install gcovr
cmake -S . -B build-coverage -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug \
  -DHYPREDRV_ENABLE_TESTING=ON -DHYPREDRV_ENABLE_EXAMPLES=ON \
  -DHYPREDRV_ENABLE_COVERAGE=ON \
  -DBUILD_SHARED_LIBS=ON -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_PREFIX_PATH=${HYPRE_PREFIX}
cmake --build build-coverage --parallel
ctest --test-dir build-coverage --output-on-failure
cmake --build build-coverage --target coverage
# Open build-coverage/coverage.html

Performance Profiling with Caliper

Overview

When HYPREDRV_ENABLE_CALIPER=ON is set during configuration, hypredrive is instrumented with Caliper markers for performance profiling. Caliper provides runtime performance measurement and profiling capabilities that can help identify performance bottlenecks and optimize code paths.

Building with Caliper

Enable Caliper support during CMake configuration:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DHYPREDRV_ENABLE_CALIPER=ON \
  -DCMAKE_PREFIX_PATH=${HYPRE_PREFIX}
cmake --build build --parallel

By default, Caliper will be automatically fetched and built from source if it’s not found. To use a pre-built Caliper installation, set the CALIPER_DIR or CALIPER_ROOT environment variables, or ensure find_package(caliper) can locate it.

Using Caliper for Profiling

To collect performance data, set the CALI_CONFIG environment variable when running hypredrive:

CALI_CONFIG=runtime-report mpirun -np 1 ./build/hypredrive-cli examples/ex1.yml

Common Caliper configurations:

  • runtime-report: Print a summary report to stdout at program end

  • runtime-report,max_column_width=200,calc.inclusive,output=stdout,mpi-report: Print a detailed report, including MPI-related information, to stdout at program end.

  • spot: Generate Caliper output files for analysis with Caliper’s spot tool

For more information about Caliper configurations and services, see the Caliper documentation.

Documentation Builds

Two documentation systems are wired:

  • Doxygen (developer reference; target: doxygen-doc)

  • Sphinx (user manual; targets: sphinx-doc, sphinx-latexpdf)

Enable via CMake: -DHYPREDRV_ENABLE_DOCS=ON. The combined docs target builds Doxygen first, then invokes Sphinx (either via docs/Makefile if available or directly via sphinx-build).

Notes:

  • Doxygen LaTeX can build a refman.pdf; when present, CI copies it to build-docs/docs/devman-hypredrive.pdf.

  • Sphinx PDF (sphinx-latexpdf) is copied to docs/usrman-hypredrive.pdf when available.

Coding Style & Project Conventions

  • clang-format is enforced in CI; run make format locally (or the format CMake target) to format .c/.h files under include/, src/, and examples/src/.

  • Some macros are guarded with // clang-format off/on to preserve required layout. Avoid editing those blocks unless necessary.

  • Follow const-correctness and safe API patterns. Error-reporting helpers must be used consistently. The HYPREDRV_SAFE_CALL macro logs location info and aborts or traps under HYPREDRV_DEBUG=1.

  • For internal runtime traces, use the environment variable HYPREDRV_LOG_LEVEL: 0 disables traces (default), 1 logs lifecycle boundaries, 2 adds decision/context messages, and 3 enables deeper parse/linear-system/scaling subphase traces. Trace output is emitted to stderr and filtered to rank 0 by default. Set HYPREDRV_LOG_STREAM=stdout to emit the same traces to stdout instead.

  • HYPREDRV_LOG_LEVEL controls hypredrive traces only. Hypre’s own logging remains controlled by HYPRE_LOG_LEVEL (forwarded during runtime initialization when supported by the linked Hypre version).

  • Public API naming: HYPRE_ for public HYPRE APIs, hypre_ or project-private helpers are not exported. In hypredrive, public C API follows the HYPREDRV_ prefix.

Troubleshooting CI

  • If a composite action cannot be found, ensure it is not excluded by .gitignore and that the repo is checked out (actions/checkout) before using local actions.

  • On Ubuntu, missing MPI headers require the include hint used by the action (-DMPI_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/openmpi/include for OpenMPI, or /usr/include/x86_64-linux-gnu/mpich for MPICH).

  • If Doxygen fails with an OUTPUT_DIRECTORY error, the project already configures it to a build- relative docs directory; rebuild after cleaning stale artifacts.

  • If clang-tidy-fix causes unexpected renames in macro contexts, restrict fixes to specific files and review diffs manually.

Where to Look in the Tree

  • CI config: .github/workflows/*.yml

  • Composite actions: .github/actions/setup-ubuntu, .github/actions/build-hypre

  • CMake options: see the top-level CMakeLists.txt and the cmake/ modules

  • Tests: tests/ and cmake/HYPREDRV_Testing.cmake