Driver Examples (hypredrive CLI)¶
This section provides several examples demonstrating how to set up input files and use
the hypredrive executable (driver) to solve different types of linear-system problems.
All example inputs can be found in the examples folder and reference outputs at
examples/refOutput.
Note
These examples exercise hypredrive as a standalone driver using YAML input files.
If you are looking for application-side usage of the library API (libHYPREDRV),
see Library Examples (libHYPREDRV).
Note
Many examples require datasets that are not included in the repository. These datasets
must be downloaded from Zenodo at https://zenodo.org/records/17471036 before running
the examples. The most convenient way to download and extract all datasets is using
the CMake data target:
$ cmake --build <build-dir> --target data
Alternatively, you can download the datasets manually from the Zenodo record and extract
them into the data/ directory. See data/README.md for more details.
CLI overrides for the driver are documented in CLI Overrides (-a/--args).
Example 1: Minimal configuration¶
In this example, we solve a basic linear system using an AMG-PCG solver with default settings. This example showcases the minimum amount of information required in the input file to execute hypredrive.
We consider a linear system matrix arising from a seven-point finite-difference discretization of the Laplace equation on a 10×10×10 Cartesian grid. The right-hand side is the vector of ones. Both are read from file and partitioned for a single MPI rank; this example must therefore be run on a single process.
Note
This example requires the ps3d10pt7 dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Prepare your linear system files (
matrix_filenameandrhs_filename).Use the YAML configuration file
ex1.yml:
general:
use_millisec: on # Turn this off for reporting times in [s]
dev_pool_size: 0.01
linear_system:
rhs_filename: data/ps3d10pt7/np1/IJ.out.b
matrix_filename: data/ps3d10pt7/np1/IJ.out.A
solver: pcg
preconditioner: amg
Run hypredrive with the configuration file:
$ mpirun -np 1 ./hypredrive-cli examples/ex1.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
use_millisec: on
dev_pool_size: 0.01
linear_system:
rhs_filename: data/ps3d10pt7/np1/IJ.out.b
matrix_filename: data/ps3d10pt7/np1/IJ.out.A
solver: pcg
preconditioner: amg
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 1000 rows and 6400 nonzeros...
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 1.500 | 1.286 | 0.262 | 3.16e+01 | 4.98e-08 | 6 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
Warning
Make sure that hypredrive is executed from the top level project folder in order for
the relative paths in matrix_filename and rhs_filename to be
correct. Otherwise, adjust the relative paths for these entries accordingly.
Example 2: Parallel run with full AMG configuration¶
In this example, we solve the same problem as in the previous example, but partitioned for 4 processes. We also showcase all available input options for PCG and AMG in the configuration file.
Note
This example requires the ps3d10pt7 dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Prepare your linear system files.
Use the YAML configuration file
ex2.yml:
general:
use_millisec: on # Turn this off for reporting times in [s]
linear_system:
rhs_filename: data/ps3d10pt7/np4/IJ.out.b
matrix_filename: data/ps3d10pt7/np4/IJ.out.A
solver:
pcg:
max_iter: 100
two_norm: yes
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
residual_tol: 0.0
conv_fac_tol: 0.0
preconditioner:
amg:
tolerance: 0.0
max_iter: 1
print_level: 1
interpolation:
prolongation_type: extended+i
restriction_type: p_transpose
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.25
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0.0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: fsai
num_levels: 1
num_sweeps: 1
fsai:
max_iter: 1
print_level: 0
algo_type: 1
max_steps: 5
max_step_size: 1
eig_max_iters: 4
kap_tolerance: 1e-2
Run hypredrive with the configuration file:
$ mpirun -np 4 ./hypredrive-cli examples/ex2.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 4 MPI ranks
------------------------------------------------------------------------------------
general:
use_millisec: on
linear_system:
rhs_filename: data/ps3d10pt7/np4/IJ.out.b
matrix_filename: data/ps3d10pt7/np4/IJ.out.A
solver:
pcg:
max_iter: 100
two_norm: yes
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
residual_tol: 0.0
conv_fac_tol: 0.0
preconditioner:
amg:
tolerance: 0.0
max_iter: 1
print_level: 1
interpolation:
prolongation_type: extended+i
restriction_type: p_transpose
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.25
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0.0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: fsai
num_levels: 1
num_sweeps: 1
fsai:
max_iter: 1
print_level: 0
algo_type: 1
max_steps: 5
max_step_size: 1
eig_max_iters: 4
kap_tolerance: 1e-2
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 1000 rows and 6400 nonzeros...
*************************
* HYPRE FSAI Setup Info *
*************************
+---------------------------+
| No. MPI tasks: 4 |
| No. threads: 1 |
| Algorithm type: 1 |
| Local solve type: 1 |
| Max no. steps: 5 |
| Max step size: 1 |
| Kap grad tol: 1.0e-02 |
| Prec. density: 0.702 |
| Eig max iters: 4 |
| Omega factor: 0.938 |
+---------------------------+
Num MPI tasks = 4
Num OpenMP threads = 1
BoomerAMG SETUP PARAMETERS:
Max levels = 25
Num levels = 3
Strength Threshold = 0.250000
Interpolation Truncation Factor = 0.000000
Maximum Row Sum Threshold for Dependency Weakening = 0.900000
Coarsening Type = PMIS
measures are determined locally
No global partition option chosen.
Interpolation = extended+i interpolation
Operator Matrix Information:
nonzero entries/row row sums
lev rows entries sparse min max avg min max
======================================================================
0 1000 6400 0.006 4 7 6.4 0.000e+00 3.000e+00
1 351 7485 0.061 7 38 21.3 -2.054e-15 4.393e+00
2 62 1986 0.517 17 52 32.0 1.110e+00 5.392e+00
Interpolation Matrix Information:
entries/row min max row sums
lev rows x cols min max avgW weight weight min max
================================================================================
0 1000 x 351 1 4 4.0 5.263e-02 4.255e-01 4.194e-01 1.000e+00
1 351 x 62 1 4 4.0 5.234e-03 4.977e-01 1.628e-01 1.000e+00
Complexity: grid = 1.413000
operator = 2.479844
memory = 3.129062
BoomerAMG SOLVER PARAMETERS:
Maximum number of cycles: 1
Stopping Tolerance: 0.000000e+00
Cycle type (1 = V, 2 = W, etc.): 1
Relaxation Parameters:
Visiting Grid: down up coarse
Number of sweeps: 1 1 1
Type 0=Jac, 3=hGS, 6=hSGS, 9=GE: 13 14 9
Point types, partial sweeps (1=C, -1=F):
Pre-CG relaxation (down): 0
Post-CG relaxation (up): 0
Coarsest grid: 0
<b,b>: 1.000000e+03
Iters ||r||_2 conv.rate ||r||_2/||b||_2
----- ------------ --------- ------------
1 6.013527e+00 0.190164 1.901644e-01
2 3.023977e-01 0.050286 9.562655e-03
3 1.497186e-02 0.049510 4.734517e-04
4 7.279855e-04 0.048624 2.302092e-05
5 2.985711e-05 0.041013 9.441648e-07
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 0.228 | 2.175 | 0.591 | 3.16e+01 | 9.44e-07 | 5 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
Example 3: Minimal multigrid reduction strategy¶
In this example, we solve a linear system derived from the discretization of a
compositional flow problem from GEOS. Details about
how this linear system was generated can be found at data/compflow6k/README.md. This
example uses a MGR-GMRES solver and showcases the minimal configuration for setting up
the multigrid reduction preconditioner for this particular kind of linear system.
Note
This example requires the compflow6k dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Prepare your linear system files.
Use the YAML configuration file
ex3.yml:
general:
use_millisec: on # Turn this off for reporting times in [s]
linear_system:
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
solver: gmres
preconditioner:
mgr:
level:
0:
f_dofs: [2]
prolongation_type: jacobi
1:
f_dofs: [1]
g_relaxation: l1-hsgs
restriction_type: columped
coarsest_level: amg
Run hypredrive with the configuration file:
$ mpirun -np 1 ./hypredrive-cli examples/ex3.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
use_millisec: on
linear_system:
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
solver: gmres
preconditioner:
mgr:
level:
0:
f_dofs: [2]
prolongation_type: jacobi
1:
f_dofs: [1]
g_relaxation: l1-hsgs
restriction_type: columped
coarsest_level: amg
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 5625 rows and 77475 nonzeros...
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 2.559 | 3.091 | 2.533 | 1.00e+00 | 1.74e-07 | 8 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
Example 4: Advanced multigrid reduction strategy¶
In this example, we solve the same problem as before, but partitioned for 4 processes. Here, we showcase a more advanced setup of MGR involving multiple options.
Note
This example requires the compflow6k dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Prepare your linear system files.
Use the YAML configuration file
ex4.yml:
general:
use_millisec: on # Turn this off for reporting times in [s]
linear_system:
rhs_filename: data/compflow6k/np4/IJ.out.b
matrix_filename: data/compflow6k/np4/IJ.out.A
dofmap_filename: data/compflow6k/np4/dofmap.out
solver:
gmres:
min_iter: 0
max_iter: 100
skip_real_res_check: no
krylov_dim: 30
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
conv_fac_tol: 0.0
preconditioner:
mgr:
tolerance: 0.0
max_iter: 1
print_level: 0
coarse_th: 0.0
level:
0:
f_dofs: [2]
f_relaxation: single
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
1:
f_dofs: [1]
f_relaxation: single
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level:
amg:
tolerance: 0.0
max_iter: 1
print_level: 0 # Turn off printout from AMG since it's done from MGR
interpolation:
prolongation_type: extended+i
restriction_type: p_transpose
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.3
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: ilu
num_levels: 0
num_sweeps: 1
Run hypredrive with the configuration file:
$ mpirun -np 4 ./hypredrive-cli examples/ex4.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 4 MPI ranks
------------------------------------------------------------------------------------
general:
use_millisec: on
linear_system:
rhs_filename: data/compflow6k/np4/IJ.out.b
matrix_filename: data/compflow6k/np4/IJ.out.A
dofmap_filename: data/compflow6k/np4/dofmap.out
solver:
gmres:
min_iter: 0
max_iter: 100
skip_real_res_check: no
krylov_dim: 30
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
conv_fac_tol: 0.0
preconditioner:
mgr:
tolerance: 0.0
max_iter: 1
print_level: 0
coarse_th: 0.0
level:
0:
f_dofs: [2]
f_relaxation: single
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
1:
f_dofs: [1]
f_relaxation: single
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level:
amg:
tolerance: 0.0
max_iter: 1
print_level: 0
interpolation:
prolongation_type: extended+i
restriction_type: p_transpose
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.3
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: ilu
num_levels: 0
num_sweeps: 1
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 5625 rows and 77475 nonzeros...
L2 norm of b: 1.000000e+00
Initial L2 norm of residual: 1.000000e+00
=============================================
Iters resid.norm conv.rate rel.res.norm
----- ------------ ---------- ------------
1 3.477559e-01 0.347756 3.477559e-01
2 1.570413e-01 0.451585 1.570413e-01
3 2.996166e-02 0.190788 2.996166e-02
4 5.941462e-03 0.198302 5.941462e-03
5 1.907352e-03 0.321024 1.907352e-03
6 3.353142e-04 0.175801 3.353142e-04
7 4.285191e-05 0.127796 4.285191e-05
8 7.523042e-06 0.175559 7.523042e-06
9 1.228061e-06 0.163240 1.228061e-06
10 1.527693e-07 0.124399 1.527693e-07
Final L2 norm of residual: 1.527692e-07
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 1.394 | 3.566 | 1.759 | 1.00e+00 | 1.53e-07 | 10 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
Example 5: Spreading input parameters in multiple files¶
In this example, we solve the same problem as in example 3, but using the same solver and
preconditioner parameters as in example 4. In addition, we define these parameters in
separate files, which are included in the main input file via the include keyword.
Note
This example requires the compflow6k dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Prepare your linear system files.
Define the input file containing the solver parameters
ex5-gmres.yml:
gmres:
min_iter: 0
max_iter: 100
skip_real_res_check: no
krylov_dim: 30
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
conv_fac_tol: 0.0
Define the input file containing the preconditioner parameters
ex5-mgr.yml:
mgr:
tolerance: 0.0
max_iter: 1
print_level: 0
coarse_th: 0.0
level:
0:
f_dofs: [2]
f_relaxation: jacobi
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
1:
f_dofs: [1]
f_relaxation: jacobi
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level:
amg:
tolerance: 0.0
max_iter: 1
print_level: 0 # Turn off printout from AMG since it's done from MGR
interpolation:
prolongation_type: extended+i
restriction_type: 0
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.3
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0.0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: ilu
num_levels: 0
num_sweeps: 1
Define the main input file
ex5.yml:
general:
use_millisec: on # Turn this off for reporting times in [s]
linear_system:
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
solver:
include: ex5-gmres.yml
preconditioner:
include: ex5-mgr.yml
Run hypredrive with the configuration file:
$ mpirun -np 1 ./hypredrive-cli examples/ex5.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
use_millisec: on
linear_system:
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
solver:
gmres:
min_iter: 0
max_iter: 100
skip_real_res_check: no
krylov_dim: 30
rel_change: no
print_level: 2
relative_tol: 1.0e-6
absolute_tol: 0.0
conv_fac_tol: 0.0
preconditioner:
mgr:
tolerance: 0.0
max_iter: 1
print_level: 0
coarse_th: 0.0
level:
0:
f_dofs: [2]
f_relaxation: jacobi
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
1:
f_dofs: [1]
f_relaxation: jacobi
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level:
amg:
tolerance: 0.0
max_iter: 1
print_level: 0
interpolation:
prolongation_type: extended+i
restriction_type: 0
trunc_factor: 0.0
max_nnz_row: 4
coarsening:
type: pmis
strong_th: 0.3
seq_amg_th: 0
max_coarse_size: 64
min_coarse_size: 0
max_levels: 25
num_functions: 1
rap2: off
mod_rap2: on
keep_transpose: off
max_row_sum: 0.9
aggressive:
num_levels: 0
num_paths: 1
prolongation_type: multipass
trunc_factor: 0.0
max_nnz_row: 0
P12_trunc_factor: 0.0
P12_max_elements: 0
relaxation:
down_type: forward-hl1gs
up_type: backward-hl1gs
coarse_type: ge
down_sweeps: -1
up_sweeps: -1
coarse_sweeps: 1
num_sweeps: 1
order: 0
weight: 1.0
outer_weight: 1.0
smoother:
type: ilu
num_levels: 0
num_sweeps: 1
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 5625 rows and 77475 nonzeros...
L2 norm of b: 1.000000e+00
Initial L2 norm of residual: 1.000000e+00
=============================================
Iters resid.norm conv.rate rel.res.norm
----- ------------ ---------- ------------
1 3.351387e-01 0.335139 3.351387e-01
2 1.465206e-01 0.437194 1.465206e-01
3 2.534289e-02 0.172965 2.534289e-02
4 2.963971e-03 0.116955 2.963971e-03
5 3.916325e-04 0.132131 3.916325e-04
6 4.467956e-05 0.114085 4.467956e-05
7 3.966829e-06 0.088784 3.966829e-06
8 3.735284e-07 0.094163 3.735284e-07
Final L2 norm of residual: 3.735284e-07
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 2.872 | 3.462 | 3.358 | 1.00e+00 | 3.74e-07 | 8 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
Example 6: Full eigenspectrum computation (single rank)¶
In this example, we enable the debug/analysis eigenspectrum capability to compute the full eigenspectrum of the matrix using dense LAPACK routines. This path is intended for small matrices and single-rank runs.
Note
You must build hypredrive with eigenspectrum support enabled, e.g. configure with
-DHYPREDRV_ENABLE_EIGSPEC=ON.
Note
This example requires the compflow6k dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Use the YAML configuration file
ex6.yml:
general:
use_millisec: off
linear_system:
# Use the compflow6k single-rank matrix (binary format available)
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
# Enable full eigenspectrum computation
eigspec:
enable: yes # Turn eigenspectrum computation on/off
vectors: no # write eigenvectors as complex pairs (Re, Im)
hermitian: no # set to yes if the matrix is symmetric/Hermitian
preconditioned: yes # compute eigenspectrum of left precontioned matrix
output_prefix: data/compflow6k/np1/eig # prefix name for eigenvectors file
# Minimal solver/preconditioner sections (preconditioner is required by parser)
solver: gmres
preconditioner:
mgr:
level:
0:
f_dofs: [2]
prolongation_type: jacobi
1:
f_dofs: [1]
g_relaxation: l1-hsgs
restriction_type: columped
coarsest_level: amg
Run hypredrive with the configuration file (single rank):
$ mpirun -np 1 ./hypredrive-cli examples/ex6.yml
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
use_millisec: off
linear_system:
rhs_filename: data/compflow6k/np1/IJ.out.b
matrix_filename: data/compflow6k/np1/IJ.out.A
dofmap_filename: data/compflow6k/np1/dofmap.out
eigspec:
enable: yes
vectors: no
hermitian: no
preconditioned: yes
output_prefix: data/compflow6k/np1/eig
solver: gmres
preconditioner:
mgr:
level:
0:
f_dofs: [2]
prolongation_type: jacobi
1:
f_dofs: [1]
g_relaxation: l1-hsgs
restriction_type: columped
coarsest_level: amg
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 5625 rows and 77475 nonzeros...
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [s] | times [s] | times [s] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 0.002 | 0.003 | 0.002 | 1.00e+00 | 1.74e-07 | 8 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
(Optional) Plot the full eigenspectrum of the preconditioned matrix using the eigplot.py script:
$ scripts/eigplot.py -f data/compflow6k/np1/eig.values.txt
Example 7: Solving a sequence of linear systems¶
In this example, we solve a sequence of linear systems stored in a hierarchical directory layout.
The systems originate from a single-rank (np1) GEOS multiphase poromechanics benchmark.
We use MGR-preconditioned FGMRES (MGR-FGMRES) as the linear solver.
See data/poromech2k/README.md for more details about the problem in GEOS.
Note
This example requires the poromech2k dataset. Make sure you have downloaded the
datasets from Zenodo (see the note at the beginning of this section).
Use the YAML configuration file
ex7.yml:
general:
use_millisec: on
linear_system:
dirname: data/poromech2k/np1/ls
init_suffix: 0
last_suffix: 24
rhs_filename: IJ.out.b
matrix_filename: IJ.out.A
dofmap_filename: dofmap.out
solver:
fgmres:
max_iter: 100
krylov_dim: 30
print_level: 1
relative_tol: 1.0e-6
preconditioner:
mgr:
max_iter: 1
tolerance: 0.0
print_level: 0
coarse_th: 1e-20
level:
0:
f_dofs: [0, 1, 2] # Eliminate displacement DOFs
f_relaxation: amg
amg:
coarsening:
type: pmis
strong_th: 0.5
num_functions: 3
filter_functions: on
g_relaxation: none
restriction_type: injection
prolongation_type: blk-jacobi
coarse_level_type: non-galerkin
1:
f_dofs: [5] # Eliminate volume constraint DOFs
f_relaxation: jacobi
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
2:
f_dofs: [4] # Eliminate component density DOFs
f_relaxation: single
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level: # Pressure DOFs
amg:
max_iter: 1
tolerance: 0.0
relaxation:
down_type: l1-jacobi
up_type: l1-jacobi
Run hypredrive with the configuration file (single rank):
$ mpirun -np 1 ./hypredrive-cli examples/ex7.yml
Your output should look like (truncated):
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
use_millisec: on
linear_system:
dirname: data/poromech2k/np1/ls
init_suffix: 0
last_suffix: 24
rhs_filename: IJ.out.b
matrix_filename: IJ.out.A
dofmap_filename: dofmap.out
solver:
fgmres:
max_iter: 100
krylov_dim: 30
print_level: 1
relative_tol: 1.0e-6
preconditioner:
mgr:
max_iter: 1
tolerance: 0.0
print_level: 0
coarse_th: 1e-20
level:
0:
f_dofs: [0, 1, 2]
f_relaxation: amg
amg:
coarsening:
type: pmis
strong_th: 0.5
num_functions: 3
filter_functions: on
g_relaxation: none
restriction_type: injection
prolongation_type: blk-jacobi
coarse_level_type: non-galerkin
1:
f_dofs: [5]
f_relaxation: jacobi
g_relaxation: none
restriction_type: injection
prolongation_type: jacobi
coarse_level_type: rap
2:
f_dofs: [4]
f_relaxation: single
g_relaxation: ilu
restriction_type: columped
prolongation_type: injection
coarse_level_type: rap
coarsest_level:
amg:
max_iter: 1
tolerance: 0.0
relaxation:
down_type: l1-jacobi
up_type: l1-jacobi
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #1 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #2 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #3 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #4 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #5 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #6 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #7 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #8 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #9 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #10 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #11 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #12 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #13 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #14 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #15 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #16 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #17 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #18 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #19 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #20 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #21 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #22 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #23 with 2187 rows and 127161 nonzeros...
====================================================================================
Solving linear system #24 with 2187 rows and 127161 nonzeros...
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 4.619 | 5.162 | 3.972 | 1.00e+00 | 9.70e-07 | 13 |
| 1 | 2.982 | 4.362 | 6.815 | 1.00e+00 | 4.51e-07 | 18 |
| 2 | 3.246 | 3.858 | 3.941 | 1.00e+00 | 8.86e-07 | 12 |
| 3 | 3.811 | 4.797 | 4.058 | 1.00e+00 | 3.94e-07 | 12 |
| 4 | 4.116 | 4.418 | 4.899 | 1.00e+00 | 6.55e-07 | 16 |
| 5 | 3.404 | 4.324 | 5.112 | 1.00e+00 | 4.88e-07 | 16 |
| 6 | 3.745 | 3.802 | 4.443 | 1.00e+00 | 9.06e-07 | 14 |
| 7 | 3.453 | 3.742 | 4.423 | 1.00e+00 | 5.48e-07 | 14 |
| 8 | 3.603 | 4.061 | 4.661 | 1.00e+00 | 8.11e-07 | 14 |
| 9 | 3.649 | 4.059 | 4.377 | 1.00e+00 | 4.39e-07 | 14 |
| 10 | 3.703 | 3.886 | 3.999 | 1.00e+00 | 5.96e-07 | 13 |
| 11 | 3.348 | 3.990 | 4.111 | 1.00e+00 | 4.05e-07 | 13 |
| 12 | 3.401 | 4.149 | 4.889 | 1.00e+00 | 4.65e-07 | 14 |
| 13 | 3.691 | 3.793 | 4.939 | 1.00e+00 | 6.07e-07 | 14 |
| 14 | 3.653 | 4.079 | 4.527 | 1.00e+00 | 5.01e-07 | 13 |
| 15 | 4.059 | 4.146 | 4.037 | 1.00e+00 | 4.86e-07 | 13 |
| 16 | 3.365 | 3.826 | 2.549 | 1.00e+00 | 5.62e-07 | 8 |
| 17 | 3.506 | 3.798 | 4.429 | 1.00e+00 | 7.91e-07 | 14 |
| 18 | 4.249 | 3.895 | 4.213 | 1.00e+00 | 5.40e-07 | 14 |
| 19 | 3.567 | 3.618 | 4.772 | 1.00e+00 | 6.37e-07 | 16 |
| 20 | 3.407 | 3.669 | 4.271 | 1.00e+00 | 4.64e-07 | 12 |
| 21 | 4.167 | 4.479 | 4.899 | 1.00e+00 | 3.67e-07 | 15 |
| 22 | 3.853 | 4.606 | 5.302 | 1.00e+00 | 8.41e-07 | 16 |
| 23 | 4.375 | 4.233 | 4.316 | 1.00e+00 | 9.49e-07 | 12 |
| 24 | 3.606 | 4.721 | 4.167 | 1.00e+00 | 8.28e-07 | 13 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
(Optional) Plot the sparsity pattern of the matrices using the spyplot.py script:
$ scripts/spyplot.py -d data/poromech2k/np1 -r 0:24 -l -t 1e-20
Example 8: Multiple preconditioner variants in one YAML¶
This example demonstrates how to run several preconditioner variants defined as a
YAML sequence under a single preconditioner block. hypredrive will loop over each
variant and report a separate stats entry per variant while reusing the same linear system.
Note
This example uses the ps3d10pt7 dataset (multiple ranks). Ensure the dataset is
available (see the note at the top of this page).
Use the YAML configuration file
ex8.yml:
general:
warmup: false
num_repetitions: 1
use_millisec: on
linear_system:
type: ij
rhs_filename: data/ps3d10pt7/np4/IJ.out.b
matrix_filename: data/ps3d10pt7/np4/IJ.out.A
solver:
pcg:
relative_tol: 1.0e-9
max_iter: 500
print_level: 0
preconditioner:
amg:
- coarsening:
type: HMIS
strong_th: 0.25
interpolation:
prolongation_type: "MM-ext+i"
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
- coarsening:
type: HMIS
strong_th: 0.5
interpolation:
prolongation_type: "MM-ext+i"
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
chebyshev:
order: 4
fraction: 0.1
- coarsening:
type: HMIS
strong_th: 0.8
interpolation:
prolongation_type: "MM-ext+i"
relaxation:
down_type: l1sym-hgs
down_sweeps: 1
up_type: l1sym-hgs
up_sweeps: 1
- coarsening:
type: HMIS
strong_th: 0.9
interpolation:
prolongation_type: "MM-ext+i"
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
smoother:
type: ilu
num_levels: 1
ilu:
type: bj-iluk
- coarsening:
type: PMIS
strong_th: 0.5
interpolation:
prolongation_type: direct_sep_weights
relaxation:
down_type: 8
down_sweeps: 2
up_type: 8
up_sweeps: 2
Run hypredrive with the configuration file (single rank):
$ mpirun -np 1 ./hypredrive-cli examples/ex8.yml -q
Your output should look like:
Date and time: YYYY-MM-DD HH:MM:SS
Using HYPREDRV_DEVELOP_STRING: HYPREDRV_VERSION_GOES_HERE
Running on 1 MPI rank
------------------------------------------------------------------------------------
general:
warmup: false
num_repetitions: 1
use_millisec: on
linear_system:
type: ij
rhs_filename: data/ps3d10pt7/np4/IJ.out.b
matrix_filename: data/ps3d10pt7/np4/IJ.out.A
solver:
pcg:
relative_tol: 1.0e-9
max_iter: 500
print_level: 0
preconditioner:
amg:
- coarsening:
type: hmis
strong_th: 0.25
interpolation:
prolongation_type: mm-ext+i
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
- coarsening:
type: hmis
strong_th: 0.5
interpolation:
prolongation_type: mm-ext+i
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
chebyshev:
order: 4
fraction: 0.1
- coarsening:
type: hmis
strong_th: 0.8
interpolation:
prolongation_type: mm-ext+i
relaxation:
down_type: l1sym-hgs
down_sweeps: 1
up_type: l1sym-hgs
up_sweeps: 1
- coarsening:
type: hmis
strong_th: 0.9
interpolation:
prolongation_type: mm-ext+i
relaxation:
down_type: 16
down_sweeps: 1
up_type: 16
up_sweeps: 1
smoother:
type: ilu
num_levels: 1
ilu:
type: bj-iluk
- coarsening:
type: pmis
strong_th: 0.5
interpolation:
prolongation_type: standard
relaxation:
down_type: 8
down_sweeps: 2
up_type: 8
up_sweeps: 2
------------------------------------------------------------------------------------
====================================================================================
Solving linear system #0 with 1000 rows and 6400 nonzeros...
====================================================================================
STATISTICS SUMMARY:
+--------+-------------+-------------+-------------+------------+------------+--------+
| | LS build | setup | solve | initial | relative | |
| Entry | times [ms] | times [ms] | times [ms] | res. norm | res. norm | iters |
+--------+-------------+-------------+-------------+------------+------------+--------+
| 0 | 0.340 | 1.407 | 0.382 | 1.58e+01 | 4.11e-10 | 7 |
| 1 | | 1.136 | 0.529 | 1.58e+01 | 1.67e-11 | 6 |
| 2 | | 0.876 | 0.565 | 1.58e+01 | 9.03e-10 | 6 |
| 3 | | 0.939 | 0.495 | 1.58e+01 | 3.00e-11 | 7 |
| 4 | | 1.258 | 0.617 | 1.58e+01 | 1.69e-11 | 6 |
+--------+-------------+-------------+-------------+------------+------------+--------+
Date and time: YYYY-MM-DD HH:MM:SS
${HYPREDRIVE_PATH}/hypredrive-cli done!
(Optional) Plot per-variant timing or iteration bars using
scripts/analyze_statistics.py. Provide one label per table entry via-ln(order matches the stats table):
$ scripts/analyze_statistics.py -f examples/refOutput/ex8.txt -m bar -p total \
-ln "AMG-1" "AMG-2" "AMG-3" "AMG-4" "AMG-5" \
-s ex8.svg -T "Example 8 total time"
# Use -p iters to compare iteration counts instead of timings
Total time (setup + solve) for each preconditioner variant in Example 8.¶
Note
A multi-file version of this input is also provided as examples/ex8-multi-1.yml,
where the individual preconditioner variants are split into separate YAML files and
pulled in via include. The solver configuration and per-variant iteration
counts/residuals should match the single-file version. Minor differences in reported
timings are expected.