Hardware (of BSC HCA cluster):
synth-hca:ssh synth-hcabananaf3:salloc -p bananaf3 -N 1 -t 01:00:00
Software:
All appropriate libraries have to be loaded with the relevant module load commands. This is done automatically for compiling and running the benchmark via make.sh and run.sh.
- For Banana Pi:
module load llvm/EPI-development
- For RAVE emulation:
module load llvm/cross/EPI-development
module load rave/development/EPI
module load sdv_trace/development
To test different formats/implementations, you can use the class template provided here. The new class inherits properties from the Matrix_Format class defined here. A conversion function (csr_to_format) and an execution function (spmv) have to be defined for the new format.
To compile this benchmark, you need to first edit the config.sh file with the appropriate benchmark configurations that you want (number of threads, desired matrices, spmv kernels ...). Then you run make.sh in the src directory.
- When trying to run the benchmark on synth-hca (RAVE emulation), you have to set
RAVE_EMULATION=1inconfig.sh, as well asSDV_TRACING=1. - When trying to run the benchmark on Banana Pi, you have to set these variables to 0.
To run SpMV you simply execute the run.sh bash script. All the configuration needed is in the config.sh file, along with documentation comments for each option.
There are two options available, configurable in config.sh through the USE_ARTIFICIAL_MATRICES variable.
- When set to 0, the benchmark uses the matrices defined in
run.shasmatrices_validationvariable and stored invalidation_matrices. To download matrices, thefilter.pyscript can be used. - When set to 1, artificial sparse matrices are generated through the generator defined
artificial-matrix-generatorand the parameters definedmatrix_generation_parameters.
In RAVE emulation, reading files from disk (validation matrices) is very slow, so the use of artificial matrices, which are generated on-the-fly, is recommended. The parameters for the artificial matrices are the following:
rows: number of rowscols: number of columnsavg_row_size: average nonzeros per rowstd_row_size: standard deviation of nonzeros per rowdistribution: distribution of nonzeros per row (currently "normal" and "gamma" distributions are supported)placement: placement of nonzeros within row (currently "random", "diagonal" and "simple" are supported)avg_bw: bandwidth of matrix, which confines the column range that a nonzero can be placed atskew_coefficient: coefficient of skewness, which indicates the imbalance of row lengthavg_num_neigh: average number of neighboring nonzeros within row (neighbors : maximum distance 1 for column values)cross_row_sim: similarity between adjacent rows (based on column position of nonzeros)random_seed: seed for the random number generator (need to reproduce results on the same artificial matrices across different platforms)
csr_naive: The baseline, non-vectorized implementation that distributes the work by balancing the row workload across threads.csr_vec: A generalized, cross-architecture row-wise vectorization strategy. It utilizes masks to set the desired vector length, therefore it always reports 256 avg VL, but still is bound by the row length.csr_vector_riscv: A row-wise vectorization specifically tailored for RISC-V. Its average VL is directly bound by the matrix's average row size.csr_vector_riscv_bulk: A RISC-V vectorization approach that packs multiple rows together to maximize throughput during the multiplication phase. However, writing the results to the output vector relies on a serialized, per-row vectorized reduction, resulting in an average VL below the 256 maximum.sell: The Sliced ELLPACK format, configured with a slice size equal to the vector length (Slice=VL). This necessitates a local transposition of the non-zero arrays to ensure contiguous memory access during wide vector loads, achieving an average VL of 256.sell_sorted: An optimized variant of the Sliced ELLPACK format that locally sorts each thread's rows by size prior to slicing. This minimizes padding overhead while maintaining an average VL of 256.sell_c_s: The widely recognized SELL-C-σ format, adapted to RVV intrinsics rather than EPI (source here). It achieves an average VL of 256.div_rf: A custom compression format developed by CSLab (publication) that achieves a dual compression of both matrix indices and values. Operating under a COO-centric logic, it sustains an average VL of 256.
(Developer notes) Changes from default repository (SpMV-Research)
- In
make.sh, added this small snippet so that RAVE emulation enables or disables the use of VEC_LEN=256 incsr_vec.cpp,sell.cpp,sell_sorted.cppdiv_gather_coords.hfiles. Otherwise default (vec_len_default_f64) is used.
if ((RAVE_EMULATION)); then
CFLAGS+=" -D'RAVE_EMULATION=1'"
fi
- In
run.sh, added this small snippet so that library loading is performed automatically.
if ((RAVE_EMULATION)); then
export ARCH='rave'
else
export ARCH="$(uname -m)"
# export ARCH='riscv64'
fi
if [[ ${ARCH} == riscv64 || ${ARCH} == rave ]]; then
case "${ARCH}" in
riscv64)
module load llvm/EPI-development
;;
rave)
module load llvm/cross/EPI-development
module load rave/development/EPI
module load sdv_trace/development
;;
esac
fi
- In
sell_sorted.cpp, fixed a small bug that lead to segmentation fault.
long ii_e_last = ii_e;
if (ii_e * VEC_LEN > i_e){
// ii_e_last -= VEC_LEN; // this is wrong!
ii_e_last--;
}
-
In
bench_spmv.cpp, removed the check of the L3 cache (and all relevant flags inconfig.sh) because it caused a seg fault in Banana Pi for a specific format only. Additionally, removed any rapl-related counters. -
In
csr_riscv.cppchangedv_idx_rowtov_idx_col. -
In
div_kernels_rf.hput Line 405 in comments. We need to disable this quicksort call, due to wide Vector Length. -
Several changes had to be made in order to test this on a custom QEMU installation:
- In
config.sh, added variables for QEMU RISC-V emulation:OCT_QEMU_EMULATION(toggle),QEMU_BIN(emulator binary path),QEMU_LD_PREFIX(/usr/riscv64-linux-gnu), andQEMU_CPU(CPU model and vector length, e.g.max,vlen=1024ormax,vlen=16384). - In
run.sh, addedARCH='qemu'mode and executed binaries through QEMU instead of native / RAVE:"$QEMU_BIN" -L "$QEMU_LD_PREFIX" -cpu "$QEMU_CPU" "$prog" "${prog_args[@]}"
- In
make.sh, added theqemutarget architecture:- Selects GNU cross-compilers (
riscv64-linux-gnu-gcc,riscv64-linux-gnu-g++,riscv64-linux-gnu-ar). - Sets
-D'OCT_QEMU_EMULATION=1'. - Replaces Clang/EPI-specific flags (
-mcpu=avispado,-mepi, etc.) with GCC RVV 1.0 flags:-march=rv64gcv -mabi=lp64d -fno-lto -flax-vector-conversions -fno-tree-slp-vectorize. - Adds
-L /usr/riscv64-linux-gnu/libtoLDFLAGS.
- Selects GNU cross-compilers (
- In
csr_vec.cpp,sell.cpp,sell_sorted.cpp, anddiv_gather_coords.h, groupedOCT_QEMU_EMULATIONwithRAVE_EMULATIONto select wide vector lengths (VEC_LEN=256for double,512for float). - In the
sell-C-sbuild system:- In
Makefile_in, forwardedARCH,DOUBLE,CC, andARto the sub-make instead of hardcodingCC=clang LLVM=clang. - In
sell-C-s/Makefile, made compiler and archiver overridable (CC ?= clang,LLVM ?= $(CC)). - In
sell-C-s/setup/riscv.mk, added anifeq ($(ARCH),qemu)configuration targetingriscv64-linux-gnu-gccand standard RVV flags without EPI/Clang-only attributes.
- In
- In