# Building OpenFHE for Best Performance

The default build configuration of OpenFHE focuses on portability and ease of installation.
As a result, runtime performace for the default configuration is often significantly worse than for the optimal configuration.

There are three important CMake flags that affect runtime performance:
* `WITH_NATIVEOPT` allows the user to turn on/off machine-specific optimizations. By default, it is set to OFF for maximum portability of generated binaries.
* `NATIVE_SIZE` specifies the word size used internally for "small" integers. By default, it is set to 64. However, when used moduli are 28 bits or below,
it is more efficient to set it to 32.
* `WITH_OPENMP` allows the user to turn on/off multithreading using OpenMP. By default, it is set to ON, and all threads are available for OpenMP multithreading. The `OMP_NUM_THREADS` environment variable can be used to limit the number of threads available in parallel regions.

The compiler used is also important. We recommend using more recent compiler versions to achieve the best runtime performance.

## Configuration specific to DM-like schemes, such as FHEW or TFHE

For STD128* modes of the schemes within the `binfhe` module with OpenMP parallelization disabled, we recommend using the following CMake command-line configuration:

```
cmake -DNATIVE_SIZE=32 -DWITH_NATIVEOPT=ON -DCMAKE_C_COMPILER=clang-12 -DCMAKE_CXX_COMPILER=clang++-12 -DWITH_OPENMP=OFF ..
```

This configuration was used to generate the runtimes for the table in [Demystifying Bootstrapping in Fully Homomorphic Encryption](https://eprint.iacr.org/2023/149).

If OpenMP parallelization is desired, the last command-line argument should be removed. If the number of OpenMP threads is set to 1 (single-threaded execution) using the `OMP_NUM_THREADS` environment variable, then runtimes will be roughly the same as for the mode when `WITH_OPENMP` is set to OFF.

Note that the use of a 32-bit word size is recommended for 'binfhe' because all STD128* configurations in OpenFHE use moduli no higher than 28 bits.

A later version of the clang compiler can also be used.

## Configuration specific to BGV-like schemes, such as BGV, BFV, and CKKS

Typically, the default configuration for schemes in the `pke` module is only to a small degree less performant than the optimal one (in contrast to DM-like schemes). Setting `WITH_NATIVEOPT` to ON may sometimes lead to a decrease in runtime (especially when using clang).

# Multithreading Configuration using OpenMP

OpenFHE uses loop parallelization via OpenMP to speed up some lower-level (mostly polynomial) operations. From a bird's eye view, built-in OpenFHE loop parallelization is applied at the following levels:
* For many Double-CRT operations (used for BGV, BFV, and CKKS implemented using RNS in OpenFHE), loop parallelization over the number of RNS limbs is automatically applied. The biggest benefit is seen when the multiplicative depth is not small (in deeper computations). For BGV and CKKS, the number of RNS limbs is roughly the same as the multiplicative depth set by the user (it is 1 or 2 larger). In BFV, it gets more complicated, but the number of RNS limbs is still proportional to the multiplicative depth.
* A higher-level loop parallelization is employed for CKKS bootstrapping and scheme switching between CKKS and FHEW/TFHE.
* Loop parallelization is also used for all schemes during key generation (but this does not have effect on online operations).

When developing C++ applications based on OpenFHE, it is advised to use OpenMP parallelization at the application level, e.g., when independent operations on multiple ciphertexts are performed, application-level OpenMP loop parallelization can be turned on.
The scaling of performance with the number of cores in this setup can approach the "ideal" linear scaling if the dimension of the loop is comparable to the number of cores. Note that turning on OpenMP parallelization at the application level typically turns off the lower-level OpenMP loop parallelization (i.e., we do not use nested loop parallelization in OpenMP), so application-level loop parallelization should be used only when you know that the application loop dimension is higher than what is expected for built-in OpenFHE OpenMP loop parallization.

Within OpenFHE, the use of hyperthreading can lead to decreased performance so the `OMP_NUM_THREADS` environment variable should be set no higher than the number of physical cores (usually half the number of logical cores). Performance also depends heavily on the execution environment. While a dedicated server allows OpenFHE to utilize all physical cores in isolation, running on a personal laptop often introduces contention with background processes and OS tasks. In such contentious environments, using every available thread can lead to frequent context switching and cache thrashing, ultimately degrading performance. To mitigate this on a general-purpose machine, it may be beneficial to further limit `OMP_NUM_THREADS` to a number slightly less than the number of physical cores, leaving sufficient headroom for the rest of the system. Benchmarking with a varying number of threads should be performed to determine the optimal `OMP_NUM_THREADS` value for your given system and workload.

If an alternative parallelization mechanism is used, e.g., pthreads, C++11 threads, or multiprocessing, OpenMP should be turned off by setting the `WITH_OPENMP` CMake flag to OFF.

# Accelerating OpenFHE using Specialized Hardware Backends #

OpenFHE supports multiple hardware acceleration backends. Currently, one such backend has been released based on the Intel HEXL library for Intel processors with AVX-512 support.

## Notes specific to the Intel HEXL backend

The Intel HEXL backend is optimized for processors with AVX512_IFMA support, e.g., Intel IceLake Xeon processors. Note that to take advantage of AVX512_IFMA optimizations, the
small moduli should be below 50 bits. If they are larger, slower instructions are used.

For best AVX512 acceleration, we recommend building the OpenFHE HEXL variant using a recent version of clang, i.e., 12 or later (exceptions for `binfhe` discussed below). This can be done using the environment variables `CC` and `CXX`. For instance,
the user can run
```
export CC=clang-12
export CXX=clang++-12
```
before following the build instructions at https://github.com/openfheorg/openfhe-hexl. Alternatively, the environment variables can be set in `bash_profile` or similar profile configuration file.

For DM-like schemes within the `binfhe` module, later versions of clang (with the exceptions of clang-14 and clang-15) produce the best performance. The Intel HEXL Library requires 64-bit data so `NATIVE_SIZE` can not be set to 32.
