# Advanced installation {#sec:advancedinstallation}

## LAN and cluster licenses

If you purchased the LAN or cluster version of RESCU, you simply need to
run `nano_infogen_linux` on a login node to generate `dat.inf`. Then
upload it on the [client portal](https://portal.nanoacademic.com/) and
follow the instructions there to complete the activation.

:::: note
::: title
Note
:::

If you are using RESCU \< 2.7.0, you need to point the licenser to the
installation node as follows. The information in `licenseNumber.txt` and
`license.dat` will only unlock RESCU on the node on which it is
installed. To avail RESCU on the entire cluster, follow these
instructions. Use a login or gateway node to install RESCU. The node's
IP address should be in the ARP cache of all worker nodes (it most
likely is already, so you likely need not worry about this). Then create
the file `ip.dat` in the `license` directory. Add the *inet* addresses
of the gateway node in `ip.dat`, one by line. For instance,

``` shell
0.150.188.70
0.16.205.3
0.150.188.73
0.20.205.3
```

The inet addresses can be listed using the command

``` shell
ifconfig | grep inet
inet addr:192.168.1.29 Bcast:192.168.1.255  Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe70:e3f5/64 Scope:Link
```

What you seek here is `192.168.1.29`.
::::

## Dependencies

RESCU is mostly written in MATLAB, and hence it does not need to be
compiled. Additional functionalities require external programs and
libraries which must be compiled however. MATLAB calls these procedures
through MEX files, which must be compiled. For one, the atomic orbital
calculations require such MEX files. In this instance, external
libraries need not be installed and linked, and hence the `makeRESCU`
function deals with it by itself. Otherwise, `makeRESCU` must be
provided with the library paths. The purpose of the present document is
to demonstrates how to build and install these external libraries and
MEX files.

The following is a list of functionalities that require an installation:

-   Calls certain routines in the
    [CUDA](https://developer.nvidia.com/cuda-zone) library. This is not
    required to run RESCU using GPUs, but it may improve performance in
    certain calculations;
-   A set of functions that call
    [LibXC](https://www.tddft.org/programs/Libxc) routines. LibXC is an
    exchange-correlation functional library including hundreds of
    correlation, exchange and kinetic energy functionals.
-   A set of functions that call MPI and
    [ScaLAPACK](http://www.netlib.org/scalapack/) routines. MPI, BLAS,
    LAPACK and ScaLAPACK libraries are required;

The function `makeRESCU` compiles and installs a package listed above
when provided with one of the following flags: `-gpu`, `-libxc`, `-smi`.
For example, type the following command to build the SMI package:

``` matlab
makeRESCU -agreeToLicense y -smi libpaths.m
```

The last command should compile the SMI interface if the `mpicc` and
`mpif90` compilers are found in the `PATH` environment variable. In
addition, the file `libpaths.m` must contain variables providing the
paths of certain external libraries. Specific instructions for each
package lie below.

## Pre-existing MPI installation {#sec:module}

Many platforms now have several libraries preinstalled. Loading them is
made easy with modern package managers via the `module` command for
instance. Then the OpenMPI installations can be listed as

``` shell
username@computer:~$ module spider openmpi

--------------------------------------------------------------
  openmpi:
--------------------------------------------------------------
    Description:
      An open source Message Passing Interface implementation

     Versions:
        openmpi/3.1.3
        openmpi/4.0.1
        openmpi/4.0.3
        openmpi/4.1.1
```

OpenMPI is then simply loaded typing

``` shell
module load openmpi/4.0.3
```

Similarly, one could seek and load LibXC, BLAS, LAPACK and ScaLAPACK as
follows

``` shell
module load openmpi/4.0.3 libxc/4.3.4 openblas/0.3.9 scalapack/2.1.0
```

Note that OpenBLAS contains both BLAS and LAPACK. Then, one only need to
specify how to link in the file `libpaths.m`

``` shell
xc        = '/software/2020/avx512/Compiler/gcc9/libxc/4.3.4';
scalapack = '-lscalapack';
blas      = '-lopenblas';
lapack    = '';
```

## `spack` installation {#sec:spack}

If you need to install OpenMPI and other libraries yourself, the
simplest way is probably to use a package manager like `spack`. This
section shows how to install the build tools and dependencies using the
`spack` package manager.

Get spack from the github repository typing

``` shell
git clone https://github.com/spack/spack.git
```

and add it to your environment as follows

``` shell
source spack/share/spack/setup-env.sh
```

You may add the last command to your `.bashrc` to make this permanent.

Copy `spack.yaml` file found in `documentation` and activate the package
extension.

``` shell
cd $RESCU_DIR
cp documentation/spack.yaml .
spacktivate .
```

`GCC` and `OpenMPI` are usually already installed on computing
platforms. You may tell `spack` to use preinstalled compilers and
libraries as follows

``` shell
spack find compiler
spack external find OpenMPI
```

Next, concretize the environment and let `spack` build the dependencies
as follows

``` shell
spack concretize -f
spack install -j8
```

You may adjust the build parallelism using the `-j` option (here I use 8
cores). When using **rescu, just activate the environment with**

``` shell
spacktivate .
```

If something goes sideways, try to find a local version (e.g. with
`perl`) and reconcretize

``` shell
spack external find perl
spack concretize -f
```

Finally, define the file `libpaths.m` as at the end of section
[1.3](#sec:module) to specify how to link the libraries.

## SMI: ScaLAPACK wrapper

In order to exploit efficiently hundreds of processes, RESCU calls
[ScaLAPACK](http://www.netlib.org/scalapack/) in heavy computational
operations via `smi_proxy.c`. If ScaLAPACK is already installed on your
computer, you may skip the following subsection. Otherwise, here are
some instructions to download, configure, compile, install and
ScaLAPACK.

### Building the ScaLAPACK library {#sec:scalib}

Download the ScaLAPACK source from
[Nanoacademic](https://storage.googleapis.com/rescumat/scalapack-2.1.0.tar.gz).
It has a couple modifications to make it work with MATLAB and handle
large matrix redistribution operations.

``` shell
wget https://storage.googleapis.com/rescumat/scalapack-2.1.0.tar.gz
tar -zxf scalapack-2.1.0.tgz
cd scalapack-2.1.0
```

Then jumps in the ScaLAPACK directory and copy the `SLmake.inc.example`
file to `SLmake.inc`.

``` shell
cd scalapack-2.1.0
cp SLmake.inc.example SLmake.inc
```

MATLAB requires *position independent code* libraries, which means
compiled with the `PIC` option. Make sure to add `-fPIC` to the
following flags in `SLmake.inc`

``` shell
NOOPT   = -O0 -fPIC
FCFLAGS = -O3 -fPIC
CCFLAGS = -O3 -fPIC
CFLAGS  = -O3 -fPIC
```

Then make the library by typing `make lib` and install it by copying
`libscalapack.a` in the appropriate location.

### ScaLAPACK interface

Before compiling the ScaLAPACK interface, paths to BLAS, LAPACK and
ScaLAPACK libraries must be defined. If no BLAS or LAPACK library is
installed, we suggest that you install
[OpenBLAS](https://www.openblas.net/), which contains both. As mentioned
above, this is done in a MATLAB script. Create an M-file and define
three variables called `blas`, `lapack`, `scalapack` containing the
paths to the BLAS, LAPACK and ScaLAPACK libraries respectively. For
example,

``` shell
cat > /tmp/installpaths.m <<EOF
scalapack = '/usr/lib/libscalapack.a';
lapack = '/usr/lib/libopenblas.a';
blas = '/usr/lib/libopenblas.a';
EOF
```

Additional flags may be required depending on the compiler. Specify them
using the variable `CFLAGS` in the M-file (e.g. `CFLAGS = ’-ifcore’`).
Finally, compile `smi_proxy.c` as follows

``` matlab
makeRESCU -smi /tmp/installpaths.m
```

## Compiling the LibXC wrapper

A few functionals have been implemented in RESCU (LDA_X, LDA_C_PW92,
GGA_XC_PBE, MGGA_X_TB09). More functionals are available via the
functional library LibXC. LibXC is part of the Easybuild toolchain, and
hence you can skip the installation of LibXC if you executed
`install.sh`. Both the Linux system and Windows system procedures to
compile and install LibXC are presented in this section.

### Linux

To install [LibXC](https://www.tddft.org/programs/Libxc) on a Linux
system, follow the following instructions. [Download
LibXC](https://www.tddft.org/programs/libxc/download/) (version 4.0 and
above for full functionality) and decompress the tarball.

``` shell
tar -xzf libxc-4.3.4.tar.gz
```

Enter the LibXC directory, configure, build and install the library.

``` shell
cd libxc-4.3.4
./configure --prefix=PATH/TO/LIBXC --disable-shared CFLAGS="-O3 -fPIC"
make && make check && make install
```

Note that MATLAB requires *position independent code* libraries, which
means compiled with the `PIC` option. We also suggest building and
linking static libraries in MATLAB, and hence the `--disable-shared`
configure option.

The LibXC library is now installed. Create an M-file and define the
variable `xc` as the path to the root of the LibXC library installation
(the path specified using the `--prefix` configure option).

``` shell
cat > /tmp/installpaths.m <<EOF
xc = '/usr/software/libxc-4.3.4';
EOF
```

Then invoke makeRESCU to compile the LibXC package.

``` matlab
makeRESCU -libxc /tmp/installpaths.m
```

The file `libxc.mexa64` will be created upon successful compilation.

### Windows

To install LibXC on Windows, follow this procedure. To begin with,
complete MathWorks' instructions to install the
[MinGW-w64](http://www.mingw.org/) Compiler. To install the compiler,
use the Add-Ons menu, select the Features box and search for MinGW. This
may not be possible if your version of MATLAB is anterior to 2016, but
don't worry there is [a
way](https://www.mathworks.com/matlabcentral/fileexchange/52848-matlab-support-for-mingw-w64-c-c-compiler)
nevertheless.

Install the MinGW-w64 Compilers through [MSYS2](https://www.msys2.org/).
The MSYS2 project is hosted on [Github](https://github.com/msys2).
Download and execute the installer; MSYS2 will install itself
automatically. It is advised to use the default paths.

Open an MSYS2 terminal using the command `C:\\msys64\\msys2_shell.cmd`
and update the MSYS2 packages as follows

``` shell
pacman -Sy pacman
pacman -Syu
pacman -Su
```

Pacman is an Arch Linux packager manager, similar to yum or apt-get on
Linux. You can search a package using the command `pacman -Ss` and
install a package using the command `pacman -S`. Now, install the
following packages

``` shell
pacman -S --needed base-devel mingw-w64-x86_64-toolchain
pacman -S --noconfirm autoconf autoconf-archive automake make libtool
```

Download [LibXC](https://tddft.org/programs/libxc/download/) and
decompress the tarball. From an MSYS2 shell, configure and make the
library.

``` shell
autoreconf -f -i -I m4
./configure --prefix '/c/lib/libxc-4.3.4' --disable-shared CFLAGS="-O3 -fPIC"
make && make install
```

If something goes wrong, you may want to refer to the following [Stack
Overflow
ticket](https://stackoverflow.com/questions/72019820/how-to-fix-configure-in-msys2/72020333)
, which contains instructions on how to build LibXC with `cmake`.

If you use `make check`, note that it generally fails on Windows because
round-off errors give slightly different results on Linux, but the error
is negligible in practice. MATLAB will look for libraries with the
extension `.lib`. The libraries in `/c/lib/libxc-4.3.4` must thus be
renamed accordingly.

``` shell
cd /c/lib/libxc-4.3.4
cp libxc.a libxc.lib
```

The LibXC library is now installed.

If you were able to install the MinGW-w64 Compiler MATLAB application,
you may execute the following instructions. Create an m-file (`*.m`) and
define the variable `xc` as the path to the root of the LibXC library
(the path specified using the `--prefix` option). Then invoke makeRESCU
to compile the LibXC package.

``` matlab
makeRESCU -libxc /tmp/installpaths.m
```

The file `libxc.mexw64` will be created upon successful compilation.

#### Unsuccessful mex compilation

If `makeRESCU` fails, you may have to compile `libxc.mexw64` from the
command line. Open an MSYS2 terminal (64-bit). Add `mex` and `gcc` to
your path as follows

``` shell
export PATH=/c/Program\ Files/MATLAB/R2017b/bin/win64:$PATH
export PATH=/c/msys64/mingw64/bin:$PATH
```

Then change directory to `rescu-x.x.x/src` and write

``` shell
mex CC=/c/msys64/mingw64/bin/gcc libxc4.c -output libxc \
-I/c/lib/libxc-4.3.4/include -L/c/lib/libxc-4.3.4/lib -lxc
```

`CC` specifies the path MSYS2\'s GCC compiler. `libxc4.c` is the LibXC-4
wrapper, replace by `libxc.c` if using LibXC-3. `-I` and `-L` are the
paths to the LibXC header files and libraries.

#### Unsuccessful MinGW-w64 install

If you were not able to install the MinGW-w64 Compiler MATLAB
application, you may execute the following instructions. To compile the
mex file, create the file mexopts.bat as follows (taken from [Dr Bogdan
Roman](http://www.damtp.cam.ac.uk/research/afha/bogdan/))

``` shell
set MINGWPATH=c:\msys64\mingw64
set PATH=%MINGWPATH%\bin;%PATH%
set PRELINK_CMDS=echo.>%TEMP%\mexstaticlibs
rem set PRELINK_CMDS1=gcc -print-file-name=libwinpthread.a >> %TEMP%\mexstaticlibs
rem set PRELINK_CMDS2=gcc -print-file-name=libquadmath.a >> %TEMP%\mexstaticlibs
rem set PRELINK_CMDS3=...
rem set POSTLINK_CMDS1=upx -9 "%OUTDIR%%MEX_NAME%%MEX_EXT%"
rem set POSTLINK_CMDS2=...
set COMPILER=g++
set COMPFLAGS=-c -I"%MATLAB%\extern\include" -DMATLAB_MEX_FILE
set OPTIMFLAGS=-O3 -funroll-loops -DNDEBUG
set DEBUGFLAGS=-g
set NAME_OBJECT=-o
set LINKER=g++
set LINKFLAGS=-shared -static-libstdc++ -static-libgcc -L"%MATLAB%\bin\win64" -L"%MATLAB%\extern\lib\win64\microsoft" -lmex -lmx -leng -lmat -lmwlapack -lmwblas
set LINKFLAGSPOST=@%TEMP%\mexstaticlibs
set NAME_OUTPUT=-o "%OUTDIR%%MEX_NAME%%MEX_EXT%"
```

Copy the file to
`%USERPROFILE%\AppData\Roaming\Mathworks\MATLAB\R2015a`, for instance.
Then open a MinGW-w64 terminal and add `mex` to your path, e.g.

``` shell
export PATH=$PATH:/c/Program\ Files/MATLAB/R2015a/bin/win64
```

and compile the mex file typing

``` shell
mex libxc.c -I/c/lib/libxc-4.3.4/include -L/c/lib/libxc-4.3.4/lib -lxc
```

Again, the file `libxc.mexw64` will be created upon successful
compilation.

## Compiling the GPU wrapper

RESCU accelerates certain calculations using GPUs. Doing so requires a
[CUDA](https://developer.nvidia.com/cuda-zone) installation and
compiling a few MEX files. Here is how to do it. Create an M-file in
which the variable `cuda` is defined as the path to your CUDA
installation. For example,

``` shell
cat > /tmp/installpaths.m <<EOF
cuda = '/usr/local/cuda-9.2';
EOF
```

Then invoke `makeRESCU` with the GPU option as follows

``` matlab
makeRESCU -gpu /tmp/installpaths.m
```

## Compiling and using RESCU as a stand-alone executable {#ssec:compile}

With the help of external libraries like
[OpenMPI](https://www.open-mpi.org/) and ScaLAPACK, RESCU can use
hundreds, even thousands of cores. However, this may require a
prohibitive number of MATLAB licenses. This problem is circumvented by
compiling RESCU into a standalone program which do not require any
license.

Note that MATLAB and its Compiler toolbox are required. Install the
optional packages (e.g. LibXC, SMI) that you intend to use. Call the
function `compileRESCU` in MATLAB's command window to compile RESCU.

``` shell
matlab -r "compileRESCU('src','bin')"
```

The first argument specifies the source directory. It should contain the
source code and all relevant MEX files. The second argument is the
directory where the RESCU application is installed.

To execute RESCU, the freely available [MATLAB Compiler
Runtime](https://www.mathworks.com/products/compiler/matlab-runtime.html)
(MCR) is required. Download and install the
[MCR](https://www.mathworks.com/products/compiler/matlab-runtime.html)
that matches the MATLAB version that you used to compile RESCU. Next,
follow the MCR manual to set up your environment properly. On Windows
and Mac, you likely have nothing to do. On linux, you must modify your
environment as follows.

``` shell
export MCR_CACHE_ROOT=/scratch/johndoe
MCRROOT=/software/MCR/v99
export LD_LIBRARY_PATH=$MCRROOT/runtime/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/bin/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/sys/os/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/extern/bin/glnxa64:$LD_LIBRARY_PATH
```

The variable `MCR_CACHE_ROOT` is optional but recommended. It tells the
MCR where to store its cache files which may grow quite large. Finally,
add the RESCU standalone application location to the `PATH`. Suppose
that RESCU is found in `/home/software`. On Linux you may add RESCU to
your `PATH` typing

``` shell
export PATH=/home/software:$PATH
```

On Windows, go to System$>$System Properties$>$Environment Variables.
Select `PATH` and click on the Modify button. Then append
`:/home/software`.
