Octopus

Sequoia/Vulcan (IBM Blue Gene/Q)

Stampede

Edison

Cori

Lonestar

Lawrencium

Curie

PFFT 1.0.5 and BigDFT 1.6.0

The Curie supercomputer, owned by GENCI and operated into the TGCC by CEA, is the first French Tier0 system open to scientists through the French participation into the PRACE research infrastructure.

Curie is offering 3 different fractions of x86-64 computing resources for addressing a wide range of scientific challenges and offering an aggregate peak performance of 2 PetaFlops.

General information: https://www-hpc.cea.fr/en/complexe/tgcc-curie.htm

Compilation with PFFT version 1.0.5 and LibISF taken from BigDFT 1.6.0 (previously compiled):

#!/bin/bash

module load c/intel
module load fortran/intel
module load bullxmpi
module load mkl
module load gsl/1.14

WPREFIX=$WORKDIR/software
PREFIX=$HOME/software
export ISF_HOME="$WORKDIR/software/libisf"

export CC=mpicc
export CFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$ISF_HOME/include"
export FC=mpif90
export FCFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$PREFIX/fftw/3.3/include -I$ISF_HOME/include"

export LIBS_BLAS="${MKL_LIBS}"
export LIBS_FFT="-Wl,--start-group -L$PREFIX/pfft/1.0.5/lib -L$PREFIX/fftw/3.3/lib -lpfft -lfftw3 -lfftw3_mpi -Wl,--end-group"
export LDFLAGS=$LDFLAGS" -L$ISF_HOME/lib -lisfsolver -lrt"

../configure --prefix=$WPREFIX/octopus/superciliosus_pfft \
--disable-openmp --with-libxc-prefix=$WPREFIX/libxc/2.0.x \
--disable-gdlib --with-gsl-prefix=/usr/local/gsl-1.14 \
--enable-mpi --enable-newuoa \
--with-pfft-prefix=$PREFIX/pfft/1.0.5

BigDFT was configured like this:

 #!/bin/bash

 export FC=mpif90

 ../configure --with-ext-linalg="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lm" \
   --with-ext-linalg-path="-L/usr/local/Intel_compilers/c/composer_xe_2011_sp1.7.256/mkl/lib/intel64" \
   --prefix=$WORKDIR/software/bigdft

PFFT 1.0.7 and BigDFT 1.7.0

 #!/bin/bash

module load c/intel
module load fortran/intel
module load bullxmpi
module load mkl
module load gsl/1.14

WPREFIX=$WORKDIR/poisson/software
WLPREFIX=$WORKDIR/poisson/local
export ISF_HOME="$WLPREFIX/libisf"
export FFTW_HOME="$WLPREFIX/fftw-3.3.3"
export PFFT_HOME="$WLPREFIX/pfft-1.0.7-alpha"

export CC=mpicc
export CFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$ISF_HOME/include"
export FC=mpif90
export FCFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$FFTW_HOME/include -I$ISF_HOME/include"

export LIBS_BLAS="${MKL_LIBS}"
export LIBS_FFT="-Wl,--start-group -L$PFFT_HOME/lib -L$FFTW_HOME/lib -lpfft -lfftw3 -lfftw3_mpi -Wl,--end-group"

export LDFLAGS=" -L/usr/lib64 -L$ISF_HOME/lib -lPSolver-1 -lrt"
export LD_LIBRARY_PATH=$ISF_HOME/lib:$LD_LIBRARY_PATH

../configure --prefix=$WPREFIX/octopus/superciliosus_pfft \
--disable-openmp --with-libxc-prefix=$WPREFIX/libxc/11066 \
--disable-gdlib --with-gsl-prefix=/usr/local/gsl-1.14 \
--enable-mpi --enable-newuoa \
--with-pfft-prefix=$PFFT_HOME

LibISF compilation:

 #!/bin/bash

 module load c/intel
 module load fortran/intel
 module load bullxmpi
 module load mkl
 module load gsl/1.14

 export FC=mpif90

 ../configure --with-ext-linalg="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lm" \
    --with-ext-linalg-path="-L/usr/local/Intel_compilers/c/composer_xe_2011_sp1.7.256/mkl/lib/intel64" \
    --prefix=$WORKDIR/poisson/local/libisf \
    --disable-libbigdft --disable-binaries --enable-dynamic-libraries

FFTW 3.3.3 compilation script is in https://www-user.tu-chemnitz.de/~mpip/software/install_fftw-3.3.3_gcc.sh and it is changed as follow:

 < myprefix=$HOME/local
 ---
 > module load c/intel
 > module load fortran/intel
 > module load bullxmpi
 > module load mkl
 > module load gsl/1.14
 >
 > myprefix=$WORKDIR/poisson/local
 23c29,30
 < wget https://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
 ---
 > - wget https://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
 > cp ../fftw-$FFTW_VERSION.tar.gz .
 876c883
 < ./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-shared \
 ---
 > ./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-shared --disable-shared \

PFFTW 1.0.7 compilation script is in https://www-user.tu-chemnitz.de/~mpip/software/install_pfft-1.0.7-alpha_gcc.sh and it is changed as follow:

 < myprefix=$HOME/local
 ---
 >
 > module load c/intel
 > module load fortran/intel
 > module load bullxmpi
 > module load mkl
 > module load gsl/1.14
 >
 > myprefix=$WORKDIR/poisson/local
 24c31,32
 < wget https://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
 ---
 > -wget https://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
 > cp ../pfft-$PFFT_VERSION.tar.gz .

Fermi/Juqueen

IBM Blue Gene/Q, Cineca, Italy

General information: https://www.hpc.cineca.it/content/ibm-fermi-user-guide

The exactly the same script has been tested in the Juqueen, Forschungszentrum Jülich, Jülich Supercomputing Centre (JSC), Jülich, Germany. General information: https://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/JUQUEEN_node.html

#/bin/bash

module load gsl
module load lapack
module load blacs
module load scalapack

export PFFT_HOME="$HOME/local/pfft-threads-1.0.5-alpha"
export FFTW3_HOME="$HOME/local/fftw-threads-3.3.2"
export LD_LIBRARY_PATH=$PFFT_HOME/lib:$FFTW3_HOME/lib:$LD_LIBRARY_PATH

export FC_INTEGER_SIZE=8
export CC_FORTRAN_INT=int
export CC=mpicc
export CFLAGS="-g -Wl,-relax -O3 -I$PFFT_HOME/include -I$FFTW3_HOME/include"
export FC=mpif90
export FCFLAGS=$CFLAGS" -qxlf90=autodealloc -qessl -qsmp=omp "
export LIBS_BLAS="-lesslsmpbg -L/opt/ibmmath/essl/4.4/lib -lesslbg"
export LIBS_LAPACK="-L$LAPACK_LIB -llapack"

export LIBS_FFT="-L$FFTW3_HOME/lib/ -lfftw3_mpi -lfftw3 -lm"
echo "GSL DIR = $GSL_HOME "
echo "PFFT DIR = $PFFT_HOME "
export LDFLAGS=$LDFLAGS" -R/opt/ibmcmp/lib/bg/bglib \
-L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/powerpc-bgp-linux/lib \
-L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2 \
-L/opt/ibmcmp/xlf/bg/11.1/bglib \
-L/opt/ibmcmp/xlmass/bg/4.4/bglib \
-L/opt/ibmcmp/xlsmp/bg/1.7/bglib \
-L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/runtime/SPI \
-L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/lib \
-L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/lib \
-L/usr/local/zlib/v1.2.3/lib \
-L/bgsys/drivers/ppcfloor/runtime/SPI \
-L$PFFT_HOME/lib -L$FFTW3_HOME/lib \
-lpfft -lfftw3_mpi -lfftw3 -lm \
-ldl -lxl -lxlopt -lrt -lpthread"

SVN_VERS=$(svn info .. | grep Revision | awk '{print $2}')
echo $SVN_VERS

../configure --prefix=$HOME/software/octopus_omp \
--with-libxc-prefix=$HOME/software/libxc_new \
--with-fft-lib="$FFTW3_HOME/lib/libfftw3.a  $FFTW3_HOME/lib/libfftw3_omp.a" \
--host=powerpc32-unknown-linux-gnu \
--build=powerpc64-unknown-linux-gnu \
--disable-gdlib \
--disable-f90-forall \
--with-blas="-L/opt/ibmmath/lib64 -lesslbg" \
--with-lapack="-L$LAPACK_LIB -llapack" \
--with-gsl-prefix=$GSL_HOME \
--enable-mpi --enable-openmp

To build FFTW3. Small changes are made to the script taken from M. Pippig web site https://www-user.tu-chemnitz.de/~mpip/software.php?lang=en It has a patch and is build with MPI and multithreaded (multithreaded may not work in Juqueen):

 #!/bin/sh -e

 myprefix=$HOME/local
 FFTW_VERSION="3.3.2"
 INSTALLDIR="$myprefix/fftw-threads-${FFTW_VERSION}"
 TMP="tmp-mine-fftw-${FFTW_VERSION}"

 # bash check if directory exists
 if [ -d $TMP ]; then
         echo "Directory $TMP already exists. Delete it? (y/n)"
 	read answer
 	if [ ${answer} = "y" ]; then
 		rm -rf $TMP
 	else
 		echo "Program aborted."
 		exit 1
 	fi
  fi

 mkdir $TMP && cd $TMP

 wget https://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
 gzip -dc fftw-$FFTW_VERSION.tar.gz | tar xvf -
 cd fftw-$FFTW_VERSION

 # fix bug in fftw alltoall routine that causes deadlocks
 patch -b -p0 <<\EOF
 --- mpi/transpose-alltoall.c
 +++ mpi/transpose-alltoall.c
 @@ -223,8 +223,8 @@
 	  sbo[pe] = (int) (pe * (b * p->tblock) * vn);
 	  rbs[pe] = (int) (db * bt * vn);
 	  rbo[pe] = (int) (pe * (p->block * bt) * vn);
 -	  if (sbs[pe] != (b * p->tblock) * vn
 -	      || rbs[pe] != (p->block * bt) * vn)
 +	  if (dbt != p->tblock
 +	      || db != p->block)
 	       equal_blocks = 0;
      }
      pln->send_block_sizes = sbs;
 EOF

 ./configure --build=powerpc64-bgq-linux-gnu --host=powerpc64-bgq-linux \
 --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-threads --disable-shared \
 CC=mpixlc_r F77=mpixlf90_r MPICC=mpixlc_r MPICXX=mpixlcxx_r MPIFC=mpixlf90_r MPILIBS=" " \
 CFLAGS='-O3 -g' \
 FCFLAGS='-O3 -g' \

 make -j 4
 make install

To build PFFT, also taken from https://www-user.tu-chemnitz.de/~mpip/software.php?lang=en

 #!/bin/sh -e

 myprefix=$HOME/local
 PFFT_VERSION=1.0.5-alpha
 FFTW_VERSION=3.3.2
 INSTDIR=$myprefix/pfft-threads-$PFFT_VERSION
 FFTWDIR=$myprefix/fftw-threads-$FFTW_VERSION
 TMP="tmp-mine-pfft-$PFFT_VERSION"

 # bash check if directory exists
 if [ -d $TMP ]; then
         echo "Directory $TMP already exists. Delete it? (y/n)"
 	read answer
 	if [ ${answer} = "y" ]; then
 		rm -rf $TMP
 	else
 		echo "Program aborted."
 		exit 1
 	fi
 fi

 mkdir $TMP && cd $TMP

 wget https://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
 gzip -dc pfft-$PFFT_VERSION.tar.gz | tar xvf -
 cd pfft-$PFFT_VERSION
 ./configure --build=powerpc64-bgq-linux-gnu --host=powerpc64-bgq-linux \
 --prefix=$INSTDIR --with-fftw3=$FFTWDIR \
 MPICC=mpif90 MPICXX=mpicxx MPIFC=mpif90 CC=mpicc FC=mpif90 \
 CFLAGS='-O3 -g -qmaxmem=-1' \
 FCFLAGS='-O3 -g -qmaxmem=-1' \
 --disable-shared

 make -j 4
 make install

Jugene

MareNostrum II

MareNostrum III

Corvo

Corvo is the computational cluster of the Nano-bio Spectroscopy Group. It consists of 1504 processor cores and 5284 GiB of RAM connected by an Infiniband network.

Script to build LIBFM library included in the Scafacos library (modified Scafacos svn version 1920, this is valid since version 9887 of Octopus):

 #!/bin/bash
 ./configure --enable-fcs-solvers=fmm,direct --enable-fcs-fmm-comm=armci \
   --enable-fcs-fmm-unrolled --enable-fcs-fmm-max-mpol=40 \
   CFLAGS=-O3 CXXFLAGS=-O3 FCFLAGS=-O3 --disable-doc CXX=mpic++ CC=mpicc \
   FC=mpif90 --prefix=$HOME/Software/scafacos \
   LDFLAGS=-L/opt/scalapack/1.8.0/lib -L/opt/blacs/1.1/lib \
   -L/opt/gotoblas2/1.08/lib -L/opt/netcdf/4.0.1/lib -L/opt/lapack/3.2/lib \
   -L/opt/etsf_io/1.0.2/lib --enable-fcs-int=int --enable-fcs-float=double \
   --enable-fcs-integer=integer --enable-fcs-real=real*8

 make
 make install

The scripts to compile FFTW 3.3.2 and PFFT 1.0.5

 #!/bin/sh -e

 myprefix=$HOME/local
 FFTW_VERSION="3.3.2"
 INSTALLDIR="$myprefix/fftw/${FFTW_VERSION}"
 TMP="tmp-fftw-${FFTW_VERSION}"

 # bash check if directory exists
 if [ -d $TMP ]; then
        echo "Directory $TMP already exists. Delete it? (y/n)"
        read answer
        if [ ${answer} = "y" ]; then
                rm -rf $TMP
        else
                echo "Program aborted."
                exit 1
        fi
 fi

 mkdir $TMP && cd $TMP

 wget https://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
 gzip -dc fftw-$FFTW_VERSION.tar.gz | tar xvf -
 cd fftw-$FFTW_VERSION

 # fix bug in fftw alltoall routine that causes deadlocks
 patch -b -p0 <<\EOF
 --- mpi/transpose-alltoall.c
 +++ mpi/transpose-alltoall.c
 @@ -223,8 +223,8 @@
           sbo[pe] = (int) (pe * (b * p->tblock) * vn);
           rbs[pe] = (int) (db * bt * vn);
           rbo[pe] = (int) (pe * (p->block * bt) * vn);
 -         if (sbs[pe] != (b * p->tblock) * vn
 -             || rbs[pe] != (p->block * bt) * vn)
 +         if (dbt != p->tblock
 +             || db != p->block)
               equal_blocks = 0;
      }
      pln->send_block_sizes = sbs;
 EOF

 ./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-threads --disable-shared \
 CC="mpicc" F77="mpif90" MPICC="mpicc" \
 CFLAGS="-O3" FFLAGS="-O3" \
 MPILIBS=" "

 make -j 4
 make install

 #!/bin/sh -e

 myprefix=/home/local
 PFFT_VERSION=1.0.5-alpha
 FFTW_VERSION=3.3.2
 INSTDIR=$myprefix/pfft/$PFFT_VERSION
 FFTWDIR=$myprefix/fftw/$FFTW_VERSION
 TMP="tmp-pfft-$PFFT_VERSION"
 module unload fftw/3.2.2

 # bash check if directory exists
 if [ -d $TMP ]; then
        echo "Directory $TMP already exists. Delete it? (y/n)"
        read answer
        if [ ${answer} = "y" ]; then
                rm -rf $TMP
        else
                echo "Program aborted."
                exit 1
        fi
 fi

 mkdir $TMP && cd $TMP
 export CC=mpicc
 export FC=mpif90

 wget https://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
 gzip -dc pfft-$PFFT_VERSION.tar.gz | tar xvf -
 cd pfft-$PFFT_VERSION
 ./configure --prefix=$INSTDIR --with-fftw3=$FFTWDIR --disable-shared

 make install

This is the script to compile Octopus with PFFT and FMM support (through Scafacos library):

#!/bin/bash
module add gcc
module add ifort
module add gsl
module add etsf_io
module add lapack
module add netcdf
module add gotoblas2
module add mpibull2
module add blacs
module add scalapack

export CC=mpicc
export FC="mpif90"

# It is important that "/opt/intel/Compiler//11.1/038/bin/intel64/" not to be in the PATH
export PATH=/opt/mpi/mpibull2-1.3.9-14.s/bin:/opt/netcdf/4.0.1/bin:/opt/etsf_io/1.0.2/bin:/opt/gsl//1.13/bin:/opt/intel/composer_xe_2011_sp1.10.319//bin/intel64/:/opt/gcc//4.4/bin:/opt/slurm/bin:/usr/kerberos/bin:/opt/cuda//bin:/usr/local/bin:/bin:/usr/bin

export LIBFM_HOME="$HOME/Software/scafacos"
export PFFT_HOME="$HOME/local/pfft-1.0.5-alpha"
export FFTW3_HOME="$HOME/local/fftw-3.3.2"
export LIBS_PFFT="-L$PFFT_HOME/lib -L$FFTW3_HOME/lib -lpfft -lfftw3_mpi -lfftw3"
export CFLAGS=" -I$PFFT_HOME/include -I$FFTW3_HOME/include -I$LIBFM_HOME/include"
export LDFLAGS=$LDFLAGS" -L$LIBFM_HOME/lib -L$PFFT_HOME/lib -L$FFTW3_HOME/lib -lpfft -lfftw3_mpi -lfftw3 -lm "
export FCFLAGS=$CFLAGS
export CPPFLAGS=" -I$LIBFM_HOME/include "
export LD_LIBRARY_PATH=$LIBFM_HOME/lib:$LD_LIBRARY_PATH

../configure --enable-mpi --with-libxc-prefix=$HOME/Software/libxc_9882 \
--prefix=$HOME/Software/octopus \
--with-blacs=/opt/blacs/1.1/lib/libblacs.a --with-scalapack \
--with-libfm="-L$LIBFM_HOME/lib -lfcs4fortran -lfcs -lfcs_direct -lfcs_fmm -lfcs_near -lfcs_gridsort -lfcs_common" \
--disable-openmp

LaPalma

LaPalma is a supercomputer of the IAC. It is a part of the old MareNostrum II computer. It was not possible to compile with XL compilers, so GNU 4.6.1 version is used.

Previously to Octopus, Libxc (2.2), FFTW (3.3.4), GSL (1.16) and OpenBLAS (0.2.13) were required to compile.

GSL

 export LD_LIBRARY_PATH=/usr/lib:/gpfs/apps/MPICH2/slurm-2.5.6/64/lib/:/gpfs/apps/GCC/4.9.2/lib64/
 export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.9.2/bin/gcc"
 export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.9.2/bin/gfortran"

 ./configure --prefix=$HOME/local/gsl/gnu --enable-static --disable-shared

OpenBLAS

 make PREFIX=$HOME/local/openblas/4.6.1 BINARY=64 CC=/gpfs/apps/GCC/4.6.1/bin/gcc FC=/gpfs/apps/GCC/4.6.1/bin/gfortran

Libxc

 export PATH=/gpfs/apps/MPICH2/mx/default/64/bin:$PATH
 export LD_LIBRARY_PATH=/gpfs/apps/MPICH2/mx/default/64/lib:/gpfs/apps/MPICH2/slurm/64/lib

 export LD_LIBRARY_PATH=/gpfs/apps/GCC/4.6.1/lib64/:$HOME/local/openblas/lib:$LD_LIBRARY_PATH
 export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.6.1/bin/gcc"
 export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.6.1/bin/gfortran"

 export CFLAGS="-m64 -O3"
 export FCFLAGS=$CFLAGS

 ./configure --prefix=$HOME/local/libxc/gnu/4.6.1/2.2

Octopus

 #!/bin/bash
export PATH=/gpfs/apps/MPICH2/mx/default/64/bin:$PATH
export LD_LIBRARY_PATH=/gpfs/apps/MPICH2/mx/default/64/lib:/gpfs/apps/MPICH2/slurm/64/lib

export LD_LIBRARY_PATH=/gpfs/apps/GCC/4.6.1/lib64/:$HOME/local/openblas/4.6.1/lib:$LD_LIBRARY_PATH
export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.6.1/bin/gcc"
export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.6.1/bin/gfortran"

export CFLAGS="-m64 -O2 -I $PWD/external_libs/isf/wrappers "
export FCFLAGS=$CFLAGS

../configure \
--with-blas=$HOME/local/openblas/4.6.1/lib/libopenblas.so.0 \
--with-lapack=/gpfs/apps/LAPACK/lib64/liblapack.a \
--disable-gdlib \
--with-gsl-prefix=$HOME/local/gsl/gnu \
--with-fft-lib=$HOME/local/fftw/3.3.4/lib/libfftw3.a \
--with-libxc-prefix=$HOME/local/libxc/gnu/4.6.1/2.2 \
--prefix=/gpfs/projects/ehu31/software/octopus/gnu/4.6.1 \
--enable-mpi --disable-f90-forall

XBES

Magerit (CeSViMa)

Magerit is a cluster located in “Centro de Supercomputación y Visualización de Madrid”. It consists on 245 nodes eServer BladeCenter PS702 with 2 Power7 processors each of 8 cores at 3'3 GHz (422,4 GFlops) and with 32 GB de RAM. Here the “optimal” configuration script which uses GNU compilers with ESSL IBM library.

#!/bin/bash
module purge
module load gcc/4.4
module load openmpi/1.6.3

export LIBSDIR='/sw/openmpi/'

export SOFTWARE_DIR="$HOME/SOFTWARE/exec"

export FC=mpif90
export CC=mpicc
export CXX=mpicxx

export STDFLAGS="-O3 -mcpu=power7 -mtune=power7"

export FCFLAGS=$STDFLAGS
export CFLAGS=$STDFLAGS
export CXXLAGS=$STDFLAGS

export C_TYPE="gnu64-4.4-essl"

export branch=`awk '{print $2}' ../.git/HEAD`
export branch=`basename $branch`

INSTALLDIR="$SOFTWARE_DIR/octopus/git-$C_TYPE/$branch"

#GSL  1.16 --compiled with gcc 4.4.6
export GSL_HOME="$LIBSDIR/GSL/1.16/"

#LAPACK gcc 4.4
export LAPACK_HOME="$LIBSDIR/LAPACK/3.4.2-gnu64-4.4"

#PARMETIS gcc 4.7.2
export PARMETIS_HOME="$LIBSDIR/METIS/PARMETIS-4.0.3/"

#METIS gcc 4.7.2
export METIS_HOME="$LIBSDIR/METIS/METIS-5.1.0/"

#FFTW3.3.4 gcc 4.4
export FFTW_HOME="$LIBSDIR/FFTW/3.3.4-gnu64-4.4"
export LD_LIBRARY_PATH=$FFTW_HOME/lib:$LD_LIBRARY_PATH

#SCALAPACK gcc 4.4.6
export SCALAPACK_HOME="$LIBSDIR/SCALAPACK/2.0.2-gnu64-4.4/"
export LD_LIBRARY_PATH=$SCALAPACK_HOME/lib:$LD_LIBRARY_PATH

#LIBXC
export LIBXC_HOME="$SOFTWARE_DIR/libxc/3.0.0-$C_TYPE/"

#LIBVDWXC
export LIBVDWXC_HOME="$SOFTWARE_DIR/libvdwxc/0.2.0-$C_TYPE/"
export CPPFLAGS="-I$LIBVDWXC_HOME/include -DHAVE_LIBVDWXC"
#export LIBS=-lvdwxcfort
export LDFLAGS="-L$LIBVDWXC_HOME/lib -lvdwxcfort -Wl,-rpath=$LIBVDWXC_HOME/lib"
export LD_LIBRARY_PATH=$LIBVDWXC_HOME/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/opt/ibmcmp/lib64/:/opt/ibmcmp/xlsmp/2.1/lib64/:/opt/ibmcmp/xlf/13.1/lib64/:$LD_LIBRARY_PATH

export LIBS_BLAS=" -L$SCALAPACK_HOME/lib -L$LAPACK_HOME/lib -lscalapack -llapack -lessl -lblacssmp -L$FFTW_HOME/lib -lfftw3 -lfftw3_threads -lfftw3_mpi / opt/gnu/gcc/4.4.6/lib64/libgfortran.a -L/opt/ibmcmp/xlf/13.1/lib64/ -lxlf90_r -lxlfmath -lxl -L/opt/ibmcmp/xlsmp/2.1/lib64/ -lxlomp_ser -Wl,--rpath /opt/ibmcmp/xlf/13.1/lib64/ /opt/ibmcmp/xlf/13.1/lib64/libxl.a -R/opt/ibmcmp/lib64/"

make distclean
make clean
../configure --prefix=$INSTALLDIR \
    --disable-gdlib --disable-gsltest \
    --with-libxc-prefix=$LIBXC_HOME \
    --with-fftw-prefix=$FFTW_HOME \
    --with-gsl-prefix=$GSL_HOME \
    --with-metis-prefix=$METIS_HOME\
    --with-parmetis-prefix=$PARMETIS_HOME \
    --enable-mpi \
    --enable-openmp

MPCDF systems (cobra, raven, ada)

Cobra and Raven are the HPC machines of the Max-Planck Compute and Data Facility in Garching. On these clusters, as well as ADA, Octopus can be built with the following script. A copy of this compilation script, its CLI arguments, config.log, and the list of required runtime modules will be all stored at .build.doc/ in installation path for future reference.

#!/bin/bash
#
# Compilation script for octopus. If this script is not executed in a subdirectory (e.g. _build) of the root octopus directory,
# you have to pass the path to octopus directory to the compilation script using the --root-path flag
# See ./build_octopus.sh --help for complete list of available options.
#
# Usage examples:
#
# build octopus for CPU, using gnu compilers and openmpi, with octopus at "$PWD/.." and install it to "$PWD/installed"
# > ./build_octopus_mpcdf.sh
#
# build octopus for CPU, using intel compilers and intel-mpi, install it to "/u/user/octopus/"
# > ./build_octopus_mpcdf.sh --compiler=intel --mpi=impi --prefix=/u/user/octopus/
#
# build octopus with no compiler optimization, enable compiler warnings, and runtime checks
# > ./build_octopus_mpcdf.sh --debug
#
# build octopus for CPU without MPI (with OpenMP support only)
# > ./build_octopus_mpcdf.sh --no-mpi
#
# build octopus for CPU + GPU, using gnu compilers and cuda-aware openmpi (if possible)
# > ./build_octopus_mpcdf.sh --cuda
#
# build octopus for CPU + GPU, using gnu compilers and cuda-unaware(!) openmpi
# > ./build_octopus_mpcdf.sh --cuda --no-cuda-aware-mpi
#
# rebuild without running the configuration script
# > ./build_octopus_mpcdf.sh --no-config
#
# rebuild without compiling the dependencies (WARNING: DO NOT USE THIS AFTER CHANGING THE FUNCTION/SUBROUTINE INTERFACES)
# > ./build_octopus_mpcdf.sh --no-dep
#
#
# NOTE: You must load the required runtime modules in your job submission script before
#       executing the octopus for your calculations. The list of these modules is printed
#       at the end of compilation and also is saved in your installation path at
#       .build.doc/required_runtime_modules file. You can also generate this list by passing
#       '--show-modules-only' flag to this script without compiling the code.

set -e

# version of this script
VERSION="1.5"


# default value of cli arguments
_arg_config="on"
_arg_dep="on"
_arg_cuda="off"
_arg_cuda_aware_mpi="on"
_arg_debug="off"
_arg_compiler="gnu"
_arg_mpi="openmpi"
_arg_prefix="$PWD/installed"
_arg_root_path="${PWD%/*}"
_arg_show_modules_only="off"


# default modules
# Latest tests:
## by MFT on COBRA, RAVEN & ADA clusters @01.2023 using octopus: fd2983011b37c086e7fd979e99afef197beab7a7
# other clusters may have different version of these modules
# update the versions if necessary!
#
## for OpenMP only
modules_gcc="gcc/11"
modules_intel="intel/21.5.0"
modules_libs_openmp="mkl/2022.0 gsl hdf5-serial netcdf-serial etsf_io libxc/5.1.5 libvdwxc cgal boost/1.74"
## for MPI
modules_intel_impi="intel/21.5.0 impi/2021.5"
modules_gcc_impi="gcc/11 impi/2021.5"
modules_gcc_openmpi="gcc/11 openmpi/4"
modules_libs_mpi="mkl/2022.0 gsl hdf5-serial netcdf-serial etsf_io libxc/5.1.5 libvdwxc-mpi cgal boost/1.74 elpa/mpi/standard/2022.11.001 metis-64 parmetis-64 dftbplus/22.1"
## For GPU
modules_cuda="cuda/11.4"
modules_cuda_aware_mpi="openmpi_gpu/4"


# configuration features to verify
## common features we expect to detect in all cases
common_features_list=('CGAL' 'NETCDF' 'ETSF_IO' 'FFTW3' 'LIBXC5' 'LIBXC_FXC' 'LIBXC_KXC' 'LIBVDWXC' 'OPENMP')
## expected features of mpi version
mpi_features_list=('MPI' 'ELPA' 'PARMETIS' 'LIBVDWXC_MPI' 'DFTBPLUS')
## expected features of cuda version
cuda_features_list=('CUDA')

expected_features_list=( "${common_features_list[@]}" )


die()
{
	_die_ret="${2:-1}"
	test "${_PRINT_HELP:-no}" = yes && print_help >&2
	echo "$1" >&2
	exit "${_die_ret}"
}

print_help()
{
	printf '%s\n' "Compilation script for octopus. This script needs to be executed in a subdirectory (e.g. _build) of the root directory or the path to octopus root directory must be set using --root-path flag."
	print_usage
}

print_usage()
{
	printf '\nUsage: %s [--(no-)config] [--(no-)dep] [--(no-)cuda] [--(no-)cuda-aware-mpi] [--(no-)debug] [--compiler <arg>] [--no-mpi] [--mpi <arg>] [--prefix <arg>] [--show-modules-only] [-v|--version] [-h|--help]\n' "$0"
	printf '\t%s\n' "--config, --no-config: run configure script (on by default)"
	printf '\t%s\n' "--dep, --no-dep: compile dependencies (on by default)"
	printf '\t%s\n' "--cuda, --no-cuda: compile with CUDA support (off by default)"
	printf '\t%s\n' "--cuda-aware-mpi, --no-cuda-aware-mpi: compile with CUDA aware MPI support (on by default)"
	printf '\t%s\n' "--<a href="/documentation//13/variables//%7b%7b%3c%20variable%20">{{&lt; variable </a>, --no-<a href="/documentation//13/variables//%7b%7b%3c%20variable%20">{{&lt; variable </a>: compile with <a href="/documentation//13/variables//%7b%7b%3c%20variable%20">{{&lt; variable </a> flags [disable optimization, activate compiler warnings and runtime checks, link to <a href="/documentation//13/variables//%7b%7b%3c%20variable%20">{{&lt; variable </a> impi] (off by default)"
	printf '\t%s\n' "--compiler: compiler type. available options: 'intel' or 'gnu' (default: 'gnu')"
	printf '\t%s\n' "--no-mpi: compile without the MPI support (equivalent to '--mpi=off')"
	printf '\t%s\n' "--mpi: MPI implementation. available options: 'impi', 'openmpi', or 'off' (default: 'openmpi')"
	printf '\t%s\n' "--prefix: installation path (default: './installed/')"
	printf '\t%s\n' "--root-path: path to the root of octopus repository (default: '../')"
	printf '\t%s\n' "--show-modules-only: print the required runtime modules and exit"
	printf '\t%s\n' "-v, --version: print compilation script version"
	printf '\t%s\n' "-h, --help: print help (this list!)"
}


parse_commandline()
{
	while test $# -gt 0
	do
		_key="$1"
		case "$_key" in
			--no-config|--config)
				test "${1:0:5}" = "--no-" && _arg_config="off"
				;;
			--no-dep|--dep)
				test "${1:0:5}" = "--no-" && _arg_dep="off"
				;;
			--no-cuda|--cuda)
				test "${1:0:5}" = "--no-" || _arg_cuda="on"
				;;
			--no-cuda-aware-mpi|--cuda-aware-mpi)
				test "${1:0:5}" = "--no-" && _arg_cuda_aware_mpi="off"
				;;
			--no-debug|--debug)
				test "${1:0:5}" = "--no-" || _arg_debug="on"
				;;
			--compiler)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_compiler="$2"
				shift
				;;
			--compiler=*)
				_arg_compiler="${_key##--compiler=}"
				;;
			--no-mpi)
				_arg_mpi="off"
				;;
			--mpi)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_mpi="$2"
				shift
				;;
			--mpi=*)
				_arg_mpi="${_key##--mpi=}"
				;;
			--prefix)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_prefix="$2"
				shift
				;;
			--prefix=*)
				_arg_prefix="${_key##--prefix=}"
				;;
			--root-path)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_root_path="$2"
				shift
				;;
			--root-path=*)
				_arg_root_path="${_key##--root-path=}"
				;;
			--show-modules-only)
				_arg_show_modules_only="on"
				;;
			-v|--version)
				echo Octopus compilation script v$VERSION
				exit 0
				;;
			-h|--help)
				print_help
				exit 0
				;;
			-h*)
				print_help
				exit 0
				;;
			*)
				_PRINT_HELP=yes die "FATAL ERROR: Unexpected argument '$1'" 1
				;;
		esac
		shift
	done
}

# verify expected features are detected in config.h as "#define HAVE_*** 1"
verify_configuration()
{
	read -ra _expected_features <<< "$@"

	for feature in "${_expected_features[@]}"; do
		printf "checking feature: %s ..." "$feature"
		_definition="#define HAVE_$feature 1"
		grep -q "$_definition" config.h || ( echo "NOT FOUND!" && echo "ERROR: Configuration script failed to detect $feature" && exit 1 )
		echo "OK!"
	done

}

parse_commandline "$@"

# expand ~ in path arguments
_arg_prefix="${_arg_prefix/#\~/$HOME}"
_arg_root_path="${_arg_root_path/#\~/$HOME}"

echo "====================================="
[ "$_arg_show_modules_only" = "off" ] && echo "run configure: $_arg_config"
[ "$_arg_show_modules_only" = "off" ] && echo "build dependencies: $_arg_dep"
echo "compiler: $_arg_compiler"
echo "mpi: $_arg_mpi"
echo "cuda support: $_arg_cuda"
[ "$_arg_mpi" = "openmpi" ] && [ "$_arg_cuda" = "on" ] && echo "cuda-aware-mpi: $_arg_cuda_aware_mpi"
[ "$_arg_show_modules_only" = "off" ] && echo "debug mode: $_arg_debug"
[ "$_arg_show_modules_only" = "off" ] && echo "installation path: $_arg_prefix"
[ "$_arg_show_modules_only" = "off" ] && echo "octopus repository path: $_arg_root_path"
echo "====================================="

if [ ! -f "$_arg_root_path/configure.ac" ]; then
	echo "Error: Could not find configure.ac in octopus repository path ($_arg_root_path). Please pass the right path to the compilation script using the \"--root-path\" flag"
	exit 1
fi


# validating script arguments

if [ "$_arg_compiler" != "intel" ] && [ "$_arg_compiler" != "gnu" ]; then
	echo "Error: Invalid --compiler option: $_arg_compiler"
	print_usage
	exit 1
fi

if [ "$_arg_mpi" != "impi" ] && [ "$_arg_mpi" != "openmpi" ] && [ "$_arg_mpi" != "off" ]; then
	echo "Error: Invalid --mpi option: $_arg_mpi"
	print_usage
	exit 1
fi

if [ "$_arg_compiler" = "intel" ] && [ "$_arg_mpi" = "openmpi" ]; then
	echo "WARNING: Compiling Octopus with intel compilers and linking to OpenMPI library is not supported!"
	echo "Will use IntelMPI (impi) instead!"
	_arg_mpi="impi"
fi



module purge

if [ "$_arg_compiler" = "intel" ]; then
	if [ "$_arg_mpi" = "off" ]; then
		# shellcheck disable=2086
		module load $modules_intel
		required_modules+=" ${modules_intel}"

		export CC=icc
		export FC=ifort
		export CXX=icpc
		MKL_FLAGS="-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"

	elif [ "$_arg_mpi" = "impi" ]; then
		# shellcheck disable=2086
		module load $modules_intel_impi
		required_modules+=" ${modules_intel_impi}"

		if [ "$_arg_debug" = "on" ]; then
			export I_MPI_LINK=dbg
		else
			export I_MPI_LINK=opt
		fi

		export CC=mpiicc
		export FC=mpiifort
		export CXX=mpiicpc
		MKL_FLAGS="-lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl"

	elif [ "$_arg_mpi" = "openmpi" ]; then
		# we should never reach this part of script! Adapt the script if we support Intel compilers + OpenMPI in future.
		echo "ERROR: Compiling Octopus with intel compilers and linking to OpenMPI library is not currently supported!"
		exit 1
	fi

	if [ "$_arg_debug" = "on" ]; then
		export CFLAGS="-O0 -g -traceback"
		FCFLAGS_EXTRA="-warn all -warn noexternals -warn nounused -check all"
	else
		export CFLAGS="-O3 -xCORE-AVX512 -qopt-zmm-usage=high -fma -g -traceback"
	fi
	export CXXFLAGS="$CFLAGS"
	export FCFLAGS="$CFLAGS $FCFLAGS_EXTRA"

elif [ "$_arg_compiler" = "gnu" ]; then
	if [ "$_arg_mpi" = "off" ]; then

		module load $modules_gcc
		required_modules+=" ${modules_gcc}"

		export CC=gcc
		export FC=gfortran
		export CXX=g++
		MKL_FLAGS="-lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"

	elif [ "$_arg_mpi" = "impi" ]; then
		# shellcheck disable=2086
		module load $modules_gcc_impi
		required_modules+=" ${modules_gcc_impi}"

		export CC=mpigcc
		export FC=mpigfortran
		export CXX=mpig++
		MKL_FLAGS="-lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -lgomp -lpthread -lm -ldl"

	elif [ "$_arg_mpi" = "openmpi" ]; then
		# shellcheck disable=2086
		module load $modules_gcc_openmpi
		required_modules+=" ${modules_gcc_openmpi}"

		export CC=mpicc
		export FC=mpif90
		export CXX=mpicxx
		MKL_FLAGS="-lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lgomp -lpthread -lm -ldl"
	fi

	if [ $_arg_debug = on ]; then
		export CFLAGS="-O0 -g"
		FCFLAGS_EXTRA="-fallow-argument-mismatch -fallow-invalid-boz -fcheck=all -Wall -Wno-c-binding-type -Wno-maybe-uninitialized -Wno-unused-dummy-argument"
	else
		export CFLAGS="-O3 -march=native -g"
		FCFLAGS_EXTRA="-fallow-argument-mismatch -fallow-invalid-boz"
	fi
	export CXXFLAGS="$CFLAGS"
	export FCFLAGS="$CFLAGS $FCFLAGS_EXTRA"
fi

if [ "$_arg_mpi" = "off" ]; then
	# shellcheck disable=2086
	module load $modules_libs_openmp
	export MKL="-L${MKLROOT}/lib/intel64 ${MKL_FLAGS}"
	ENABLE_MPI=
	SCALAPACK=no
	PARMETIS_HOME=no
	METIS_HOME=no
	ELPA_HOME=no
	DFTBPLUS_HOME=no
else
	# shellcheck disable=2086
	module load $modules_libs_mpi
	export MKL="-L${MKLROOT}/lib/intel64 ${MKL_FLAGS}"
	expected_features_list+=( "${mpi_features_list[@]}" )
	ENABLE_MPI=--enable-mpi
	SCALAPACK=$MKL

fi


if [ "$_arg_compiler" = "intel" ]; then
	# gcc 10 or later is needed for newer C++ libraries. This needs to be loaded after $modules_libs_...
	module load gcc/11
fi

if [ "$_arg_cuda" = "on" ]; then
	module load $modules_cuda
	required_modules+=" ${modules_cuda}"
	expected_features_list+=( "${cuda_features_list[@]}" )

	if [ "$_arg_mpi" = "openmpi" ] && [ "$_arg_cuda_aware_mpi" = "on" ] ; then
		# on COBRA the underlying network drivers do not support CUDA-aware MPI
		module -s load $modules_cuda_aware_mpi && required_modules+=" ${modules_cuda_aware_mpi}"\
		|| echo "WARNING: CUDA-aware MPI module does not seem to be available in the current environment. Proceeding without the CUDA-aware MPI support!"
	fi
	CUDA_FLAGS="--enable-cuda --enable-nvtx --with-cuda-prefix=$CUDA_HOME"
	LDFLAGS="$LDFLAGS:$CUDA_HOME/lib64"
else
	CUDA_FLAGS=""
fi

if [ "$_arg_show_modules_only" = "on" ]; then
	echo -e "\nRequired runtime modules for the selected configuration would be: ${required_modules}\n"
	exit 0
fi

module list

export CXXFLAGS="$CXXFLAGS -std=c++14"
export LDFLAGS="-Xlinker -rpath=$MKL_HOME/lib/intel64:$GSL_HOME/lib:$NETCDF_HOME/lib:$ELPA_HOME/lib:$METIS_HOME/lib:$PARMETIS_HOME/lib:$LIBXC_HOME/lib:$DFTBPLUS_HOME/lib"

if [ "$_arg_config" != "off" ]; then
	pushd "$_arg_root_path"
	autoreconf -i
	popd

	# shellcheck disable=2086
	"$_arg_root_path"/configure $CUDA_FLAGS \
	FCFLAGS_FFTW="-I$MKLROOT/include/fftw" \
	--prefix="$_arg_prefix" \
	$ENABLE_MPI \
	--enable-openmp \
	--disable-gdlib \
	--enable-shared --disable-static \
	--with-gsl-prefix="$GSL_HOME" \
	--with-cgal="$CGAL_HOME" \
	--with-libxc-prefix="$LIBXC_HOME" \
	--with-libvdwxc-prefix="$LIBVDWXC_HOME" \
	--with-blas="$MKL" \
	--with-lapack="$MKL" \
	--with-blacs="$SCALAPACK" \
	--with-scalapack="$SCALAPACK" \
	--with-netcdf-prefix="$NETCDF_HOME" \
	--with-etsf-io-prefix="$ETSF_IO_HOME" \
	--with-metis-prefix="$METIS_HOME" \
	--with-parmetis-prefix="$PARMETIS_HOME" \
	--with-boost="$BOOST_HOME" \
	--with-elpa-prefix="$ELPA_HOME" \
	--with-dftbplus-prefix="$DFTBPLUS_HOME"

	verify_configuration "${expected_features_list[@]}"
fi


if [ "$_arg_dep" = "off" ]; then
	NODEP=1
else
	NODEP=0
fi

make NODEP=$NODEP -j20
make NODEP=$NODEP -j20 install

# keep the compilation script, its cli arguments, config.log, and the list of required runtime modules
# at .build.doc/ in installation path for future reference
mkdir -p "$_arg_prefix"/.build.doc/
cp -f config.log "$_arg_prefix"/.build.doc/
cp -f "$0" "$_arg_prefix"/.build.doc/
echo "$0 $*" > "$_arg_prefix"/.build.doc/compilation_command
echo "$required_modules" > "$_arg_prefix"/.build.doc/required_runtime_modules
echo "====================================="
echo "Compilation done!"
echo "A copy of this compilation script, its CLI arguments, config.log, and the list of required runtime modules are all stored in $_arg_prefix/.build.doc/ for future reference"
echo -e "\nRequired runtime modules: ${required_modules}\n"

Specific architectures

Generic Ubuntu 10.10

United States

Europe

PFFT 1.0.5 and BigDFT 1.6.0

PFFT 1.0.7 and BigDFT 1.7.0

Specific architectures

Generic Ubuntu 10.10

United States

Europe

PFFT 1.0.5 and BigDFT 1.6.0

PFFT 1.0.7 and BigDFT 1.7.0

prev next