Manual:Specific architectures

From OctopusWiki
Jump to navigation Jump to search

Here the general instructions for building Octopus are supplemented by instructions for some specific large supercomputers. Included also is how to configure the optional ETSF_IO library, and how you can run the testsuite in parallel.

Generic Ubuntu

Install the following packages: 

* Required: gfortran, gcc, g++, make, automake, m4, libtool, libgsl0-dev, libblas-dev, liblapack-dev, libfftw3-dev, libxc-dev
* Optional: mpi-default-dev, libgd2-xpm-dev, libsparskit-dev, libnetcdf-dev, libtrilinos-dev

[Unfortunately you should not use libblacs-mpi-dev, libscalapack-mpi-dev packages because they can give incorrect results. To use BLACS and ScaLAPACK (optional) you must build them yourself.]

You can also compile libxc yourself if you want a later version than is available in the package: First download libxc, extract with tar xzf, enter the resulting directory, and execute: 

./configure --prefix=`pwd` CC=gcc FC=gfortran FCCPP="/lib/cpp -ansi -freestanding" FCFLAGS="-O3 -ffree-line-length-none" CFLAGS=-O3
make install
make check

For octopus, download and extract, and execute the following (inserting the path where you put libxc): 

./configure --prefix=`pwd` CC=gcc CXX=g++ FC=gfortran FCCPP="/lib/cpp -ansi -freestanding" FCFLAGS="-O3 -ffree-line-length-none" CFLAGS=-O3 --with-libxc-prefix=LIBXC_PATH
make install

If you are using MPI, replace the compilers by CC=/usr/bin/mpicc CXX=/usr/bin/mpicxx FC=/usr/bin/mpif90 and add --enable-mpi.

To build ETSF_IO (optional), libnetcdf-dev is required. 

./configure --prefix=`pwd` CC=gcc FC=gfortran FCFLAGS="-O3 -ffree-line-length-none" CFLAGS="-O3" --with-netcdf-ldflags="-lnetcdff"
make install

Add --with-etsf-io-prefix="$DIR/etsf_io-1.0.4" to the Octopus configure line, where $DIR/etsf_io-1.0.4 is where you installed ETSF_IO.

(Sept 2021)

Generic MacOS

Octopus is now available in MacPorts. For more info, and how to build by hand with MacPorts libraries, see Compiling_octopus_in_OS_X.

United States

Sequoia/Vulcan (IBM Blue Gene/Q)

Supercomputers at Lawrence Livermore National Laboratory based on the IBM Blue Gene/Q architecture. This was tested in Vulcan, but it should work on Sequoia as well.

https://computing.llnl.gov/tutorials/bgq/ 

export LIBS_BLAS="-L/usr/local/tools/essl/5.1/lib/ -lesslsmpbg"
export LIBS_LAPACK="/usr/local/tools/lapack/lib/liblapack.a"
export LIBS_FFT="-L/usr/local/tools/fftw-3.3.3/lib/ -lfftw3_omp"
export FC_INTEGER_SIZE=4
export CC_FORTRAN_INT=int
export CC=mpixlc_r
export FC=mpixlf95_r
export LDFLAGS="-qsmp=omp"
export CFLAGS="-g -O3"
export FCFLAGS=$CFLAGS" -qxlf90=autodealloc -qessl"
./configure --with-libxc-prefix=$HOME --prefix=$HOME --host=powerpc32-unknown-linux-gnu  --build=powerpc64-unknown-linux-gnu --with-gsl-prefix=$HOME --enable-openmp --enable-mpi

(This build does not use Scalapack)

Stampede

10 PFLOPS (PF) Dell Linux Cluster based on 6,400+ Dell PowerEdge server nodes, each outfitted with 2 Intel Xeon E5 (Sandy Bridge) processors and an Intel Xeon Phi Coprocessor (MIC Architecture). Texas Advanced Computing Center, University of Texas, Austin, USA.

https://portal.tacc.utexas.edu/user-guides/stampede 

module swap mvapich2 impi
module load fftw3 gsl hdf5 netcdf
MKL_DIR=$MKLROOT
./configure --prefix=`pwd` CC=mpicc FC=mpif90 CFLAGS="-O3" FCFLAGS="-O3" --enable-mpi \
--with-blas="-L$MKL_DIR/lib/intel64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread" --with-fftw-prefix="$TACC_FFTW3_DIR" \
--with-netcdf-prefix="$TACC_NETCDF_DIR" --enable-newuoa --with-libxc-prefix=`pwd`/../libxc-2.2.2 --with-blacs="$MKL_DIR/lib/intel64/libmkl_blacs_intelmpi_lp64.a" \
--with-scalapack="$MKL_DIR/lib/intel64/libmkl_scalapack_lp64.a"
make -j 6 install

For libxc: 

./configure --prefix=`pwd` CC=mpicc FC=mpif90 FCFLAGS="-O3" CFLAGS="-O3"
make -j 6 install
make check

testsuite script: 

#!/usr/bin/env bash

#SBATCH -n 16
#SBATCH -p development
#SBATCH -t 01:00:00
#SBATCH --export=ALL

module load perl
WORKDIR=$PWD
export TEMPDIRPATH=$SCRATCH/tmp
cd $HOME/octopus
export OCT_TEST_NJOBS=8
export MPIEXEC=`which ibrun`
make check &> $WORKDIR/makecheck

(13 September 2016)

Cori

Cray XC40 at National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory, USA

http://www.nersc.gov/users/computational-systems/cori/ 

module load cray-fftw gsl metis cray-netcdf-hdf5parallel
./configure CC=cc CXX=CC FC=ftn --prefix=`pwd` --with-libxc-prefix=/usr/common/software/libxc/4.3.4/intel/haswell --enable-mpi --enable-openmp --with-metis-prefix=$METIS_DIR FCFLAGS_FFTW=-I$FFTW_INC
make -j 6 install

To compile for KNL instead, start by the following lines to set up the modules, and to run the configure step on a KNL node. The "make install" part does not need to be on a KNL node. 

module swap craype-haswell craype-mic-knl
salloc -N 1 -C knl -t 00:30:00 -p debug

To run the testsuite in parallel (for KNL, swap haswell->knl in the -C line): 

#!/bin/bash -l
#SBATCH -J pulpo
#SBATCH -N 1
#SBATCH -C haswell
#SBATCH -p regular
#SBATCH -t 04:00:00
#SBATCH --export=ALL

cd $HOME/cori/octopus
export OMP_NUM_THREADS=1
export MPIEXEC=`which srun`
export TEMPDIRPATH=$SCRATCH/tmp
export OCT_TEST_NJOBS=15
make check &> $SLURM_SUBMIT_DIR/makecheck

(Version 9.2, 31 Dec 2020)

Lonestar

Dell Linux cluster at Texas Advanced Computing Center (TACC), University of Texas, Austin, USA

Part of National Science Foundation's TeraGrid

http://services.tacc.utexas.edu/index.php/lonestar-user-guide 

module load gsl
module load mkl
module load netcdf
module load fftw3
cd libxc
./configure --prefix=`pwd`/.. --enable-mpi CC=mpicc FC=mpif90 F77=mpif90 FCFLAGS=-O3 CFLAGS=-O3
make install
cd ..
export SLKPATH=/opt/apps/intel11_1/mvapich2_1_6/scalapack/1.8.0/lib
./configure --prefix=`pwd` --enable-mpi CC=mpicc FC=mpif90 \
--with-blas="-Wl,--start-group $TACC_MKL_LIB/libmkl_intel_lp64.a $TACC_MKL_LIB/libmkl_sequential.a $TACC_MKL_LIB/libmkl_core.a -Wl,--end-group -lpthread" \
--with-fft-lib="-L$TACC_FFTW3_LIB -lfftw3" --with-netcdf-prefix="$TACC_NETCDF_DIR" --disable-gdlib \
--with-etsf-io-prefix="$HOME/etsf_io-1.0.3" --enable-newuoa --with-libxc-prefix=`pwd` \
--with-blacs="$SLKPATH/libscalapack.a $SLKPATH/blacs_MPI-LINUX-0.a $SLKPATH/blacsF77init_MPI-LINUX-0.a $SLKPATH/blacs_MPI-LINUX-0.a"
make install

To build ETSF_IO: 

module load netcdf
./configure --prefix=`pwd` FC=mpif90 --with-netcdf-module-path=$TACC_NETCDF_INC --with-netcdf-ldflags=-L$TACC_NETCDF_LIB FCFLAGS=-O3
make install
make check

To run the testsuite in parallel: 

#!/bin/bash

#$ -N test_pulpo
#$ -cwd
#$ -pe 6way 12
#$ -q development
#$ -l h_rt=00:45:00
#$ -V

cd $HOME/octopus
export TEMPDIRPATH=$SCRATCH/tmp
if [ ! -d $TEMPDIRPATH ]; then
    mkdir $TEMPDIRPATH
fi

export MPIEXEC=`which ibrun`
export OCT_TEST_NPROCS=1
make check &> makecheck

(2 May 2011)

Lawrencium

Dell Linux cluster at Lawrence Berkeley National Laboratory, USA

Cluster available only to Lawrence Berkeley National Laboratory researchers

http://lrc.lbl.gov/html/Lawrencium.html 

module load icc/10.1.018
module load ifort/10.1.018
module load mkl/2011.1.107
module load openmpi/1.4.3-intel
module load netcdf/4.0-intel
module load fftw/3.1.2-intel
module load autoconf/2.65
module load gsl/1.13-intel
module load gdlib/2.0.35
module load libtool/2.2.6b

./configure --prefix=`pwd` CC=mpicc FC=mpif90 CFLAGS="-O3" FCFLAGS="-O3 -check bounds" --enable-mpi \
--with-blas="-L$MKLROOT/lib/intel64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread" \
--with-fft-lib="-L/global/software/centos-5.x86_64/modules/fftw/3.1.2-intel/lib -lfftw3" --with-netcdf-prefix="$NETCDF" --with-netcdf-include="$NETCDF/include" --enable-newuoa \
--with-libxc-prefix=`pwd` --with-blacs="$MKL_DIR/lib/intel64/libmkl_blacs_openmpi_lp64.a" --with-scalapack="$MKL_DIR/lib/intel64/libmkl_scalapack_lp64.a"

To build ETSF_IO: 

module load netcdf/4.2-intel-p
./configure --prefix=`pwd` CC=mpicc FC=mpif90 CFLAGS="-O3" FCFLAGS="-O3" --with-netcdf-module-path=$NETCDF_DIR/include/ --with-netcdf-ldflags="-L$NETCDF_DIR/lib/ -lnetcdf"

(21 Aug 2012)

Europe

Curie

PFFT 1.0.5 and BigDFT 1.6.0

The Curie supercomputer, owned by GENCI and operated into the TGCC by CEA, is the first French Tier0 system open to scientists through the French participation into the PRACE research infrastructure.

Curie is offering 3 different fractions of x86-64 computing resources for addressing a wide range of scientific challenges and offering an aggregate peak performance of 2 PetaFlops.

General information: http://www-hpc.cea.fr/en/complexe/tgcc-curie.htm

Compilation with PFFT version 1.0.5 and LibISF taken from BigDFT 1.6.0 (previously compiled): 

#!/bin/bash

module load c/intel
module load fortran/intel
module load bullxmpi
module load mkl
module load gsl/1.14

WPREFIX=$WORKDIR/software
PREFIX=$HOME/software
export ISF_HOME="$WORKDIR/software/libisf"

export CC=mpicc
export CFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$ISF_HOME/include"
export FC=mpif90
export FCFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$PREFIX/fftw/3.3/include -I$ISF_HOME/include"

export LIBS_BLAS="${MKL_LIBS}"
export LIBS_FFT="-Wl,--start-group -L$PREFIX/pfft/1.0.5/lib -L$PREFIX/fftw/3.3/lib -lpfft -lfftw3 -lfftw3_mpi -Wl,--end-group"
export LDFLAGS=$LDFLAGS" -L$ISF_HOME/lib -lisfsolver -lrt"
 
../configure --prefix=$WPREFIX/octopus/superciliosus_pfft \
  --disable-openmp --with-libxc-prefix=$WPREFIX/libxc/2.0.x \
  --disable-gdlib --with-gsl-prefix=/usr/local/gsl-1.14 \
  --enable-mpi --enable-newuoa \
  --with-pfft-prefix=$PREFIX/pfft/1.0.5

BigDFT was configured like this: 

#!/bin/bash 

export FC=mpif90

../configure --with-ext-linalg="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lm" \
  --with-ext-linalg-path="-L/usr/local/Intel_compilers/c/composer_xe_2011_sp1.7.256/mkl/lib/intel64" \
  --prefix=$WORKDIR/software/bigdft


PFFT 1.0.7 and BigDFT 1.7.0

#!/bin/bash

module load c/intel
module load fortran/intel
module load bullxmpi
module load mkl
module load gsl/1.14

WPREFIX=$WORKDIR/poisson/software
WLPREFIX=$WORKDIR/poisson/local
export ISF_HOME="$WLPREFIX/libisf"
export FFTW_HOME="$WLPREFIX/fftw-3.3.3"
export PFFT_HOME="$WLPREFIX/pfft-1.0.7-alpha"
 
export CC=mpicc
export CFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$ISF_HOME/include"
export FC=mpif90
export FCFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$FFTW_HOME/include -I$ISF_HOME/include"

export LIBS_BLAS="${MKL_LIBS}"
export LIBS_FFT="-Wl,--start-group -L$PFFT_HOME/lib -L$FFTW_HOME/lib -lpfft -lfftw3 -lfftw3_mpi -Wl,--end-group"

export LDFLAGS=" -L/usr/lib64 -L$ISF_HOME/lib -lPSolver-1 -lrt"
export LD_LIBRARY_PATH=$ISF_HOME/lib:$LD_LIBRARY_PATH

../configure --prefix=$WPREFIX/octopus/superciliosus_pfft \
  --disable-openmp --with-libxc-prefix=$WPREFIX/libxc/11066 \
  --disable-gdlib --with-gsl-prefix=/usr/local/gsl-1.14 \
  --enable-mpi --enable-newuoa \
  --with-pfft-prefix=$PFFT_HOME

LibISF compilation: 

#!/bin/bash

module load c/intel
module load fortran/intel
module load bullxmpi
module load mkl
module load gsl/1.14

export FC=mpif90

../configure --with-ext-linalg="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lm" \
   --with-ext-linalg-path="-L/usr/local/Intel_compilers/c/composer_xe_2011_sp1.7.256/mkl/lib/intel64" \
   --prefix=$WORKDIR/poisson/local/libisf \
   --disable-libbigdft --disable-binaries --enable-dynamic-libraries

FFTW 3.3.3 compilation script is in http://www-user.tu-chemnitz.de/~mpip/software/install_fftw-3.3.3_gcc.sh and it is changed as follow: 

< myprefix=$HOME/local
--- 
> module load c/intel
> module load fortran/intel
> module load bullxmpi
> module load mkl
> module load gsl/1.14
> 
> myprefix=$WORKDIR/poisson/local
23c29,30
< wget http://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
---
> # wget http://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
> cp ../fftw-$FFTW_VERSION.tar.gz .
876c883
< ./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-shared \
---
> ./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-shared --disable-shared \


PFFTW 1.0.7 compilation script is in http://www-user.tu-chemnitz.de/~mpip/software/install_pfft-1.0.7-alpha_gcc.sh and it is changed as follow: 

< myprefix=$HOME/local
---
> 
> module load c/intel
> module load fortran/intel
> module load bullxmpi
> module load mkl
> module load gsl/1.14
> 
> myprefix=$WORKDIR/poisson/local
24c31,32
< wget http://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
---
> #wget http://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
> cp ../pfft-$PFFT_VERSION.tar.gz .

Fermi/Juqueen

IBM Blue Gene/Q, Cineca, Italy

General information: http://www.hpc.cineca.it/content/ibm-fermi-user-guide

The exactly the same script has been tested in the Juqueen, Forschungszentrum Jülich, Jülich Supercomputing Centre (JSC), Jülich, Germany. General information: http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/JUQUEEN_node.html 

#/bin/bash

module load gsl
module load lapack
module load blacs
module load scalapack 

export PFFT_HOME="$HOME/local/pfft-threads-1.0.5-alpha"
export FFTW3_HOME="$HOME/local/fftw-threads-3.3.2"
export LD_LIBRARY_PATH=$PFFT_HOME/lib:$FFTW3_HOME/lib:$LD_LIBRARY_PATH 

export FC_INTEGER_SIZE=8
export CC_FORTRAN_INT=int
export CC=mpicc
export CFLAGS="-g -Wl,-relax -O3 -I$PFFT_HOME/include -I$FFTW3_HOME/include"
export FC=mpif90
export FCFLAGS=$CFLAGS" -qxlf90=autodealloc -qessl -qsmp=omp "
export LIBS_BLAS="-lesslsmpbg -L/opt/ibmmath/essl/4.4/lib -lesslbg"
export LIBS_LAPACK="-L$LAPACK_LIB -llapack"

export LIBS_FFT="-L$FFTW3_HOME/lib/ -lfftw3_mpi -lfftw3 -lm"
echo "GSL DIR = $GSL_HOME "
echo "PFFT DIR = $PFFT_HOME "
export LDFLAGS=$LDFLAGS" -R/opt/ibmcmp/lib/bg/bglib \
   -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/powerpc-bgp-linux/lib \
   -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2 \
   -L/opt/ibmcmp/xlf/bg/11.1/bglib \
   -L/opt/ibmcmp/xlmass/bg/4.4/bglib \
   -L/opt/ibmcmp/xlsmp/bg/1.7/bglib \
   -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/runtime/SPI \
   -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/lib \
   -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/lib \
   -L/usr/local/zlib/v1.2.3/lib \
   -L/bgsys/drivers/ppcfloor/runtime/SPI \
   -L$PFFT_HOME/lib -L$FFTW3_HOME/lib \
   -lpfft -lfftw3_mpi -lfftw3 -lm \
   -ldl -lxl -lxlopt -lrt -lpthread"

SVN_VERS=$(svn info .. | grep Revision | awk '{print $2}')
echo $SVN_VERS

../configure --prefix=$HOME/software/octopus_omp \
   --with-libxc-prefix=$HOME/software/libxc_new \
   --with-fft-lib="$FFTW3_HOME/lib/libfftw3.a  $FFTW3_HOME/lib/libfftw3_omp.a" \
   --host=powerpc32-unknown-linux-gnu \
   --build=powerpc64-unknown-linux-gnu \
   --disable-gdlib \
   --disable-f90-forall \
   --with-blas="-L/opt/ibmmath/lib64 -lesslbg" \
   --with-lapack="-L$LAPACK_LIB -llapack" \
   --with-gsl-prefix=$GSL_HOME \
   --enable-mpi --enable-openmp

To build FFTW3. Small changes are made to the script taken from M. Pippig web site http://www-user.tu-chemnitz.de/~mpip/software.php?lang=en It has a patch and is build with MPI and multithreaded (multithreaded may not work in Juqueen): 

#!/bin/sh -e 

myprefix=$HOME/local
FFTW_VERSION="3.3.2"
INSTALLDIR="$myprefix/fftw-threads-${FFTW_VERSION}"
TMP="tmp-mine-fftw-${FFTW_VERSION}" 

# bash check if directory exists
if [ -d $TMP ]; then
        echo "Directory $TMP already exists. Delete it? (y/n)" 
	read answer 
	if [ ${answer} = "y" ]; then 
		rm -rf $TMP 
	else
		echo "Program aborted."
		exit 1
	fi
 fi 

mkdir $TMP && cd $TMP 

wget http://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
gzip -dc fftw-$FFTW_VERSION.tar.gz | tar xvf -
cd fftw-$FFTW_VERSION

# fix bug in fftw alltoall routine that causes deadlocks
patch -b -p0 <<\EOF
--- mpi/transpose-alltoall.c
+++ mpi/transpose-alltoall.c
@@ -223,8 +223,8 @@
	  sbo[pe] = (int) (pe * (b * p->tblock) * vn);
	  rbs[pe] = (int) (db * bt * vn);
	  rbo[pe] = (int) (pe * (p->block * bt) * vn);
-	  if (sbs[pe] != (b * p->tblock) * vn
-	      || rbs[pe] != (p->block * bt) * vn)
+	  if (dbt != p->tblock
+	      || db != p->block)
	       equal_blocks = 0;
     }
     pln->send_block_sizes = sbs;
EOF

./configure --build=powerpc64-bgq-linux-gnu --host=powerpc64-bgq-linux \
--prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-threads --disable-shared \
CC=mpixlc_r F77=mpixlf90_r MPICC=mpixlc_r MPICXX=mpixlcxx_r MPIFC=mpixlf90_r MPILIBS=" " \
CFLAGS='-O3 -g' \
FCFLAGS='-O3 -g' \ 

make -j 4
make install

To build PFFT, also taken from http://www-user.tu-chemnitz.de/~mpip/software.php?lang=en 

#!/bin/sh -e 

myprefix=$HOME/local
PFFT_VERSION=1.0.5-alpha
FFTW_VERSION=3.3.2
INSTDIR=$myprefix/pfft-threads-$PFFT_VERSION
FFTWDIR=$myprefix/fftw-threads-$FFTW_VERSION
TMP="tmp-mine-pfft-$PFFT_VERSION" 

# bash check if directory exists
if [ -d $TMP ]; then
        echo "Directory $TMP already exists. Delete it? (y/n)"
	read answer 
	if [ ${answer} = "y" ]; then 
		rm -rf $TMP 
	else
		echo "Program aborted."
		exit 1
	fi
fi

mkdir $TMP && cd $TMP

wget http://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
gzip -dc pfft-$PFFT_VERSION.tar.gz | tar xvf -
cd pfft-$PFFT_VERSION
./configure --build=powerpc64-bgq-linux-gnu --host=powerpc64-bgq-linux \
--prefix=$INSTDIR --with-fftw3=$FFTWDIR \
MPICC=mpif90 MPICXX=mpicxx MPIFC=mpif90 CC=mpicc FC=mpif90 \
CFLAGS='-O3 -g -qmaxmem=-1' \
FCFLAGS='-O3 -g -qmaxmem=-1' \
--disable-shared

make -j 4
make install

Hydra

Hydra is a supercomputer located at the RZG/MPG. More information: http://www.rzg.mpg.de/services/computing/hydra/about-the-system

Instruction that work with version 13307 

#!/bin/bash

. /etc/profile.d/modules.sh

module purge  
module load intel/14.0  
module load mkl/11.1  
module load mpi.ibm/1.3.0  
module load gsl  
module load fftw
module load netcdf-serial

export MKL="-L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm"

export NFF_HOME="/hydra/u/system/SLES11/soft/nfft/3.2.3/intel-14.0/mpi.ibm-1.3"
export ISF_HOME=$HOME/local/PSolver/1.7.6

export CC=mpiicc 
export CFLAGS="-g -O3 -xHost -openmp -I$FFTW_HOME/include"
export FC=mpiifort 
export FCFLAGS="-g -O3 -xHost -openmp -I$FFTW_HOME/include" 

export LDFLAGS="-I$FFTW_HOME/include -L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -Xlinker -rpath=$MKL_HOME/lib/intel64:$GSL_HOME/lib:$NETCDF_HOME/lib:$NFF_HOME/lib" 

export LD_LIBRARY_PATH=$ISF_HOME/lib:$LD_LIBRARY_PATH 

../configure --enable-mpi  --disable-gdlib --enable-openmp \
   --prefix=$HOME/software/octopus/repository \
   --with-gsl-prefix=$GSL_HOME \
   --with-nfft=$NFF_HOME  \
   --with-libxc-prefix=/hydra/u/system/SLES11/soft/libxc/2.0.3/intel-14.0/mpi.ibm-1.3 \
   --with-blas="$MKL" \
   --with-lapack="$MKL" \
   --with-isf-prefix=$ISF_HOME \
   --with-isf-include=$ISF_HOME/include \
   --with-metis-prefix=$HOME/local/parmetis \
   --with-parmetis-prefix=$HOME/local/parmetis \
   --with-netcdf-prefix=$NETCDF_HOME

Jugene

FZJ-JSC IBM Blue Gene/P, JÜLICH SUPERCOMPUTING CENTRE (JSC), Germany

May also work on other Blue Gene/P computers

General information: http://www.fz-juelich.de/jsc/jugene

Usage information: http://www.fz-juelich.de/jsc/jugene/usage/ 

module load gsl;
module load lapack;
autoreconf -i;
export FC_INTEGER_SIZE=4;
export CC_FORTRAN_INT=int;
export CC=mpixlc_r;
export CFLAGS='-g -O3 -qarch=450d';
export FC=mpixlf90_r;
export FCFLAGS='-g -O3 -qarch=450d -qxlf90=autodealloc -qessl -qsmp=omp';
export LIBS_BLAS="-lesslsmpbg -L/opt/ibmmath/essl/4.4/lib -lesslbg";
export LIBS_LAPACK="-L$LAPACK_LIB -llapack";
export LIBS_FFT="-L/bgsys/local/fftw3/lib/ -lfftw3 -lm";
./configure --prefix=$outdir \
     --host=powerpc32-unknown-linux-gnu \
     --build=powerpc64-unknown-linux-gnu \
     --disable-gdlib \
     --with-gsl-prefix=$GSL_DIR \
     --enable-mpi --enable-openmp;

To run you have to change to $WORK directory and run LoadLeveler. Below is an example of the tutorial (http://www.tddft.org/programs/octopus/wiki/index.php/Tutorial:Benzene_molecule) executing in 32 nodes in the SMP mode (4 threads per chip). The execution mode is hybrid; OpenMP/MPI.

Here is a example job input script: 

# @ job_name = Octopus_Sample_1
# @ comment = "BGP Job by Size"
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL;
# @ wall_clock_limit = 00:30:00
# @ notification = error
# @ notify_user = foo@gmail.com
# @ job_type = bluegene
# @ bg_size = 32
# @ queue
mpirun  -exe $OCTOPUS_HOME/bin/octopus_mpi -mode SMP -verbose 1

To execute the above script: 

llsubmit executing_script

MareNostrum II

MareNostrum is a supercomputer in Europe, the most powerful in Spain. This is one of the seven supercomputers of the Spanish Supercomputing Network. The supercomputer consists of 2,560 JS21 blade computing nodes, each with 2 dual-core IBM 64-bit PowerPC 970MP processors running at 2.3 GHz for 10,240 CPUs in total. The computing nodes of MareNostrum communicate primarily through Myrinet.

You may need to compile GSL and FFTW libraries before starting then installation.

Here is a script to build Octopus: 

#!/bin/bash

export LD_LIBRARY_PATH=/usr/lib
export CC=mpicc
export FC=mpif90

export CFLAGS="-q64 -O3 -qtune=ppc970 -qarch=ppc970 -qcache=auto -qnostrict -qignerrno -qinlglue"
export FCFLAGS=$CFLAGS" -qessl -qxlf90=autodealloc" 

export LIBS_BLAS="-L/usr/lib64 -lessl" 

./configure   \
   --with-lapack=/gpfs/apps/LAPACK/lib64/liblapack.a \
   --disable-gsltest \
   --disable-gdlib \
   --with-gsl-prefix=$outdirGSL \
   --with-fft-lib=$outdirFFT/lib/libfftw3.a  \
   --prefix=$outdir --enable-mpi --disable-f90-forall

As you can see in the script the --with-gsl-prefix has to point to your GSL and --with-fft-lib to your FFTW. You should compile those with the same CFLAGS and FCFLAGS. Here is an example of configuring GSL (we use GSL-1.11): 

./configure --prefix=$outdirGSL CC=gcc -m64 --enable-static --disable-shared

We use FFTW-3.1.3. To configure FFTW3 just write: 

./configure --prefix=$outdirFFT

Recently Marenostrum has changed to the SLURM manager. To execute a job you have to submit it with jobsubmit (more details in the User's Guide).

MareNostrum III

MareNostrum III is a supercomputer based on Intel SandyBridge processors, iDataPlex Compute Racks, a Linux Operating System and an Infiniband interconnection. More information: http://www.bsc.es/marenostrum-support-services/mn3 

module load MKL
module load GSL

export PREFIX=$HOME/Software
export FFTW3_HOME=$HOME/local/fftw-3.3.2
export PFFT_HOME=$HOME/local/pfft-1.0.5-alpha

export MKL_LIBS=$MKL_HOME/lib/intel64
export MKLROOT=$MKL_HOME
export MKL_LIBS="-L$MKLROOT/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm"
export CC=mpicc
export CFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$ISF_HOME/include  -openmp -I$MKLROOT/include"
export FC=mpif90
export FCFLAGS="-xHost -O3 -m64 -ip -prec-div -static-intel -sox -I$FFTW3_HOME/include  -openmp  -I$ISF_HOME/include -I$MKLROOT/include"

export LIBS_BLAS="${MKL_LIBS}"
export LIBS_FFT="-Wl,--start-group -L$PFFT_HOME/lib -L$FFTW3_HOME/lib -lpfft -lfftw3 -lfftw3_mpi -Wl,--end-group"


../configure --prefix=$PREFIX/octopus/ \
 --enable-openmp --with-libxc-prefix=$PREFIX/libxc \
 --disable-gdlib --with-gsl-prefix=/apps/GSL/1.15 \
 --enable-mpi --enable-newuoa \
 --with-pfft-prefix=$PFFT_HOME

Corvo

Corvo is the computational cluster of the Nano-bio Spectroscopy Group. It consists of 1504 processor cores and 5284 GiB of RAM connected by an Infiniband network.

Script to build LIBFM library included in the Scafacos library (modified Scafacos svn version 1920, this is valid since version 9887 of Octopus): 

./configure --enable-fcs-solvers=fmm,direct --enable-fcs-fmm-comm=armci \
  --enable-fcs-fmm-unrolled --enable-fcs-fmm-max-mpol=40 \
  CFLAGS=-O3 CXXFLAGS=-O3 FCFLAGS=-O3 --disable-doc CXX=mpic++ CC=mpicc \
  FC=mpif90 --prefix=$HOME/Software/scafacos \
  LDFLAGS=-L/opt/scalapack/1.8.0/lib -L/opt/blacs/1.1/lib \
  -L/opt/gotoblas2/1.08/lib -L/opt/netcdf/4.0.1/lib -L/opt/lapack/3.2/lib \
  -L/opt/etsf_io/1.0.2/lib --enable-fcs-int=int --enable-fcs-float=double \
  --enable-fcs-integer=integer --enable-fcs-real=real*8 

make
make install

The scripts to compile FFTW 3.3.2 and PFFT 1.0.5 

#!/bin/sh -e 

myprefix=$HOME/local
FFTW_VERSION="3.3.2"
INSTALLDIR="$myprefix/fftw/${FFTW_VERSION}"
TMP="tmp-fftw-${FFTW_VERSION}"

# bash check if directory exists
if [ -d $TMP ]; then
       echo "Directory $TMP already exists. Delete it? (y/n)"
       read answer
       if [ ${answer} = "y" ]; then
               rm -rf $TMP
       else
               echo "Program aborted."
               exit 1
       fi
fi

mkdir $TMP && cd $TMP

wget http://www.fftw.org/fftw-$FFTW_VERSION.tar.gz
gzip -dc fftw-$FFTW_VERSION.tar.gz | tar xvf -
cd fftw-$FFTW_VERSION

# fix bug in fftw alltoall routine that causes deadlocks
patch -b -p0 <<\EOF
--- mpi/transpose-alltoall.c
+++ mpi/transpose-alltoall.c
@@ -223,8 +223,8 @@
          sbo[pe] = (int) (pe * (b * p->tblock) * vn);
          rbs[pe] = (int) (db * bt * vn);
          rbo[pe] = (int) (pe * (p->block * bt) * vn);
-         if (sbs[pe] != (b * p->tblock) * vn
-             || rbs[pe] != (p->block * bt) * vn)
+         if (dbt != p->tblock
+             || db != p->block)
              equal_blocks = 0;
     }
     pln->send_block_sizes = sbs;
EOF

./configure --prefix=$INSTALLDIR --enable-mpi --enable-openmp --enable-threads --disable-shared \
CC="mpicc" F77="mpif90" MPICC="mpicc" \
CFLAGS="-O3" FFLAGS="-O3" \
MPILIBS=" "

make -j 4
make install



#!/bin/sh -e

myprefix=/home/local
PFFT_VERSION=1.0.5-alpha
FFTW_VERSION=3.3.2
INSTDIR=$myprefix/pfft/$PFFT_VERSION
FFTWDIR=$myprefix/fftw/$FFTW_VERSION
TMP="tmp-pfft-$PFFT_VERSION"
module unload fftw/3.2.2

# bash check if directory exists
if [ -d $TMP ]; then
       echo "Directory $TMP already exists. Delete it? (y/n)"
       read answer
       if [ ${answer} = "y" ]; then
               rm -rf $TMP
       else
               echo "Program aborted."
               exit 1
       fi
fi

mkdir $TMP && cd $TMP
export CC=mpicc
export FC=mpif90 

wget http://www.tu-chemnitz.de/~mpip/software/pfft-$PFFT_VERSION.tar.gz
gzip -dc pfft-$PFFT_VERSION.tar.gz | tar xvf -
cd pfft-$PFFT_VERSION
./configure --prefix=$INSTDIR --with-fftw3=$FFTWDIR --disable-shared

make install

This is the scrip to compile Octopus with PFFT and FMM support (through Scafacos library): 

module add gcc
module add ifort
module add gsl
module add etsf_io
module add lapack
module add netcdf
module add gotoblas2
module add mpibull2 
module add blacs
module add scalapack

export CC=mpicc
export FC="mpif90"

# It is important that "/opt/intel/Compiler//11.1/038/bin/intel64/" not to be in the PATH
export PATH=/opt/mpi/mpibull2-1.3.9-14.s/bin:/opt/netcdf/4.0.1/bin:/opt/etsf_io/1.0.2/bin:/opt/gsl//1.13/bin:/opt/intel/composer_xe_2011_sp1.10.319//bin/intel64/:/opt/gcc//4.4/bin:/opt/slurm/bin:/usr/kerberos/bin:/opt/cuda//bin:/usr/local/bin:/bin:/usr/bin

export LIBFM_HOME="$HOME/Software/scafacos"
export PFFT_HOME="$HOME/local/pfft-1.0.5-alpha"
export FFTW3_HOME="$HOME/local/fftw-3.3.2"
export LIBS_PFFT="-L$PFFT_HOME/lib -L$FFTW3_HOME/lib -lpfft -lfftw3_mpi -lfftw3"
export CFLAGS=" -I$PFFT_HOME/include -I$FFTW3_HOME/include -I$LIBFM_HOME/include"
export LDFLAGS=$LDFLAGS" -L$LIBFM_HOME/lib -L$PFFT_HOME/lib -L$FFTW3_HOME/lib -lpfft -lfftw3_mpi -lfftw3 -lm "
export FCFLAGS=$CFLAGS
export CPPFLAGS=" -I$LIBFM_HOME/include "
export LD_LIBRARY_PATH=$LIBFM_HOME/lib:$LD_LIBRARY_PATH
 
../configure --enable-mpi --with-libxc-prefix=$HOME/Software/libxc_9882 \
   --prefix=$HOME/Software/octopus \
   --with-blacs=/opt/blacs/1.1/lib/libblacs.a --with-scalapack \
   --with-libfm="-L$LIBFM_HOME/lib -lfcs4fortran -lfcs -lfcs_direct -lfcs_fmm -lfcs_near -lfcs_gridsort -lfcs_common" \
   --disable-openmp

LaPalma

LaPalma is a supercomputer of the IAC. It is a part of the old MareNostrum II computer. It was not possible to compile with XL compilers, so GNU 4.6.1 version is used.

Previously to Octopus, Libxc (2.2), FFTW (3.3.4), GSL (1.16) and OpenBLAS (0.2.13) were required to compile.

GSL 

export LD_LIBRARY_PATH=/usr/lib:/gpfs/apps/MPICH2/slurm-2.5.6/64/lib/:/gpfs/apps/GCC/4.9.2/lib64/
export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.9.2/bin/gcc"
export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.9.2/bin/gfortran"

./configure --prefix=$HOME/local/gsl/gnu --enable-static --disable-shared

OpenBLAS 

make PREFIX=$HOME/local/openblas/4.6.1 BINARY=64 CC=/gpfs/apps/GCC/4.6.1/bin/gcc FC=/gpfs/apps/GCC/4.6.1/bin/gfortran

Libxc 

export PATH=/gpfs/apps/MPICH2/mx/default/64/bin:$PATH
export LD_LIBRARY_PATH=/gpfs/apps/MPICH2/mx/default/64/lib:/gpfs/apps/MPICH2/slurm/64/lib

export LD_LIBRARY_PATH=/gpfs/apps/GCC/4.6.1/lib64/:$HOME/local/openblas/lib:$LD_LIBRARY_PATH
export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.6.1/bin/gcc"
export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.6.1/bin/gfortran"

export CFLAGS="-m64 -O3"
export FCFLAGS=$CFLAGS

./configure --prefix=$HOME/local/libxc/gnu/4.6.1/2.2

Octopus 

export PATH=/gpfs/apps/MPICH2/mx/default/64/bin:$PATH
export LD_LIBRARY_PATH=/gpfs/apps/MPICH2/mx/default/64/lib:/gpfs/apps/MPICH2/slurm/64/lib

export LD_LIBRARY_PATH=/gpfs/apps/GCC/4.6.1/lib64/:$HOME/local/openblas/4.6.1/lib:$LD_LIBRARY_PATH
export CC="/gpfs/apps/MPICH2/mx/default/64/bin/mpicc -cc=/gpfs/apps/GCC/4.6.1/bin/gcc"
export FC="/gpfs/apps/MPICH2/mx/default/64/bin/mpif90 -fc=/gpfs/apps/GCC/4.6.1/bin/gfortran"

export CFLAGS="-m64 -O2 -I $PWD/external_libs/isf/wrappers "
export FCFLAGS=$CFLAGS

../configure   \
   --with-blas=$HOME/local/openblas/4.6.1/lib/libopenblas.so.0 \
   --with-lapack=/gpfs/apps/LAPACK/lib64/liblapack.a \
   --disable-gdlib \
   --with-gsl-prefix=$HOME/local/gsl/gnu \
   --with-fft-lib=$HOME/local/fftw/3.3.4/lib/libfftw3.a  \
   --with-libxc-prefix=$HOME/local/libxc/gnu/4.6.1/2.2 \
   --prefix=/gpfs/projects/ehu31/software/octopus/gnu/4.6.1 \
   --enable-mpi --disable-f90-forall

XBES

Located at the University of Hamburg in Hamburg. It has a login node and a 24 nodes with Intel Xeon. Here a basic configuration script: 

./configure \
PATH="$PATH:/home/common/lib/gsl-1.16-intel/bin" \
CC="mpiicc -m64" CFLAGS=" -O3 -march=native  -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include" \
FC="mpiifort -m64" FCFLAGS="-O3  -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include" \
GSL_CONFIG=/home/common/lib/gsl-1.16-intel/bin/gsl-config --with-gsl-prefix="/home/common/lib/gsl-1.16-intel/" \
--with-libxc-prefix=$LIBXC_PREFIX   \
--with-blas="${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread -lm -ldl" \
--with-lapack="${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread -lm -ldl" \
--with-fft-lib=/home/common/lib/fftw-3.3.4-intel/lib/libfftw3.a

Magerit (CeSViMa)

Magerit is a cluster located in "Centro de Supercomputación y Visualización de Madrid". It consists on 245 nodes eServer BladeCenter PS702 with 2 Power7 processors each of 8 cores at 3'3 GHz (422,4 GFlops) and with 32 GB de RAM. Here the "optimal" configuration script which uses GNU compilers with ESSL IBM library. 

module purge
module load gcc/4.4
module load openmpi/1.6.3

export LIBSDIR='/sw/openmpi/'

export SOFTWARE_DIR="$HOME/SOFTWARE/exec"

export FC=mpif90
export CC=mpicc
export CXX=mpicxx

export STDFLAGS="-O3 -mcpu=power7 -mtune=power7"

export FCFLAGS=$STDFLAGS
export CFLAGS=$STDFLAGS
export CXXLAGS=$STDFLAGS

export C_TYPE="gnu64-4.4-essl"

export branch=`awk '{print $2}' ../.git/HEAD`
export branch=`basename $branch`

INSTALLDIR="$SOFTWARE_DIR/octopus/git-$C_TYPE/$branch"

#GSL  1.16 --compiled with gcc 4.4.6
export GSL_HOME="$LIBSDIR/GSL/1.16/"

#LAPACK gcc 4.4
export LAPACK_HOME="$LIBSDIR/LAPACK/3.4.2-gnu64-4.4"

#PARMETIS gcc 4.7.2
export PARMETIS_HOME="$LIBSDIR/METIS/PARMETIS-4.0.3/"

#METIS gcc 4.7.2
export METIS_HOME="$LIBSDIR/METIS/METIS-5.1.0/"

#FFTW3.3.4 gcc 4.4
export FFTW_HOME="$LIBSDIR/FFTW/3.3.4-gnu64-4.4"
export LD_LIBRARY_PATH=$FFTW_HOME/lib:$LD_LIBRARY_PATH

#SCALAPACK gcc  4.4.6
export SCALAPACK_HOME="$LIBSDIR/SCALAPACK/2.0.2-gnu64-4.4/"
export LD_LIBRARY_PATH=$SCALAPACK_HOME/lib:$LD_LIBRARY_PATH

#LIBXC
export LIBXC_HOME="$SOFTWARE_DIR/libxc/3.0.0-$C_TYPE/"

#LIBVDWXC
export LIBVDWXC_HOME="$SOFTWARE_DIR/libvdwxc/0.2.0-$C_TYPE/"
export CPPFLAGS="-I$LIBVDWXC_HOME/include -DHAVE_LIBVDWXC"
#export LIBS=-lvdwxcfort
export LDFLAGS="-L$LIBVDWXC_HOME/lib -lvdwxcfort -Wl,-rpath=$LIBVDWXC_HOME/lib"
export LD_LIBRARY_PATH=$LIBVDWXC_HOME/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/opt/ibmcmp/lib64/:/opt/ibmcmp/xlsmp/2.1/lib64/:/opt/ibmcmp/xlf/13.1/lib64/:$LD_LIBRARY_PATH

export LIBS_BLAS=" -L$SCALAPACK_HOME/lib -L$LAPACK_HOME/lib -lscalapack -llapack -lessl -lblacssmp -L$FFTW_HOME/lib -lfftw3 -lfftw3_threads -lfftw3_mpi / opt/gnu/gcc/4.4.6/lib64/libgfortran.a -L/opt/ibmcmp/xlf/13.1/lib64/ -lxlf90_r -lxlfmath -lxl -L/opt/ibmcmp/xlsmp/2.1/lib64/ -lxlomp_ser -Wl,--rpath /opt/ibmcmp/xlf/13.1/lib64/  /opt/ibmcmp/xlf/13.1/lib64/libxl.a -R/opt/ibmcmp/lib64/"

make distclean
make clean
../configure --prefix=$INSTALLDIR \
    --disable-gdlib --disable-gsltest \
    --with-libxc-prefix=$LIBXC_HOME \
    --with-fftw-prefix=$FFTW_HOME \
    --with-gsl-prefix=$GSL_HOME \
    --with-metis-prefix=$METIS_HOME\
    --with-parmetis-prefix=$PARMETIS_HOME \
    --enable-mpi \
    --enable-openmp


MPCDF systems (draco, cobra, raven, ada)

Draco, Cobra, Raven, and Ada are the HPC machines of the Max-Planck Compute and Data Facility in Garching. On these, Octopus can be built with the following script: 

#! /bin/bash -l
# compilation script for octopus
# needs to be executed in a subfolder (e.g. _build) of the root directory
#
# if you want to skip the configure step, call as ./build_octopus.sh noconfig
# if you want to skip the configure step and the dependency checking,
#   call as ./build_octopus.sh noconfig nodep

if [[ ! -f ../configure.ac ]]; then
  echo "Error! Please execute this script in a subfolder of the root directory."
  exit 1
fi

#compiler=intel
compiler=gnu
cuda=yes

#mpi=impi
mpi=openmpi

echo Using $compiler compiler.
[[ $cuda == yes ]] && echo Building with CUDA support.

module purge
if [[ $compiler == intel ]]; then
  # newest intel version
  module load intel/21.5.0 impi/2021.5 mkl/2022.0

  #export I_MPI_LINK=dbg
  export I_MPI_LINK=opt

  if [[ $mpi == impi ]]; then
    export CC=mpiicc
    export FC=mpiifort
    export CXX=mpiicpc
    export MKL="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl"
  elif [[ $mpi == openmpi ]]; then
    echo Not supported!
    exit 1
  else
    export CC=icc
    export FC=ifort
    export CXX=icpc
    export MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
  fi
  export CFLAGS="-O3 -xCORE-AVX512 -qopt-zmm-usage=high -fma -ip -g -traceback"
  export FCFLAGS="$CFLAGS"
  export CXXFLAGS="$CFLAGS"
elif [[ $compiler == gnu ]]; then
  if [[ $mpi == impi ]]; then
    module load gcc/11 impi/2021.5 mkl/2022.0
    export CC=mpicc
    export FC=mpif90
    export CXX=mpicxx
    export MKL="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -lgomp -lpthread -lm -ldl"
  elif [[ $mpi == openmpi ]]; then
    module load gcc/11 openmpi/4 mkl/2022.0
    export CC=mpicc
    export FC=mpif90
    export CXX=mpicxx
    export MKL="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lgomp -lpthread -lm -ldl"
  else
    export CC=gcc
    export FC=gfortran
    export CXX=g++
    export MKL="-L${MKLROOT}/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"
  fi
  #export CFLAGS="-O3 -march=skylake-avx512 -g"
  export CFLAGS="-O3 -march=native -g"
  #export FCFLAGS="$CFLAGS -Wall -Wno-maybe-uninitialized  -Wno-unused-dummy-argument -Wno-c-binding-type -fbacktrace -fcheck=all -fbounds-check -finit-real=snan -ffpe-trap=zero,invalid"
  export FCFLAGS="$CFLAGS -Wall -Wno-maybe-uninitialized  -Wno-unused-dummy-argument -Wno-c-binding-type -fallow-argument-mismatch -fallow-invalid-boz"
  export CXXFLAGS="$CFLAGS"
else
  echo "Compiler $compiler unknown."
  exit 1
fi

module load gsl hdf5-serial netcdf-serial libxc cgal boost/1.74 elpa/mpi/standard/2021.05.001
#module load metis parmetis
# for large grid support
module load metis-64 parmetis-64
if [[ $compiler == intel ]]; then
  # gcc 10 or later needed for newer c++ libraries
  module load gcc/11
fi
export CXXFLAGS="$CXXFLAGS -std=c++14"
export LDFLAGS="-Xlinker -rpath=$MKL_HOME/lib/intel64:$GSL_HOME/lib:$NETCDF_HOME/lib:$ELPA_HOME/lib:$METIS_HOME/lib:$PARMETIS_HOME/lib"

if [[ $cuda == yes ]]; then
  module load cuda/11.4
  # for CUDA-aware MPI:
  if [[ $mpi == openmpi ]]; then
    module load openmpi_gpu/4
  fi
  CUDA_FLAGS="--enable-cuda --enable-nvtx --with-cuda-prefix=$CUDA_HOME"
  LDFLAGS="$LDFLAGS:$CUDA_HOME/lib64"
else
  CUDA_FLAGS=""
fi
module list

export INSTALLDIR=$PWD/installed

if [[ $mpi == impi ]]; then
  ENABLE_MPI=--enable-mpi
  SCALAPACK=$MKL
elif [[ $mpi == openmpi ]]; then
  ENABLE_MPI=--enable-mpi
  SCALAPACK=$MKL
else
  ENABLE_MPI=
  SCALAPACK=no
  PARMETIS_HOME=no
fi

export FCFLAGS_ELPA=$ELPA_FCFLAGS
export LIBS_ELPA=$ELPA_LIBS

if [[ "$1" != "noconfig" ]]; then
  pushd .. && autoreconf -i && popd  
  ../configure $CUDA_FLAGS \
    FCFLAGS_FFTW="-I$MKLROOT/include/fftw" \
    FCCPP="cpp -ffreestanding" \
    --prefix=$INSTALLDIR \
    $ENABLE_MPI \
    --enable-openmp \
    --disable-gdlib \
    --enable-shared --disable-static \
    --with-gsl-prefix="$GSL_HOME" \
    --with-libxc-prefix="$LIBXC_HOME" \
    --with-blas="$MKL" \
    --with-lapack="$MKL" \
    --with-blacs="$SCALAPACK" \
    --with-scalapack="$SCALAPACK" \
    --with-netcdf-prefix="$NETCDF_HOME" \
    --with-metis-prefix="$METIS_HOME" \
    --with-parmetis-prefix="$PARMETIS_HOME" \
    --with-boost="$BOOST_HOME" \
    --with-elpa-prefix="$ELPA_HOME" \
    || exit 1
fi

echo "\n\nBuilding octopus...\n"

if [[ "$2" == "nodep" ]]; then
  NODEP=1
else
  NODEP=0
fi
make NODEP=$NODEP -j20 && make -j20 NODEP=$NODEP install || exit 1

mkdir -p $INSTALLDIR/.build.doc/
cp -f config.log $INSTALLDIR/.build.doc/
cp -f $0 $INSTALLDIR/.build.doc/

echo "... done"

Previous Tutorial:Running Octopus on Graphical Processing Units (GPUs) - Next Manual:Appendix:Porting Octopus and Platform Specific Instructions

Back to Manual

Retrieved from "https://octopus-code.org/mediawiki/index.php?title=Manual:Specific_architectures&oldid=11584"