Developers:BuildBot

From OctopusWiki
Jump to: navigation, search

Generalities

Buildbot is a Python program to automate software builds and tests. It operates a server that triggers jobs on a number of workers. These workers may be running on the same machine as the server but also on different ones. This opens the possibility to compile and check on different architectures and operating systems. More information about Buildbot can be found in the manual.

The server, which in the Buildbot terminology is called a master, is running on octopus-code.org. We also have several dedicated machines, the workers, that take care of running the builders. Each builder performs a predefined set of tasks. These tasks consist mainly in compiling the source code and running the testsuite, but there are also builders to perform more specific tasks, like generating the code documentation. The builders are triggered by schedulers.

Schedulers

There are several schedulers that are configured, each one triggering different types of builders under different circumstances:

  • The branch_push scheduler: it is triggered whenever there is a push event or tag event. It triggers the tests builders.
  • The nightlyPT scheduler: every night, if any changes have been made to the develop branch, this scheduler triggers the nighthly_tests builder.

Whenever the tests or nighthly_tests builders fail, an e-mail is sent to octopus-notify@tddft.org. This e-mail contains a link to the output of all commands run by the builder. In addition to this, the tests builder reports its result back to GitLab, so one can easily keep track of the status of the tests for each branch. This is particularly useful when deciding whether to accept or not a merge request.

Workers

A list of all the workers can be found in the workers web page. Further information, like the location, OS, etc, of each worker can be obtained by clicking on the worker name.

Builders

A complete list of all the builders can be found in the builders web page.

Among the builders, two of them have the sole purpose of triggering other builders:

  • The tests builder: depending on the event and the branch that triggered this builder, it will in turn trigger different builders.
  • The nighlty_tests builder: it starts several builders that compile the code and run the testsuite (see table bellow).

The builders that perform special tasks (i.e., that do not run the testsuite) are the following:

  • The documentation builder. This builder generates the variables info and the doxygen source code documentation and uploads it to octopus-code.org. It is triggered when there is a push to the 'develop' and release branches, or when a new tag is created.
  • The dist builder. This builder generates a tarball with the source code and uploads it to octopus-code.org. It is triggered when there is a push to the a release branches or when a new tag is created.

The following table contains a list of all the remaining builders that run the tests, and these are triggered for all push events but not when a new tag is created:

Builder/Option CC CFLAGS FC FCFLAGS MPI MPI2 OpenMP OpenMP/SIMD cuda OpenCL --disable-debug maxdim Arpack BerkeleyGW Blacs ELPA ETSF_IO FEAST GDLIB LibFMM Libxc 3+ Libxc 4+ Netcdf NFFT NLOPT PARMETIS Parpack PFFT PNFFT poke pspio Scalapack Sparskit Ext. LibISF YAML CLBLAS CLFFT cuBLAS cuFFT
distcheck gcc -O0 gfortran -O0 -ffree-line-length-none no no no no no no no 3 no no no no no no no no yes no no no no no no no no no no no no no no no no no no
foss4.9 gcc -std=gnu99 -Wall -O2 -march=native gfortran -Wall -ffree-line-length-none -O2 -march=native -fbacktrace no no no no no no no 3 yes yes no yes yes no yes no yes no yes yes yes no no no no yes yes no yes no no no no no no
foss4.9_mpi mpicc -std=gnu99 (gcc) -Wall -O2 -march=native mpifort (gfortran) -Wall -ffree-line-length-none -O2 -march=native -fbacktrace yes yes no no no no no 3 yes yes yes yes yes no yes no yes no yes yes yes yes yes yes yes yes yes yes yes yes no no no no no
foss_debug gcc -Wall -O2 -march=native -Wextra -pedantic gfortran -Wall -ffree-line-length-none -O2 -march=native -fbacktrace -fcheck=all -fbounds-check no no no no no no no 4 yes yes no yes yes no yes no yes no yes yes yes no no no no yes yes no yes no no no no no no
foss_min gcc -Wall -O2 -march=native gfortran -Wall -ffree-line-length-none -O2 -march=native -fbacktrace no no no no no no no 3 no no no yes no no no no yes no no no no no no no no no no no no no no no no no no
foss_mpi_debug mpicc (gcc) -Wall -O2 -march=native -Wextra -pedantic mpifort (gfortran) -Wall -ffree-line-length-none -O2 -march=native -fbacktrace -fcheck=all -fbounds-check yes yes no no no no no 4 yes yes yes yes yes no yes no yes no yes yes yes yes yes yes yes yes yes yes yes yes no no no no no
foss_mpi_min mpicc (gcc) -Wall -O2 -march=native mpifort (gfortran) -Wall -ffree-line-length-none -O2 -march=native -fbacktrace yes yes no no no no no 3 no no no yes no no no no yes no no no no yes no no no no no no no no no no no no no
foss_mpi_opt mpicc (gcc) -Wall -O3 -march=native -funroll-loops mpifort (gfortran) -Wall -ffree-line-length-none -O3 -march=native -fbacktrace -funroll-loops yes yes no no no no no 3 yes yes yes yes yes no yes no yes no yes yes yes yes yes yes yes yes yes yes yes yes no no no no no
foss_opt gcc -Wall -O3 -march=native -funroll-loops gfortran -Wall -ffree-line-length-none -O3 -march=native -fbacktrace -funroll-loops no no no no no no no 3 yes yes no yes yes no yes no yes no yes yes yes no no no no yes yes no yes no no no no no no
gnu4.9_valgrind gcc -std=gnu99 -g -O2 -march=native gfortran -I/usr/lib64/gfortran/modules -g -ffree-line-length-none -O2 -march=native no no no no no no no 4 yes no no no yes no no no no no yes no no no no no no no no no yes no no no no no no
gnu_cl_amd gcc -std=gnu99 -O3 -funroll-loops -ffast-math -I/opt/clFFT/include/ -I/opt/clBLAS/include/ gfortran -O3 -funroll-loops -ffast-math -fbounds-check -ffree-line-length-none no no no no no yes no 3 no no no no yes no no no no no yes no no no no no no no no no no no no yes yes no no
intel icc -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall ifort -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all no no no no no no no 4 yes yes no yes yes no no no yes no yes no yes no no no no no yes no yes no no no no no no
intel-2017a icc -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall ifort -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all no no no no no no no 4 yes yes no yes yes no no no no yes yes no yes no no no no no yes no yes no no no no no no
intel-2017a_impi mpiicc (icc) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall mpiifort (ifort) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all yes yes no no no no no 4 yes yes yes yes yes no no no no yes yes no yes yes yes no no no yes yes yes yes no no no no no
intel_impi mpiicc (icc) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall mpiifort (ifort) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all yes yes no no no no no 4 yes yes yes yes yes no no no yes no yes no yes yes yes no no no yes yes yes yes no no no no no
intel_impi_omp mpiicc (icc) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall mpiifort (ifort) -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all -qopenmp -qsmp=omp yes yes yes no no no no 4 yes yes yes yes yes no no no yes no yes no yes yes yes no no no yes yes yes yes no no no no no
intel_omp icc -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -Wall ifort -O3 -xHost -ftz -fp-speculation=safe -fp-model source -traceback -warn all -qopenmp -qsmp=omp no no yes no no no no 4 yes yes no yes yes no no no yes no yes no yes no no no no no yes no yes no no no no no no