Starting to develop

Gitlab account

The first step is to make an account on gitlab, and get associated to the octopus-code group. The group administrator will also add you to the octopus-devel and octopus-notify mailing lists.

To get access to the repository you will also need to have an ssh key associated to your gitlab account. gitlab provides extensive documentation about ssh keys and how to generate them here .

Workflow

If you have never used git before, you should make yourself familiar with it. There are many good online tutorials about how to use git, so this should not be a problem.

The development is done following the oneflow workflow.

Cloning

Once you are part of the octopus-code group, you need to clone the main repository on your desktop

git clone git@gitlab.com:octopus-code/octopus.git

With this you get a complete copy of the Git repository that you can use for your developments.

At this point, you might want to compile and install the version from the develop branch following these instructions.

Working on the code

Before starting to implement some new feature, we recommend you create a new issue on gitlab explaining what you intent to do. The next step is to create a branch where to implement some new feature. You should always branch from the develop branch

git checkout develop
git checkout -b myfeature

You can commit your modification in two steps

git add modified_or_new_files
git commit -m "Your commit message."

If you only have modified files, the two steps can be combined into one unique command

git commit -a -m "Your commit message."

Note that the “-m” flags is not mandatory: if you do not give a message, git will open a text editor where you can type your commit message. Make sure that you give a ‘‘‘meaningful description’’’ of your changes to help other developers understand what you did. It might be a good idea to have a look at the changes that other developers made to see some examples of their messages.

Do not be afraid to change things in Octopus, you can always revert them back to the original version that you got in the beginning. This is one of the best features of using a version control system like git. You will also have the opportunity to get feedback from some of the other developers before your changes are included in the main development branch.

To share your work with the other developers, you need to send your changes to the main repository. This is done with the push command. The first time you push your branch, you need to tell git where to send your changes. In this case you want to send them to the origin , which is the default name of the original repository you cloned

git push --set-upstream origin myfeature

Git will remember this, so the next time you push your branch you will only need to do

git push

In order to prevent the testsuite from failing on code-formatting issues, it is highly recommended to run a style check before pushing any changes to a merge request. You can do that by running octopus-findent, which can be found in this repository. Instructions on how to compile and install the findent tool can be found there as well.

In most cases, your changes will be pushed to the repository without problems. However, if you have been developing in your branch for some time, it is very likely that the develop branch will have evolved as well. You need to make sure that you are keeping up to date with those changes. Otherwise it might not be possible to accept a merge request based on your branch, either because of conflicts or because the testsuite fails (more about merge requests and the testsuite bellow). So from time to time you should incorporate new changes from the develop branch into you branch by doing a git merge (another option is to use git rebase, but this requires some extra care). First switch to the develop branch and make sure it is up to date

git checkout develop
git pull

Then switch back to your branch and merge develop into it

git checkout myfeature
git merge develop

There is the possibility for conflicts between your changes and the ones in the develop branch, so read the output of the merge command carefully. If the changes in the develop branch are unrelated to your changes the merge will just go through and you can continue to make changes in your branch. If you did the merge because of a failed git push, you can try again and it should be accepted now.

In case of conflicts, the merge output it will tell you in which files the conflicts appear. Open one of these files in your favorite text editor and search for HEAD, this will mark the start of the conflicting lines. You will first see the lines from your branch and then the corresponding lines from the develop branch. You will need to make a choice which lines you keep. Depending on the situation different solutions can be appropriate, so select carefully. You then need to delete the lines that mark the conflict, like the HEAD that you initially searched for. Solve all the conflicts in the file and save your changes. Then let git know that you have resolved the conflict in this file by using

git add filename

If there is more than one file having conflicts, move on to the next one until you have resolved all conflicts in all files. Commit the new version of your branch with

git commit

It will automatically create a message saying that this commit is for merging develop into myfeature and list the files that had conflicts. If you started the whole procedure because git push failed, you can try again and it should be accepted.

Testsuite

Octopus comes with a whole set of tests in various categories. You can find them in the testsuite/ folder in your Octopus directory. You should make sure that your changes do not lead to a failure in this testsuite. When you installed the code with CMake, you can run the testsuite with:

ctest --test-dir ./cmake-build-release --print-labels
Test project /octopus/cmake-build-release
All Labels:
  components
  errors
  long-run
  short-run
...
ctest --test-dir ./cmake-build-release -L short-run
Test project /octopus/cmake-build-release
  Test   #1: components/01-derivatives_1d
  Test   #2: components/02-derivatives_2d
  Test   #3: components/03-derivatives_3d
...

For full information, see here

If you installed the code with Autotools, you can use the following commands:

make check

For greater convenience, the tests are also divided into two subsets: “long” and “short” tests. You can run these by using

make check-short

and

make check-long

The first should take about 20 minutes and the second about 40 minutes (depending on your computer, of course). If you start the tests before going for lunch, they should be done easily by the time you return, so no excuses for not running them.

If you are implementing a new feature, make sure that you also provide a test inside the testsuite so that other people do not break your code.

Any changes to existing tests ‘‘‘must’’’ be discussed with the other developers first.

For more details on the testsuite, you might want to check the [[media:Strubbe_Octopus_testsuite.pptx.pdf | talk by David Strubbe]] and the testsuite/README file in the source code.

Buildbot

Octopus has a set of computers with different architectures set up that run the tests with different compilers etc. Whenever a branch gets pushed to the main repository on gitlab, the full testsuite is ran on those computers. We use Buildbot to do this with minimal human intervention. Once the tests are done, the results are sent back to gitlab and will appear in the pipelines page. The result will also be displayed next to the corresponding commit, in the commits listing. In case something went wrong, an email will be sent to the octopus-notify mailing list, so keep an eye on it.

In case the tests appear as failed, you will probably want to know more about what went wrong. To do this, go to the pipelines page and select the pipeline corresponding to your push. Next click on the button that says “Octopus Buildbot”. This should send you to the Octopus buildbot page. You need to be logged in to see its contents. The authentication is done using your gitlab credentials.

Due to the complexity of available architectures (e.g. parallelism, use of accelerator cards) results can slightly differ from one to another computer. In order to account for those variations, the test files contain precision goals for each number. These should be set as tight as possible. We have a tool, which can analyze the different results and visualize the spread of the values, which can help to identify whether there is some real problem, giving rise to outliers on some computers. The tool can be accesses through the web app. There also exists a tutorial video.

Merge requests

Once a new feature is ready, it should be merged into the develop branch. This should be done through a merge request. The simplest way to do this is by using the gitlab UI to create a [https://gitlab.com/octopus-code/octopus/merge_requests/new new merge request]. To create the request, you need to specify myfeature as the source branch and develop as the target branch. You will also need to provide a description of the merge request. Take your time to write a good description of your new feature and the changes you made to the code, as this will greatly help the other developers to understand your work. Once the merge request has been created, another developer needs to review it and decide if it should be accepted or not. Note that one of the requirements to accept a merge request is that all the buildbot tests pass for the source branch. Another requirement is that your changes follow our [[Developers:Coding_standards|coding standards]].

You can also create a merge request in case you want to get some feedback about your work from the other developers before a new feature is ready to be merged. In that case create a merge request as explained above, but start the merge request title with WIP:.

Bug fixes

Sooner or later new bugs are found in the code that need to be fixed. If a bug is found, the first thing to do is to create a new issue explaining what the problem is. The exact workflow to fix the bug will depend whether the bug is found in the master branch or not. In any case, once the bug has been fixed, we recommend that you add a regression test, or modify an existing test, such that it returns a failure in the event that the same bug is reintroduced in the code. For more details on how to handle bug fixes and more unusual situations, please see the [[Developers:Workflow]] page.

Bugs present in develop branch, but ‘‘’not’’’ in master

In this case follow the same workflow as above to introduce new features: create a new branch from develop , fix the bug, commit your changes, push them to gitlab, and create a merge request. Make sure that in this case the branch does ‘‘’not’’’ contain the word hotfix , otherwise the buildbot will treat the branch as if it was going to be merged into the master branch, not develop .

Bugs present in master

If the bug is present in the master branch, the workflow to be followed is slightly different, because we want to make the bug fix available to users ASAP through a bug fix release. To do this, first create a new branch from the master branch. The name of this new branch should start with hotfix- to make its purpose clear. Then fix the bug, bump the Octopus release number in the configure.ac file (see the [[FAQ-How_do_the_version_numbers_in_Octopus_work.3F|FAQ]] to learn how version numbers in Octopus work), commit your changes, and push everything to gitlab. Next, create a merge request using master as the target branch. Make sure the hotfix branch is not deleted when the merge request is accepted. Once this merge request is accepted, create a new merge request with develop as the target branch. Note that it is very likely that a merge conflict arises when trying to merge into develop because of the change of release number. Make sure you fix the conflict by keeping the current release number of the develop branch. When both merge requests are accepted, a new tag corresponding to a new release should be created from the master branch.

Using a fork

If you do not have write permissions to the main git repository, you first need to fork it. This is done using the gitlab web interface and will create a new project in your gitlab account with a clone of the main repository. Most of the development cycle should be very similar to the one outlined above, namely the use of feature branches and merge requests. The main differences are explained bellow.

Note that you can always do your developments using a fork if you think this is more convenient, even if you have write permissions to the main git repository.

Cloning

Forking will create a new git repository that lives in your gitlab user space. You need to take this into account when cloning it to your desktop

git clone git@gitlab.com:my-username/octopus.git

Note the ‘‘my-username’’ in the git url. This should be replaced by the appropriate name.

Staying synchronized with the main repository

You will want to regularly get the changes made by others and accepted in merge requests. For this you need to set a pointer, called a remote, to the main repository of Octopus:

git remote add upstream git@gitlab.com:octopus-code/octopus.git

In this case we have named this pointer upstream . This is a widely used convention, but you can name it differently if you prefer. To check that the remote was set properly, do

>git remote -v
origin git@gitlab.com:my-username/octopus.git (fetch)
origin git@gitlab.com:my-username/octopus.git (push)
upstream git@gitlab.com:octopus-code/octopus.git (fetch)
upstream git@gitlab.com:octopus-code/octopus.git (push)}}

Once this is set, you can get the latest changes from the main repository

git fetch upstream

The fetch command will get the changes, but will not merge them into your local branches. Usually the local branch that you want to keep synchronized with the upstream is the develop branch. This is because you typically branch off from develop to create a feature branch (as explained above). To merge the upstream develop into the local one, do

git checkout develop
git merge upstream/develop

Since you are not supposed to directly change your local develop branch, this should not raise any conflicts.

Merge requests

Merge requests work pretty much like before, except that you need to specify that the source branch lives in your forked repository instead of the main repository.