Writing tutorials

The Octopus tutorials are written as Jupyter Notebooks and converted to static HTML using Sphinx and myst-nb. Notebooks consist of a mix of markdown cells, in which MyST markdown can be used, and code cells.

Writing a tutorial

Organizing content and naming conventions

All tutorials are located in the octopus repository under doc/jupyter_tutorials/<index>-<category>. Tutorials are grouped into different topics (indicated by category). Tutorials should be self-contained as much as possible and not require any additional files.

Each notebook should have a name <index>-<tutorial name>.ipynb, the <index> (starting at 1) determines the tutorial order per group of tutorials.

Octopus input/output must not be written into the base directory containing the notebook, instead a subdirectory <index>-<tutorial name> should be created at the top of each notebook (when performing multiple calculations, additional subdirectories inside that base directory can be created and used as needed).

Each notebook should start with a single level-1 heading (a single # in markdown syntax), which is the tutorial title. The title should be descriptive but also concise (as it appears in a navigation sidebar with limited space on the webpages). All sections in the notebook should be (at least) level-2 headings (double ## in markdown syntax).

Editing tutorials

The easiest way to work on tutorials is to open them in Jupyter lab. The most convenient option to get all required Python dependencies (without also compiling Octopus as part of the installation process) is using uv: in the repo root or a tutorial subdirectory run uv run --only-group tutorial-dev jupyter lab. Ensure that octopus (and an MPI implementation) are in your PATH. You can also use a Docker image with Octopus via podman run -it -v=$(pwd):/io -p 8888:8888 registry.gitlab.com/octopus-code/octopus:main bash (executed in the repository root) and run jupyter lab via uv inside the container.

Before committing notebooks make sure to remove all outputs (or use nbstripout as pre-commit hook/git filter). A CI job will check that notebooks do not contain any output. If you accidentally commit output please remove it locally, amend the commit and force-push the changes to avoid increasing the repository size.

Creating input files

Input files required for octopus (inp, coordinates, …) should be created as part of the notebook using %%writefile magic: the content of the input file is contained in a code cell with an additional first line %%writefile <path/to/file>. inp files must not be provided as additional separate files. Benefits are (i) users can easily modify the content of an input file from within the notebook and (ii) each tutorial only requires a single file making it easier to share and distribute tutorials guaranteeing complete information.

Use %%writefile for all text input files. Some tutorials may require additional larger (binary) files. These can be committed to the repository as additional files if recreating them within a notebook is not possible.

You should generally redirect stdout/stderr to files (using suitable input options) to (i) keep the executed notebook short (and free from too much information with little value) and (ii) have them available for further use e.g. within postopus.

Executing octopus

Depending on context use one of the following two options to execute octopus:

1. For single execution use the syntax !cd <index>-<tutorial name>[/optionally/subdirectory] && octopus in a dedicated cell.

   This will temporarily change into the subdirectory containing the inp file and execute octopus in there. The working directory of the notebook remains unchanged (which improves stable and consistent execution of notebooks).

2. For calling octopus multiple times, e.g. to systematically vary a parameter in the inp file, use Python to create the required input files and use subprocess.run("octopus", cwd="<index>-<tutorial name>[/optionally/subdirectory]").

   Similar to the ! syntax, this will run octopus in the correct directory without changing the working directory of the notebook. The syntax is easier to understand withing a longer Python block and therefore preferable over ! syntax in loops.

Parallelisation

Tutorials should make use of parallelisation via MPI and/or OpenMP if sensible. Tutorials should use at most 8 CPUs (fewer if it improves performance), which can be expected as a lower limit on all reasonably modern hardware.

For MPI always use mpirun <mpi options> octopus when calling octopus (instead of orterun, prterun, …; some people may not have mpirun but they can easily set an alias; mpirun is the clearest command, in particular for inexperienced users).

To control the number of OpenMP threads, set OMP_NUM_THEADS via Python, e.g. for 2 threads use (note the use of a string on the right-hand side):

import os
os.environ["OMP_NUM_THREADS"] = "2"

After running this command, all subsequent calls will use 2 threads, it can therefore e.g. be placed at the top of a notebook.

Tutorials should ideally not require users to have a dedicated GPU, which is in particular in notebooks generally not the case. (When tutorials are used for a workshop where participants get access to GPU machines or slurm, e.g. via a Jupyter hub, it is straightforward to substitute all mpirun calls with a suitable other command/other flags in an automated fashion.)

Data analysis

Tutorials should generally make use of Postopus to support data analysis, visualisation and further postprocessing. If a file format is not yet supported please reach out to the Postopus developers and provide a concise example to create the file in question.

Tutorial validation checks

The last section of each notebook is a special validation section. It should start with the heading Tutorial Validation Checks.

The first tutorial of the Octopus basics section states its purpose:

The purpose of these checks is to automatically inform the developers if tutorials have to be updated, due to changes in Octopus. These tests are not meant to be regression tests, as all functionality is tested in the testsuite.

Please, note that the reference values are determined by the original input files in the tutorials. The tests might fail if you experiment with the input variables, e.g. to test the convergence, or try different algorithms.

The checks in this section should focus on a few key quantities that can flag up significant changes in the tutorial/octopus that affect the tutorial qualitatively, e.g. convergence in a wrong local energy minimum, no convergence at all, missing/additional peaks in a spectrum, etc.

Tolerances for these checks are on purpose not updated automatically and should be chosen suitably wide to try to ensure that these checks will pass on any computer independent of e.g. used compiler flags, while still capturing significant changes. Picking suitable quantities depends on the physics discussed in the individual tutorials.

To read octopus output, you should generally use postopus (if a quantity/file is not yet supported please open an issue). Checks use assert or numpy.testing.assert_allclose for floating-point numbers. Each cell should only contain a single assertion. The Octopus CI regularly runs all notebooks to ensure tests are up-to-date and all assertions pass.

Useful MyST syntax

In the markdown cells you can use everything that is supported by MyST markdown. They will typically only render nicely in the HTML version of the notebook but their syntax is easy to read and does not affect readability of the notebook. A few useful commands are shown in the following subsections:

Admonitions

Sphinx supports a number of specially highlighted boxes for information, warnings, notes, etc. To use them wrap the paragraph(s) in triple colons (called colon fence) and add the type to the opening fence. Inside the block you can use full MyST/mardown syntax. To nest multiple blocks vary the number of colons. Here are two examples for source code and how it is rendered (the theming will look different for notebooks).

:::{info}
A simple info block.
:::

is rendered as:

::::{warning}
An outer block with `monospace` and an info block:
:::{info}
The inner info block.
:::
::::

Equations

For equations use normal LaTeX. Equations can be wrapped in single/double dollar signs and you can use amsmath.

Code blocks

If you need to include code in a markdown cell you can use normal markdown syntax, i.e. wrapping in single backticks for inline code or surrounding by triple backticks for code blocks. The language for syntax highlighting can be specified after the opening triple backticks. If nothing is specified, sphinx will assume Python.

Static images

Static images can be included using normal markdown syntax. The size and a few other parameters can be controlled with MyST syntax (the trailing block in braces):

![alt text for figure](path/to/figure.webp){w=400px}

Images should be in png or webp format, eps can not be rendered.

You should only include images that cannot dynamically be generated from the Octopus output.

Removing cells in HTML

Cells in Jupyter notebooks can carry additional metadata (use advanced tools in the right sidebar of jupyter lab to edit the metadata). You can remove or hide cell input/output in the static html version. E.g. to remove input of a cell add the following metadata:

"mystnb": {
    "remove_code_source": true
}

Conversion to static HTML

To display a static version of the tutorials on the Octopus webpages, all notebooks are executed and subsequently converted to HTML (using myst-nb) as part of the CI.

Organization of the HTML pages

A few additional index.md files are required for the static pages. The top-level file (doc/jupyter_tutorials/index.md) contains instructions on how to execute tutorials as well as links to the different groups of tutorials.

The directory for each group contains an index.md file with some general information about this set of tutorials, a hidden toc directive (required for sphinx to collect all notebooks) and a list of tutorials with their title (inserted automatically by sphinx) and a one-line description. (Sphinx' table of contents does not support the additional description, therefore this duplication is required.)

The following example shows parts of the index.md file for the first set of tutorials:

# Octopus Basics

This is a series ...

- Lesson 1: {doc}`1-getting_started` — Learn how to run the code
- Lesson 2: {doc}`2-basic_input_options` — Obtain the ground state of the nitrogen atom.
- ...

:::{toctree}
:maxdepth: 1
:glob:
:hidden:

*
:::

When adding a new tutorial, all you have to do is add the new file and summary to the bullet point list.

Building static HTML pages locally

To build static HTML run uv run --only-group sphinx make html inside doc/jupyter_tutorials. Running this command will not re-execute notebooks and is therefore quick. Instead it will use the notebooks in whatever state they currently are. To browse the output open doc/jupyter_tutorials/_build/html/index.html in a browser (or run a webserver in doc/jupyter_tutorials/_build/html, e.g. python -m http.server; running a a webserver is generally not required to inspect the output; a few advanced options such as download buttons for notebooks may require it.).

Building static HTML pages in the CI

In the CI static pages are build in two steps:

1. Each notebook is executed inside a Docker container using an image created for the current commit. Executed notebooks containing outputs are kept as CI artifacts and passed to the second step.
2. Sphinx is executed (using the same uv run --only-group sphinx make html) and the output under _build/html is copied to the Octopus webserver.

The CI exports OCT_PARSE_ENV=1 and OCT_SilenceProgressBar=yes to suppress progress bars in the notebook output.