# Developer information¶

This page contains information about GEMSEO development and how to contribute to it. The source code of GEMSEO is available on gitlab, this is the place where code contributions shall be submitted. Note also that it is required to accompany any contribution with a Developer Certificate of Origin, certifying that the contribution is compatible with GEMSEO software licence.

We aim to have industrial quality standards for software, in order to:

• have a good and running software,

• facilitate collaborative work with the team,

• facilitate distribution to our partners.

To meet these goals, we use best practices described below, these practices are not optional, they are fully part of the development job.

## Quick start¶

First time setup:

• Create a main environment with tox-conda, see Requirements.

• then on linux

tox -e dev

• or on windows

tox -e dev-win


Run the tests

• on linux

tox -e py27,py37

• on windows

tox -e py27-win,py37-win


## Environments¶

We use tox for handling the environments related to the development, be it for coding, testing, documenting, checking … This tool offers a simplified and high level interface to many ingredients used in development, while providing reproducible and isolated outcomes that are as much independent as possible of the platform and environment from which it is used.

All the settings of tox are defined in the file tox.ini. It contains the descriptions of the environments:

• version of python to use,

• packages to install,

• environment variables to set or pass from the current environment,

• commands to execute.

All the directories created by tox are stored under .tox next to tox.ini. In particular, .tox contains the environments in directories named after the environments.

### Requirements¶

We use tox and tox-conda along with anaconda, you need to have them installed before moving along. Create an anaconda environment with tox-conda:

conda create -n tox python=3.8 pip
conda activate tox
pip install tox-conda
conda deactivate
conda activate tox


The last two commands are necessary to have the tox executable available in the just created environment.

### How to use tox¶

The environments created by tox and their usage are described in the different sections below. In this section we give the common command line usages and tips.

Create the environment named env and run its commands with:

tox -e env


The first invocation of this command line may take some time to proceed, further invocations will be faster because tox shall not create a new environment from scratch unless, for instance, some of the dependencies have been modified.

You may run (sequentially) more than one environment with:

tox -e env,env2,env3


Recreate an existing environment with:

tox -e env -r


This may be necessary if an environment is broken or if tox cannot figure out that a dependency has been updated (for instance with dependencies defined by a git branch).

We use tox with anaconda environments, activate the tox environment named env with:

conda activate .tox/env


Note

An anaconda environment created by tox has no anaconda name, thus conda cannot activate it by its name as usual.

Activating environments may be useful for instance to investigate a particular issue that happens in a specific environment and not others. You may modify an activated environment just like any other anaconda environment, in case of trouble just recreate it. Be aware that the environment variables defined in tox.ini will not be set with an manually activated environment.

Show available environments with:

tox -a


Use a double -- to pass options to an underlying command, for example:

tox -e env -- ARG1 --opt1


Not all the environment allows this feature, see the specific topics below for more information.

Note

On windows, the environment names shall be suffixed with -win. This is a limitation of tox.

## Coding¶

### Coding environment¶

Create the development environment:

tox -e dev


tox -e dev-win


This will create an environment with:

• GEMSEO installed in editable mode,

• all the GEMSEO dependencies,

• tools used for development (debugging, code checking and formatting)

• git settings (see Git)

With an editable installation, GEMSEO appears installed in the development environment created by tox, but yet is still editable in the source tree.

Note

You do not need to activate this environment for contributing to GEMSEO.

### Coding Style¶

We use the pep8 convention. The formatting of the source code is done with isort and black. The code is systematically checked with flake8 and on demand with pylint. A git commit shall have no flake8 violations.

Except for pylint, all these tools are used:

• either automatically by the git hooks when creating a commit,

• or manually by running tox -e style.

Use tox -e pylint to run pylint.

### Coding guidelines¶

String formatting

Do not format strings with + or with the old printf-style formatting: format strings with format() (documentation).

Logging

Loggers shall be defined at module level and named after the module with:

LOGGER = logging.getLogger(__name__)


This means that logger names track the package/module hierarchy, and it’s intuitively obvious where events are logged just from the logger name.

## Git¶

### Workflow¶

We use the gitflow for managing git branches. For the daily work, this basically means that evolutions of GEMSEO are done in feature branches created from the develop branch and merged back into it when finished. The feature branches shall be prefixed with ft_.

### Git hooks¶

When a commit is being created, git will perform predefined actions:

• remove the trailing whitespaces,

• fix the end of files,

• check toml, yaml and json files are well formed,

• check that no big file is committed,

• check or fix some of the python docstrings formatting,

• fix the python import order,

• fix the python code formatting,

• check for python coding issues (see Coding Style),

• check the commit message (see Commit message),

• check for forbidden print() usage,

• check for misused logging formatting,

• check for .rst files issues.

Those actions will eventually modify the files about to be committed. In this case your commit is denied and you have to check that the modifications are OK, then add the modifications to the commit staged files before creating the commit again.

### Commit message¶

We use conventional commits for writing clear and useful git commit messages. The commit message should be structured as follows:

<type>(optional scope): <description>

[optional body]

[optional footer(s)]


Where:

• <type> defines the type of change you are committing

• feat: A new feature

• fix: A bug fix

• docs: Documentation only changes

• style: Changes that do not affect the meaning of the code

• refactor: A code change that neither fixes a bug nor adds a feature

• perf: A code change that improves performance

• test: Adding missing tests or correcting existing tests

• build: Changes that affect the build system or external dependencies

• ci: Changes to our CI configuration files and scripts

• (optional scope) provide additional contextual information and is contained within parenthesis

• <description> is a concise description of the changes, imperative, lower case and no final dot

• [optional body] with the motivation for the change and contrast this with previous behavior

• [optional footer(s)] with information about Breaking Changes and reference issues that this commit closes

From the .tox/dev environment, you may use commitizen to easily create commits that follow conventional commits. Run it and and let it drive you through with:

cz commit


Commit message examples:

feat(study): open browser when generating XDSM

fix(scenario): xdsm put back filename arg


### Commit best practices¶

The purpose of these best practices is to ease the code review, commit reverting (rollback changes) bisecting (find regressions), branch merging or rebasing.

Write atomic commits

Commits should be logical, atomic units of change that represent a specific idea as well as its tests. Do not rename and modify a file in a single commit. Do not combine cosmetic and functional changes in a single commit.

Commits history

Try to keep the commit history as linear as possible by avoiding unnecessary merge commit, git can help to achieve this with (this is done automatically when the .tox/dev environment is created):

git config pull.rebase true

Rework commit history

You may reorder, split or combine the commits of a branch. Such history modifications must be done before the branch has been pushed to the main repository.

Tests

Avoid commits that break tests, only push a branch that passes all the tests for py27 and py38 on your machine.

## Testing¶

Testing is mandatory in any engineering activity, which is based on trial and error. All developments shall be tested:

• this gives confidence to the code,

• this enables code refactoring with mastered consequences: tests must pass!

### Tests writing guidelines¶

We use pytest for writing and executing all the GEMSEO tests. Older tests were written with the unittest module from the python standard library but newer tests shall be written with pytest.

Logic

Follow the Arrange, Act, Assert, Cleanup steps by splitting the testing code accordingly. Limit the number of assertions per test functions in a consistent manner by writing more test functions. Use the pytest fixtures features. Tests shall be independent, any test function shall be executable alone.

Logging

Do no create loggers in the tests, instead let pytest manage the logging and use its builtin features. Some pytest logging settings are already defined in pyproject.toml.

Messages

The information provided to the user by the error and logging messages shall be correct. Use the caplog fixture for checking the logging messages. Use pytest.raises for checking the error messages.

Skipping under windows

Use the pytest marker like:

@pytest.mark.skip_under_windows
def test_foo():

Validation of images

For images generated by matplotlib, use the image_comparison() decorator provided by the matplotlib testing tools. See tests/post/dataset/test_surfaces.py for an example.

### Executing tests¶

For python 2.7, run the tests with:

tox -e py27


Replace py27 by py38 for testing with python 3.8, you may run the tests accordingly with python 2.7, 3.6, 3.7 and 3.8. With tox, you can pass options to pytest after --, for instance:

tox -e py38 -- --last-failed --step-wise


Run the tests for several python versions with for instance:

tox -e py27,py38


Under windows, append -win to the names of the test environments, for instance:

tox -e py27-win,py38-win


### Tests coverage¶

For a selected python version (for instance python 3.8), get the coverage information with:

tox -e py38 -- --cov --cov-report=term


See pytest-cov_ for more information.

## Documentation¶

### Generating the doc¶

The documentation is written with sphinx. Generate the documentation with:

tox -e doc


Under windows, append -win to the names of the this environment.

Pass options to sphinx-build after --, for instance:

tox -e doc -- -vv -j2


Check the links in the generated documentation with:

tox -e doc-linkchecker


Note

doc-linkchecker does not work on windows.

### Writing style¶

Use semantic linefeeds by starting a new line at the end of each sentence, and splitting sentences themselves at natural breaks between clauses, a text file becomes far easier to edit and version control. You can give a look at the current page’s source for instance.

## Versioning¶

We use semantic versioning for defining the version numbers of GEMSEO. Given a version number MAJOR.MINOR.PATCH, we increment the:

1. MAJOR version when we make incompatible API changes,

2. MINOR version when we add functionality in a backwards compatible manner, and

3. PATCH version when we make backwards compatible bug fixes.