Developer information¶
This page contains information about GEMSEO development and how to contribute to it. The source code of GEMSEO is available on gitlab, this is the place where code contributions shall be submitted. Note also that it is required to accompany any contribution with a Developer Certificate of Origin, certifying that the contribution is compatible with GEMSEO software licence.
We aim to have industrial quality standards for software, in order to:
have a good and running software,
be confident about it,
facilitate collaborative work with the team,
facilitate distribution to our partners.
To meet these goals, we use best practices described below, these practices are not optional, they are fully part of the development job.
Quick start¶
First time setup:
Clone the repository:
git clone https://gitlab.com/gemseo/dev/gemseo.git
Install the Requirements.
From the root of the git clone, run the tests for Python 3.9 and create a development environment under
.tox/py39
:
tox -e py39 --develop
Run the checks:
tox -e check
Environments¶
We use tox for handling the environments related to the development, be it for coding, testing, documenting, checking … This tool offers a simplified and high level interface to many ingredients used in development, while providing reproducible and isolated outcomes that are as much independent as possible of the platform and environment from which it is used.
All the settings of tox are defined in the file tox.ini
.
It contains the descriptions of the environments:
version of Python to use,
packages to install,
environment variables to set or pass from the current environment,
commands to execute.
All the directories created by tox
are stored under .tox
next to tox.ini
.
In particular,
.tox
contains the environments
in directories named after the environments.
Requirements¶
Make sure Python 3 is installed, preferably 3.9.
Install pipx first:
python -m pip install --user pipx
python -m pipx ensurepath
You may need to log out and back in for the system path update to be taken into account.
Then install tox and pre-commit:
pipx install tox
pipx install pre-commit
Finally, make sure that graphviz is installed (for rendering graphs).
How to use tox¶
The environments created by tox and their usage are described in the different sections below. In this section we give the common command line usages and tips.
Create and execute the environment named env and run its commands with:
tox -e env
The first invocation of this command line may take some time to proceed, further invocations will be faster because tox shall not create a new environment from scratch unless, for instance, some of the dependencies have been modified.
You may run (sequentially) more than one environment with:
tox -e env,env2,env3
Recreate an existing environment with:
tox -e env -r
This may be necessary if an environment is broken or if tox cannot figure out that a dependency has been updated (for instance with dependencies defined by a git branch).
Activate the tox environment named env with:
On Linux and MacOS:
source .tox/env/bin/activate
On Windows:
.tox\env\Scripts\activate.bat
Activating environments may be useful for instance
to investigate a particular issue that happens
in a specific environment and not others.
You may modify an activated environment
just like any other environment,
in case of trouble just recreate it.
Be aware that the environment variables defined in tox.ini
will not be set with a manually activated environment.
Show available environments with:
tox -a
Use a double --
to pass options to an underlying command,
for example:
tox -e env -- ARG1 --opt1
Not all the environments allow this feature, see the specific topics below for more information.
Coding¶
Coding environment¶
Create a development environment:
tox -e py39 --develop
This will create an environment based on Python 3.9 with GEMSEO installed in editable mode, With an editable installation, GEMSEO appears installed in the development environment created by tox, but yet is still editable in the source tree.
Note
You do not need to activate this environment for coding into GEMSEO.
Coding Style¶
We use the pep8 convention. The checking and formatting of the source code is done with ruff. A git commit shall have no checkers violations.
All these tools are used:
either automatically by the git hooks when creating a commit,
or manually by running tox -e check.
Coding guidelines¶
- String formatting
Do not format strings with + or with the old printf-style formatting: format strings with
format()
(documentation).- Logging
Loggers shall be defined at module level and named after the module with:
LOGGER = logging.getLogger(__name__)
This means that logger names track the package/module hierarchy, and it’s intuitively obvious where events are logged just from the logger name.
- Error messages
Error messages will be read by humans: they shall be explicit and valid sentences.
- Enumerations
Use
StrEnum
from thestrenum
package for creating collections of constants that are compatible with strings. This allows to easily work with non-Python API like REST.
Git¶
Workflow¶
We use the gitflow for managing git branches. For the daily work, this basically means that evolutions of GEMSEO are done in feature branches created from the develop branch and merged back into it when finished.
Initial setup¶
Create your fork of the gemseo repository on gitlab.com.
Clone your fork to your local machine:
git clone <url of your fork>
Go to the directory of your fork.
Add the reference upstream repository to you fork with:
git remote add upstream git@gitlab.com:gemseo/dev/gemseo.git
Get access to the IRT CI:
from your account on gitlab.com,
go to Settings > CI/CD and expand the Runners section,
under Specific runners, copy the registration token and send it to a maintainer.
Working on a new feature¶
Update your local copy of the upstream repository:
git fetch upstream
Create a new feature branch on your local clone from the up to date upstream develop branch:
git checkout upstream/develop -b my_new_feature_branch
Add commits to your feature branch.
On a regular basis (ideally everyday), keep your feature branch up to date with the upstream evolution of the develop branch so to make the future merge into develop easier:
git fetch upstream
git rebase upstream/develop
When rebasing turns to be to cumbersome, you may use merge:
git rebase --abort
git merge upstream/develop
Push your current local feature branch to your fork at least once a day:
git push origin HEAD
Once pushed, the gitlab CI will run the tests on your branch, you will receive an email notification in case of failure.
Finishing a feature¶
When your feature branch is ready to be merged in the upstream develop branch, your branch shall become a merge request (MR).
If applicable, add a changelog fragment that will be later inserted into the changelog. To do so, create one or more files named after the issue number and kind of change (added, changed, deprecated, fixed, removed or security), for instance
123.fixed.rst
, inchangelog/fragments
.How to create a MR.
Assign the MR to a maintainer (AntoineD by default) which will handle the choice of the reviewers (discussed during the scrum meeting).
Set the milestone.
Set the issue relating or closing the MR, if any.
If for some reasons the branch of the MR requires more work, the MR may be set to Draft.
If a review discussion goes beyond the scope of a branch, one or more review threads of a MR may be turned into a new issue to be resolved in a future branch.
If a review thread has not been resolved by a new commit to the reviewed branch and shall not be dealt with in a new issue, it shall be marked as resolved by the reviewer.
If changes have been pushed to the branch of a MR, the reviewers shall be notified.
When all the MR discussion threads are resolved:
The reviewers shall approve the MR,
The MR creator shall ask the branch to be merged.
Reviewing a MR¶
You can choose how the changes of the MR branch are displayed.
You may leave reviews or comments on one or more lines.
You may make code suggestions that could be committed as is the reviewed branch.
Once done, you shall submit your review.
You shall check that your review comments have been addressed, if so you shall mark them as resolved.
When all the reviews have been resolved, you shall approve the MR.
Git hooks¶
When a commit is being created, git will perform predefined actions:
remove the trailing whitespaces,
fix the end of files,
check toml, yaml and json files are well formed,
check that no big file is committed,
check bad symbolic links,
check or fix some of the python docstrings formatting,
fix the Python import order,
fix the Python code formatting,
check for Python coding issues (see Coding Style),
fix some of the above coding issues.
fix outdated Python syntax,
check the commit message (see Commit message),
check for forbidden
print()
usage,check for misused
logging
formatting,check for
.rst
files issues.check or fix license headers,
check for docstrings formatting,
check for docstrings coverage,
Those actions will eventually modify the files about to be committed. In this case your commit is denied and you have to check that the modifications are OK, then add the modifications to the commit staged files before creating the commit again.
Commit message¶
We use conventional commits for writing clear and useful git commit messages. The commit message should be structured as follows:
<type>(optional scope): <description>
[optional body]
[optional footer(s)]
Where:
<type> defines the type of change you are committing
feat: A new feature
fix: A bug fix
docs: Documentation only changes
style: Changes that do not affect the meaning of the code
refactor: A code change that neither fixes a bug nor adds a feature
perf: A code change that improves performance
test: Adding missing tests or correcting existing tests
build: Changes that affect the build system or external dependencies
ci: Changes to our CI configuration files and scripts
(optional scope) provide additional contextual information and is contained within parentheses
<description> is a concise description of the changes, imperative, lower case and no final dot
[optional body] with the motivation for the change and contrast this with previous behavior
[optional footer(s)] with information about Breaking Changes and reference issues that this commit closes
You may use commitizen to easily create commits that follow conventional commits. Install it with:
pip install commitizen --user
Run it and and let it drive you through with:
cz commit
Commit message examples:
feat(study): open browser when generating XDSM
fix(scenario): xdsm put back filename arg
Commit best practices¶
The purpose of these best practices is to ease the code reviews, commit reverting (rollback changes) bisecting (find regressions), branch merging or rebasing.
- Write atomic commits
Commits should be logical, atomic units of change that represent a specific idea as well as its tests. Do not rename and modify a file in a single commit. Do not combine cosmetic and functional changes in a single commit.
- Commits history
Try to keep the commit history as linear as possible by avoiding unnecessary merge commit. When possible, prefer rebasing over merging, git can help to achieve this with:
git config pull.rebase true git config rerere.enabled true
- Rework commit history
You may reorder, split or combine the commits of a branch. Such history modifications shall be done before the branch has been pushed to the main repository.
- Tests
Avoid commits that break tests, only push a branch that passes all the tests for py39 on your machine.
Testing¶
Testing is mandatory in any engineering activity, which is based on trial and error. All developments shall be tested:
this gives confidence to the code,
this enables code refactoring with mastered consequences: tests must pass!
Tests writing guidelines¶
We use pytest for writing and executing all the GEMSEO tests. Older tests were written with the unittest module from the Python standard library but newer tests shall be written with pytest.
- Logic
Follow the Arrange, Act, Assert, Cleanup steps by splitting the testing code accordingly. Limit the number of assertions per test functions in a consistent manner by writing more test functions. Use the pytest fixtures or import the GEMSEO ones in a _conftest.py_ file:
from gemseo.utils.pytest_conftest import skip_under_windows
Tests shall be independent, any test function shall be executable alone.
- Logging
Do no create loggers in the tests, instead let pytest manage the logging and use its builtin features. Some pytest logging settings are already defined in
pyproject.toml
.- Messages
The information provided to the user by the error and logging messages shall be correct. Use the caplog fixture for checking the logging messages. Use pytest.raises for checking the error messages.
- Skipping under Windows
Use the pytest marker like:
@pytest.mark.skip_under_windows def test_foo():
- Validation of images
For images generated by matplotlib, use the
image_comparison
decorator provided by the matplotlib testing tools. Seetests/post/dataset/test_surfaces.py
for an example. When image comparison fails, set the environment variableGEMSEO_KEEP_IMAGE_COMPARISONS
such that theresult_images
directory with the comparisons is available at the root of the repository.- Validation of arrays
For NumPy arrays, use the NumPy testing tools.
- Generated files
Tests that create files shall use the
tmp_wd
fixture such that the files are created in a temporary directory instead of polluting the root directory.
Executing tests¶
For Python 3.9, run the tests with:
tox -e py39
Replace py39 by py310 for testing with Python 3.10.
With tox,
you can pass options to pytest after --
,
for instance:
tox -e py39 -- --last-failed --step-wise
Run the tests for several Python versions with for instance (on Linux):
tox -e py39,py310,py311
Tests coverage¶
For a selected python version, get the coverage information with:
tox -e py39-coverage
See pytest-cov for more information.
Documentation¶
The documentation of the develop branch is available online: develop documentation.
Generating the doc¶
The documentation is written with sphinx. On Linux, generate the documentation with:
tox -e doc
Pass options to sphinx-build
after --
,
for instance:
tox -e doc -- -vv -j2
Writing guidelines¶
Documenting classes, functions, methods, attributes, modules, etc… is mandatory. End users and developers shall not have to guess the purpose of an API and how to use it.
Style¶
Use the Google Style Docstrings format for documenting the code. This Example Google Style Docstrings shows how to write such docstrings.
Type hints¶
The type hints are used when generating the functions and methods documentation, they will also be used gradually to check and improved the code quality with the help of a type checker like mypy. See Example Google Style Docstrings for a typical example.
Functions and methods arguments shall use standard duck typing.
In practice, use Iterable
or Sequence
etc…
instead of list
when appropriate,
similarly for Mapping
instead of dict
.
For *args
and **kwargs
arguments,
use only the value types with no container.
Return types shall match exactly the type of the returned object.
Type hinting may cause circular imports,
if so, use the special constant TYPE_CHECKING
that’s False
by default
and True
when type checking:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from gemseo import create_discipline
Line feeds¶
Use semantic line feeds by starting a new line at the end of each sentence, and splitting sentences themselves at natural breaks between clauses, a text file becomes far easier to edit and version control. You can have a look at the current page’s source for instance.
Example¶
Have a look to the uncertainty module for an example of proper code documentation.
Check that the examples run correctly with:
tox -e py39 -- tests/test_doc_examples.py -m doc_examples
Versioning¶
We use semantic versioning for defining the version numbers of GEMSEO. Given a version number MAJOR.MINOR.PATCH, we increment the:
MAJOR version when we make incompatible API changes,
MINOR version when we add functionality in a backwards compatible manner, and
PATCH version when we make backwards compatible bug fixes.
Benchmarking¶
Use pyperf to create valid benchmark, mind properly tuning the system for the benchmark (see the docs).
Profiling¶
The Python standard library provides a profiler, mind using it with controlled system like for benchmarking. The profiling data could be analyzed with one of these tools:
kcachegrind, after having converted the profiling data with pyprof2calltree
Configure PyCharm¶
PyCharm is one of the best tools for writing Python code. We provide some configuration files to help configuring it for developing GEMSEO.
Code style¶
Configure PyCharm to match the code style used by GEMSEO.
Download this file
,
open the PyCharm settings,
go to Editor > Code Style > Python
and
select Import Scheme...
:
Check and format¶
Some tools used by the Git hooks can be executed in order to be notified of code issues earlier and avoid having to fix files when creating a commit.
Install the Ruff
plugin by opening the PyCharm settings,
and searching in Plugins > Marketplace
.
Then,
activate all the options and
provide the path to the ruff
executable
that shall be in the following directory relative
to the root of the git clone of gemseo:
On Linux and MacOS:
.tox/check/bin/ruff
.On Windows:
.tox\check\Scripts\ruff.exe
.
Warning
For AutoPyDiscipline
functions,
ruff
will refactor the return
line in an incompatible manner.
You shall append # noqa: RET504
to the return
line.
Environment variables¶
Configure PyCharm so that the test environments do not open graphical windows during test execution:
Click on Run > Edit Configurations… in the main menu.
Click
on Edit configuration templates… on the bottom left and then on Python tests > pytest in the tree on the left to set the environment variables for all the Python test environments,
or on Python tests > {configuration name} to set the environment variables for a specific Python test environment.
Open the section Configuration > Environment on the right.
Write MPLBACKEND=AGG in the text field Environment variables (or click on the button in this field and add a new environment variable with MPLBACKEND as name and AGG as value).
Configure VSCode¶
vscode could serve as an alternative to PyCharm.
To configure it for developing GEMSEO,
we offer the base settings.json
and extensions.json
,
which need to be placed within the local .vscode
directory.
Download Configuration Files¶
Installation¶
Place both downloaded files in the .vscode
directory of your project.
Configuration¶
Modify settings.json
according to your preferences.
You can adjust the settings either
globally (User parameters)
or per project (Workspace parameters).
Extensions¶
Ensure you install all recommended extensions mentioned in extensions.json
.
These extensions enhance
the functionality and productivity
of vscode for Python development.