Maintainers information

Packages upgrading

This section contains information about how and when to upgrade the packages used by GEMSEO and by the development environments.

Dependencies

GEMSEO is a library and not a self contained application, it can be installed in environments with varying and unknown constraints on the versions of its dependencies. Thus the versions of its dependencies cannot be pinneds, but a range of compatible versions shall be defined.

All the dependencies shall be defined in setup.cfg, this files does not tell where the packages will be pulled from. The dependencies could be provided by the packages repositories pypi, anaconda or conda-forge.

Getting GEMSEO to work with a common set of packages versions on several platforms and python versions is tricky and challenging. This kind of work is mostly done by trials and errors.

To reduce maintenance and complexity, our testing environments shall have the same packages providers for all the platforms and all the python versions. Furthermore it shall be identical to the references end-user environments under the same constraints.

In the context of tox, the versions of the dependencies that shall be installed with conda are defined in requirements/gemseo-conda-python2,3.txt, they are pulled from conda-forge. All other dependencies are installed with pip, they are pulled from pypi.

When a dependency is changed, setup.cfg shall always be modified. If the changed dependency is installed with conda, then both gemseo-conda-python2,3.txt and environment-py2,3.yml shall be modified.

Documentation files like CREDITS.rst and dependencies.rst shall also be updated accordingly.

CI cache

To optimize the usage of the CI cache with gitlab, the cache shall match closely the contents of the tox environments used for testing. This is currently done by defining a cache affinity with the dependencies of the environments and the environment name. The dependencies are stored in the file requirements/gemseo.in which is created automatically by a custom pre-commit hook defined in :py:`tools/extract_req_in_from_setup_cfg.py`.

Development dependencies

As opposed to the dependencies of GEMSEO, the development dependencies can be fully controlled. Thus their versions are pinned so all developers are provided with reproducible and working environments. The dependencies shall be updated at least once in a while (couple months) to benefit from packages improvements and bug fixes.

This is done with pip-tools and from input requirements files. These input requirements files contain the minimum pinning requirements and are intended to be modified by maintainers. The pip-tools package provides the pip-compile which can process an input requirements file to produce a fully pinned requirements file.

We have the following input requirements files:

  • dev.in: with development tools with python 3 only,

  • doc.in: for building the documentation with python 3 only.

To update them:

conda run -p .tox/dev pip-compile -U requirements/dev.in
conda run -p .tox/doc pip-compile -U requirements/doc.in

Note

Append -win to the environment names under windows.

Note

To reduce discrepancy among the environments, requirements/test-python3.txt shall be working for all the python 3 testing environments.

Git hooks are defined and run with pre-commit. It relies on packages that are managed with pre-commit instead of pip-tools. To update them:

conda run -p .tox/dev pre-commit autoupdate

Note

Append -win to the environment names under windows.

Warning

All environments and tools shall be checked whenever dependencies have been changed.

Test dependencies

The test dependencies are defined in setup.cfg so a end-user can easily run the GEMSEO tests.

To update them, change the test key of the [options.extras_require] section in setup.cfg, then execute

tox -e style

This will call a pre-commit hook that will update requirements/test.in. Then update the actual test requirements with:

conda run -p .tox/dev pip-compile -U requirements/test.in -o requirements/test-python3.txt
conda run -p .tox/py27 pip-compile -U requirements/test.in -o requirements/test-python2.txt

Note

Append -win to the environment names under windows.

Warning

All environments and tools shall be checked whenever dependencies have been changed.

Testing pypi packages

Run (append -win on windows)

tox -e pyX-pypi

For all the supported Python versions X.

Testing conda-forge packages

Run (append -win on windows)

tox -e pyX-conda-forge

For all the supported Python versions X.

Testing anaconda environment file

Run (append -win on windows)

tox -e anaconda-env-file

Making a new release

  1. Create a release branch.

  2. Make sure the full test suite passes.

  3. Replace Unreleased by the new version in CHANGELOG.rst.

  4. Hardcode the version number in conf.py.

  5. Push the branch.

  6. Build the docs for this branch on rtd, check the version and changelog.

  7. Merge to master.

  8. Tag.

  9. Run tox -e create-dist to create the distribution archives.

  10. Run twine upload dist/* -u <your login> to upload to pypi.org.

  11. Test the pypi packages.

  12. Update the recipe for conda-forge once the update bot sends the PR.

  13. Test the conda-forge packages.

  14. Merge master to develop so the last tag is a parent commit for defining the dev versions.

  15. Remove the hardcoded version number in conf.py.