Maintainers information¶
Packages upgrading¶
This section contains information about how and when to upgrade the packages used by GEMSEO and by the development environments.
Dependencies¶
GEMSEO is a library and not a self contained application, it can be installed in environments with varying and unknown constraints on the versions of its dependencies. Thus the versions of its dependencies cannot be pinneds, but a range of compatible versions shall be defined.
All the dependencies shall be defined in setup.cfg,
this files does not tell where the packages will be pulled from.
The dependencies could be provided by the packages repositories
pypi, anaconda or conda-forge.
Getting GEMSEO to work with a common set of packages versions on several platforms and python versions is tricky and challenging. This kind of work is mostly done by trials and errors.
To reduce maintenance and complexity, our testing environments shall have the same packages providers for all the platforms and all the python versions. Furthermore it shall be identical to the references end-user environments under the same constraints.
In the context of tox,
the versions of the dependencies
that shall be installed with conda
are defined in requirements/gemseo-conda-python2,3.txt,
they are pulled from conda-forge.
All other dependencies are installed with pip,
they are pulled from pypi.
When a dependency is changed,
setup.cfg shall always be modified.
If the changed dependency is installed with conda,
then both gemseo-conda-python2,3.txt
and environment-py2,3.yml shall be modified.
Documentation files like CREDITS.rst
and dependencies.rst shall also be updated accordingly.
CI cache¶
To optimize the usage of the CI cache with gitlab,
the cache shall match closely the contents of the tox environments used for testing.
This is currently done by defining a cache affinity with
the dependencies of the environments and the environment name.
The dependencies are stored in the file
requirements/gemseo.in which is created automatically by a custom
pre-commit hook defined in
:py:`tools/extract_req_in_from_setup_cfg.py`.
Development dependencies¶
As opposed to the dependencies of GEMSEO, the development dependencies can be fully controlled. Thus their versions are pinned so all developers are provided with reproducible and working environments. The dependencies shall be updated at least once in a while (couple months) to benefit from packages improvements and bug fixes.
This is done with pip-tools and from input requirements files. These input requirements files contain the minimum pinning requirements and are intended to be modified by maintainers. The pip-tools package provides the pip-compile which can process an input requirements file to produce a fully pinned requirements file.
We have the following input requirements files:
test.in: for running the tests with python 2 and 3 (this file is automatically generated by a pre-commit hook),
dev.in: with development tools with python 3 only,
doc.in: for building the documentation with python 3 only.
To update them:
conda activate .tox/dev
pip-compile -U requirements/test.in -o requirements/test-python3.txt
pip-compile -U requirements/dev.in
conda activate .tox/doc
pip-compile -U requirements/doc.in
conda activate .tox/py27
pip-compile -U requirements/test.in -o requirements/test-python2.txt
Note
To reduce discrepancy among the environments,
requirements/test-python3.txt
shall be working for all the python 3 testing environments.
Git hooks are defined and run with pre-commit. It relies on packages that are managed with pre-commit instead of pip-tools. To update them:
conda activate .tox/dev
pre-commit autoupdate
Warning
All environments and tools shall be checked whenever the dependencies have been changed.
Anaconda testing¶
Before a release or whenever required,
the package for conda-forge
and the anaconda environment files
shall be tested.
Pypi distribution testing¶
Run (append -win on windows)
tox -e py27-pypi,py36-pypi,py37-pypi,py38-pypi
Full packaging¶
The full package is an archive named
gemseo-full-package.tbz
for delivering a release,
it contains:
GEMSEO distribution
generated documentation
tests
anaconda environment files
README.rst
Create it with
tox -e full-package