Maintainers information¶
Packages upgrading¶
This section contains information about how and when to upgrade the packages used by GEMSEO and by the development environments.
Dependencies¶
GEMSEO is a library and not a self contained application, it can be installed in environments with varying and unknown constraints on the versions of its dependencies. Thus the versions of its dependencies cannot be pinneds, but a range of compatible versions shall be defined.
All the dependencies shall be defined in setup.cfg
,
this files does not tell where the packages will be pulled from.
The dependencies could be provided by the packages repositories
pypi, anaconda or conda-forge.
Getting GEMSEO to work with a common set of packages versions on several platforms and python versions is tricky and challenging. This kind of work is mostly done by trials and errors.
To reduce maintenance and complexity, our testing environments shall have the same packages providers for all the platforms and all the python versions. Furthermore it shall be identical to the references end-user environments under the same constraints.
In the context of tox,
the versions of the dependencies
that shall be installed with conda
are defined in requirements/gemseo-conda-python2,3.txt
,
they are pulled from conda-forge.
All other dependencies are installed with pip,
they are pulled from pypi.
When a dependency is changed,
setup.cfg
shall always be modified.
If the changed dependency is installed with conda,
then both gemseo-conda-python2,3.txt
and environment-py2,3.yml
shall be modified.
Documentation files like CREDITS.rst
and dependencies.rst
shall also be updated accordingly.
CI cache¶
To optimize the usage of the CI cache with gitlab,
the cache shall match closely the contents of the tox environments used for testing.
This is currently done by defining a cache affinity with
the dependencies of the environments and the environment name.
The dependencies are stored in the file
requirements/gemseo.in
which is created automatically by a custom
pre-commit hook defined in
:py:`tools/extract_req_in_from_setup_cfg.py`.
Development dependencies¶
As opposed to the dependencies of GEMSEO, the development dependencies can be fully controlled. Thus their versions are pinned so all developers are provided with reproducible and working environments. The dependencies shall be updated at least once in a while (couple months) to benefit from packages improvements and bug fixes.
This is done with pip-tools and from input requirements files. These input requirements files contain the minimum pinning requirements and are intended to be modified by maintainers. The pip-tools package provides the pip-compile which can process an input requirements file to produce a fully pinned requirements file.
We have the following input requirements files:
dev.in: with development tools with python 3 only,
doc.in: for building the documentation with python 3 only.
To update them:
conda run -p .tox/dev pip-compile -U requirements/dev.in
conda run -p .tox/doc pip-compile -U requirements/doc.in
Note
Append -win
to the environment names under windows.
Note
To reduce discrepancy among the environments,
requirements/test-python3.txt
shall be working for all the python 3 testing environments.
Git hooks are defined and run with pre-commit. It relies on packages that are managed with pre-commit instead of pip-tools. To update them:
conda run -p .tox/dev pre-commit autoupdate
Note
Append -win
to the environment names under windows.
Warning
All environments and tools shall be checked whenever dependencies have been changed.
Test dependencies¶
The test dependencies are defined in setup.cfg
so a end-user can easily run the GEMSEO tests.
To update them,
change the test
key of the
[options.extras_require]
section
in setup.cfg
,
then execute
tox -e style
This will call a pre-commit hook that will update
requirements/test.in
.
Then update the actual test requirements with:
conda run -p .tox/dev pip-compile -U requirements/test.in -o requirements/test-python3.txt
conda run -p .tox/py27 pip-compile -U requirements/test.in -o requirements/test-python2.txt
Note
Append -win
to the environment names under windows.
Warning
All environments and tools shall be checked whenever dependencies have been changed.
Anaconda testing¶
Before a release or whenever required,
the package for conda-forge
and the anaconda environment files
shall be tested.
Anaconda package¶
Run (append -win
on windows)
tox -e anaconda-package
Anaconda environment¶
Run (append -win
on windows)
tox -e anaconda-envs
Pypi distribution testing¶
Run (append -win
on windows)
tox -e py27-pypi,py36-pypi,py37-pypi,py38-pypi
Making a new release¶
Create a release branch.
Make sure the full test suite passes.
Replace
Unreleased
by the new version inCHANGELOG.rst
.Hardcode the version number in
conf.py
.Push the branch.
Build the docs for this branch on rtd, check the version and changelog.
Merge to master.
Tag.
Run tox -e create-dist to create the distribution archives.
Run twine upload dist/* -u <your login> to upload to pypi.org.
Run tox -e py*-pypi{-win} for all envs to test the pypi package.
Update the recipe for conda-forge once the update bot sends the PR.
Test the conda-forge packages (TODO: create a tox target)
Merge master to develop so the last tag is a parent commit for defining the dev versions.