Development
To get the source code from GitHub, you can do
git clone git@github.com:yatiml/yatiml.git
cd yatiml
Run tests (including coverage and type checking) with:
pip install tox
tox
Error message evaluation mode can be enable by
tox -e show-errors
This will disable pytest.raises, causing all the expected exceptions raised by the tests to actually be raised and their tests to fail. As a result, you can see the error messages generated by YAtiML and see if they’re easy to understand.
A local copy of the documentation can be generated using:
tox -e docs
Contributing
If you want to contribute some improvements to YAtiML, please use the following process:
(important) Announce your plan to the rest of the community before you start working. This announcement should be in the form of a (new) issue.
(important) wait until some kind of consensus is reached about your idea being a good idea.
If needed, fork the repository to your own Github profile and create your own feature branch off of the latest
develop
commit. While working on your feature branch, make sure to stay up to date with thedevelop
branch by pulling in changes, possibly from the ‘upstream’ repository (follow the instructions here and here).Make sure the existing tests still work by running
python setup.py test
.Add your own tests (if necessary).
Update or expand the documentation.
Use
yapf
to fix the readability of your code style andisort
to format and group your imports.Push your feature branch to (your fork of) the YAtiML repository on GitHub.
create the pull request, e.g. following the instructions here.
In case you feel like you’ve made a valuable contribution, but you don’t know how to write or run tests for it, or how to generate the documentation: don’t let this discourage you from making the pull request; we can help you! Just go ahead and submit the pull request, but keep in mind that you might be asked to append additional commits to your pull request.
Making a release
YAtiML uses Git on GitHub for version management, using the Git Flow branching model. Making a release involves quite a few steps, so they’re listed here to help make the process more reliable; this information is really only useful for the maintainers.
Check metadata
Check the metadata in
setup.py
, and update as necessary.Check the copyright date and owners in README.rst and docs/conf.py and update as necessary.
Update the changelog
Each release should have an entry in the CHANGELOG.rst describing the new features and fixed problems. Since we’ll want to carry these entries forward, we’ll make them first, on the develop branch. Use the git log to get a list of the changes, and switch to the development branch:
git log <your favourite options>
git switch develop
and then edit CHANGELOG.rst and commit.
git add CHANGELOG.rst
git commit -m 'Add version x.y.z to the change log'
Make release branch
To start the release process, make a release branch
git switch -c release-x.y.z develop
YAtiML uses Semantic Versioning, so name the new version accordingly.
Update version
Next, the version should be updated. There is a version tag in setup.py
and
two for the documentation in docs/conf.py
(search for version
and
release
). There is also an __version__
in __init__.py
. On the
development branch, these should be set to x.y.z.dev0
, where x.y.z
is
the expected next version. On the release branch, they should be set to
x.y.z
(with here the actual number of this release of course).
Check documentation
Since we’ve just changed the documentation build configuration, the build should be run locally to test:
tox -e docs
Next, point your web browser to docs/_build/index.html
and verify that the
documentation built correctly. In particular, the new version number should be
in the browser’s title bar as well as in the blue box on the top left of the
page.
Run tests
Before we make a commit, the tests should be run, and this is a good idea anyway
if we’re making a release. So run tox
and check that everything is in order.
Commit the version update
This is the usual Git poem:
git add setup.py docs/conf.py yatiml/__init__.py
git commit -m 'Set release version to x.y.z'
git push --set-upstream origin release-x.y.z
This will trigger the Continuous Integration, so check that that’s not giving any errors while we’re at it.
Fix badges
The badges in the README.rst normally point to the development branch versions
of everything. For the master branch, they should point to the master version.
Note that for the ReadTheDocs badge, develop
should be changed to
latest
, and that for Codacy there is only one badge, so no change is needed.
# edit README.rst
git add README.rst
git commit -m 'Update badges to point to master'
git push
Merge into the master branch
If all seems to be well, then we can merge the release branch into the master
branch and tag it, thus making a release, at least as far as Git Flow is
concerned. We use the -X theirs
option here to resolve the merge conflict
caused by the version update that was done for the previous release, which we
don’t have on this branch. The last command is to push the tag, which is
important for GitHub and GitHub integrations.
git switch master
git merge --no-ff -X theirs release-x.y.z
git tag -a x.y.z -m 'Release x.y.z'
git push
git push origin x.y.z
Build and release to PyPI
Finally, the new version needs to be built and uploaded to PyPI, so that people can start using it. To build, use:
python3 setup.py sdist bdist_wheel
Then, we can upload to the test instance of PyPI:
twine upload --repository-url https://test.pypi.org/legacy/ dist/yatiml-x.y.z*
To test that we can install it, run this in a fresh virtualenv. Note that the PyPI test server doesn’t have the dependencies, so we need to install those from the production server first.
pip install PyYAML typing_extensions
pip install --index-url https://test.pypi.org/simple/ yatiml
And if all seems well, we can upload to the real PyPI:
twine upload dist/yatiml-x.y.z*
Update conda-forge feedstock
(Note: we’re skipping a local rerender here in favour of letting the conda-forge bot handle it on GitHub. If that becomes an issue we’ll change it, but this way we don’t need to have conda installed locally.)
First, we need a fork of https://github.com/conda-forge/yatiml-feedstock, so create one if you don’t have one yet, and clone it locally.
If you are reusing an existing clone of an existing fork, then it may need some updates first. First, add the conda-forge remote if it’s not there yet:
git remote add conda-forge https://github.com/conda-forge/yatiml-feedstock.git
Now that our clone is connected to both the fork (origin) and the original repository (conda-forge), we can synchronise them:
git switch release-w.x.y # previous release
git pull # get changes made by conda in the fork
git switch main
git pull conda-forge main # get the previous merge
git push origin # update the fork
This should make it so that conda-forge, the fork, and the clone all agree on which commits we have.
Then we can create a branch to work on:
git switch main
git pull
git switch -c release-x.y.z
Next, we need to get a checksum for the package we uploaded to PyPI. In the main yatiml directory, run:
sha256sum dist/yatiml-x.y.z.tar.gz
Next, in yatiml-feedstock
, edit recipe/meta.yaml
:
Update to the new version at the top
Replace the checksum with the one for the new release
We can then test the new build by running python3 build-locally.py
. This
will build the package inside of a Docker container, so you need to have Docker
installed and have a couple GB of free disk space.
If it all works, then we can commit the changes to the local branch:
git add recipe/meta.yaml
git commit -m 'Update to version x.y.z'
git push --set-upstream origin release-x.y.z
Note that this pushes to the fork, not to conda-forge/yatiml-feedstock
,
which is exactly what we want. Pushing to upstream directly will break the
automation.
Instead, go to the fork, and make a pull request for merging the changes into
conda-forge/yatiml-feedstock:main
. Run through the checklist in the
template. To check whether the license file is included, in the yatiml
directory do:
tar tf dist/yatiml-x.y.z.tar.gz
and check that LICENSE and NOTICE are both there.
Add a @conda-forge-admin, please rerender
to the text to rerender the
feedstock. This will upgrade the auto-generated parts of meta.yaml
to the
latest configuration, so it adds another commit to the branch.
So, wait for the conda-forge-linter
to lint, and for conda-forge-admin
to rerender, and then merge the PR using the GitHub GUI. The new package will
now be staged and built and copied over to the Anaconda repository. This may
take a couple of hours, so don’t worry if it doesn’t appear immediately.
As a final test, you can do:
docker run -ti conda/miniconda3
# conda install -c conda-forge yatiml
which should install the new version.
Finally, we need to re-sync the fork and clone with the original repository, because changes have been made in the cloud. By doing this, we avoid having to do it next time (see above) if we reuse this fork and/or clone.
git switch release-x.y.z
git pull # get changes made by conda in the fork
git switch main
git pull conda-forge main # get the merge we just made
git push origin # update the fork for next time
Make a GitHub Release
Go to Releases on the GitHub page and make a new release from the tag. For the release notes, use this template and copy-paste the content from the CHANGELOG:
# YAtiML
YAML-based file formats can be very handy, as YAML is easy to write by humans, and parsing support for it is widely available. Just read your YAML file into a document structure (a tree of nested dicts and lists), and manipulate that in your code.
As long as that YAML file contains exactly what you expect, that works fine. But if it contains a mistake, then you're likely to crash the program with a cryptic error message, or worse (especially if the YAML file was loaded from the Internet) it may do something unexpected.
To avoid that, you can validate your YAML using various schema checkers. You write a description of what your YAML file must look like, then feed that to a library which checks the incoming file against the description. That gives you a better error message, but it's a lot of work.
YAtiML takes a different approach. Instead of a schema, you write a Python class. You probably already know how to do that, so no need to learn anything. YAtiML then generates loading and dumping functions for you, which convert between YAML and Python objects. If needed, you can add some extra code to make the YAML look nicer or implement special features.
# <x.y.z>
## Incompatible changes
* <change>
## New functionality
* <new>
## Fixes
* <fixed>
## Removed
* <removed>
The preamble is there because this text ends up on the Zenodo page, and people who end up there will probably want to know what it is before learning about the latest changes.
There’s no need to upload binaries, GitHub will create tar files with snapshots for Zenodo automatically, and we’ve already put things on PyPI and Conda.
Merge release branch back into develop
To continue developing, merge the release branch back into develop
git switch develop
git merge --no-commit release-x.y.z
Make sure that the badges are set to develop, and that the version number is set to the next expected version x.y.{z+1}.dev0 (it’s fine if x.{y+1}.0 is what ends up being released eventually). Then you can commit and continue developing:
git commit
git push
Update issues
Go through the issues on GitHub and close the ones for which a fix was released. Or if they were created by someone else, ask the user to check that the new version solves their problem and then close the issue if it does.