Tech
Jerry Pussinen
Aug 11, 2022
Introducing a project template for modern Python packages
Spinning up new projects should be a breeze when developing software at scale. At Wolt, we have over 50 product teams which do software development. We are heavy users of project templates which both help us to get a head start and also ensure that best practices are applied. We recently open sourced wolt-python-package-cookiecutter. It’s a cookiecutter project template for rapidly developing new open source Python packages with best practices and all the modern bells and whistles included. In this blog post, we’ll describe what it contains and discuss the choices made while developing it.
Taking project templates to the next level with Cookiecutter & cruft
Cookiecutter in combination with cruft are the go-to tools for us when it comes to templating. Cookiecutter is the templating tool while cruft gives us the possibility for keeping the generated projects up-to-date with the template from which they were generated from. Although both Cookiecutter and cruft are implemented in Python, they are actually language-agnostic considering templates they are used for. Thus, you either need to have Python installed on your machine or have a containerized setup in order to use them, but it doesn’t matter what kind of project templates they are used for.
Here’s a short demo about how the project skeleton is created from the template:
To get an idea about the power of Cookiecutter + cruft combination, consider that you have a cookiecutter project template and twenty projects generated from it. Then, say, you’d like to add a new shiny linter or even just enable one configuration flag for one of the existing linters. You’ll naturally update this in the project template to get the change for all the new projects. However, you are probably too lazy to update the change in those twenty existing projects. This is where cruft steps in. Just run
1cruft update
in each of those twenty projects, and it’ll automatically pull the changes from the updated template. We took this to the next level by creating a GitHub Actions workflow which runs the cruft update periodically, commits the changes, and creates a pull request. In other words, all the projects which are generated from wolt-python-package-cookiecutter get a PR when we, for example, update some best practices in the template.
If you’d like to learn more about the power of Cookiecutter + cruft combination, watch Alexander Frenzel’s Python meetup presentation about the topic:
The essentials of a Python package
Let’s dive into the actual project template and see what it consists of.
Dependency management
We trust in Poetry. In our experience, Poetry has been more reliable (read: less buggy, fewer surprises, and faster) compared to Pipenv and it also seems to be the go-to in the Python community.
Poetry is especially handy in the development of packages which are published in some package index, such as PyPI in the case of open source. Among the other powerful features of Poetry, it has built-in support for Semantic Versioning (aka semver). To be precise, it’s not actually built-in to the source code of Poetry, but they are using semver package which got extracted from Poetry some years ago. Thanks to the semver support, building and publishing a new minor release (e.g. 1.1.0 -> 1.2.0) is as simple as:
1 2poetry version minor poetry --build publish
Linters
Our go-to linter is Flake8 which wraps Pyflakes (correctness), pycodestyle (PEP8) and mccabe (complexity). Flake8 is not necessarily as comprehensive as PyLint but it’s faster in most use cases and certainly requires less tuning on the configuration side to avoid false positives. Flake8 also has a rich plugin ecosystem and it’s fairly easy to implement a use case specific plugin in case there’s no suitable plugin available in the ecosystem.
In the wolt-python-package-cookiecutter, we decided to include the following Flake8 plugins:
flake8-bugbear for detecting bugs and potential design problems,
flake8-builtins for verifying that builtins are not used as variable names,
flake8-comprehensions for writing better and consistent comprehensions,
flake8-debugger for checking that there are no forgotten breakpoints,
flake8-eradicate to avoid “dead” or commented code,
flake8-logging-format for consistent logging,
pep8-naming for verifying that PEP8 naming conventions are followed, and
tryceratops to avoid exception anti-patterns.
Typing & static type checking
Type hints are an essential part of modern Python. They give superpowers for IDEs, remove the need for documentation (which would be outdated anyway), and help in catching silly bugs early on.
Mypy is our tool of choice for static type checking. We like to use fairly strict configuration for it in order to gently enforce the usage of type hints and a set of best practices related to typing. There’s a detailed blog post about professional-grade mypy configuration if you’re interested in diving deeper.
Auto-formatters
Luckily nowadays, there are plenty of great tools which help developers focus on writing code rather than formatting code. The latter doesn’t really bring any business value and it’s also time-consuming to argue about it in code reviews. So, it’s better to let the computer handle formatting completely.
In the wolt-python-package-cookiecutter, we decided to use:
isort for sorting imports,
autoflake for removing unused imports, and
Black for formatting the rest of the code.
Documentation
The quality of documentation has a significant impact on how the package will be received by the potential user base. There are a number of great tools which help in creating professional-looking documentation with minimal effort. In the project template, we decided to use Material for MkDocs which generates an aesthetically pleasing static website from a set of Markdown files. After all, Markdown is probably the most developer-friendly format for writing documentation.
The source code-level documentation (e.g. docstrings) is included in the auto-generated static website with the help of mkdocstrings which is an MkDocs plugin. As MkDocs and its plugins are Python packages, they are just regular development dependencies listed in pyproject.toml, which enables hassle-free development setup also for documentation.
Changelog is an important, yet often overlooked piece of documentation. We went with Semantic Versioning and keep a changelog which specifies the exact format for the changelog. Well-specified formats are great for consistency and also an enabler for automated tooling. For automating changelog-related functionality, we decided to use python-kacl which helps in verifying the format of the changelog, updating the changelog automatically when it’s time to release, and fetching the changelog of a single release.
We decided to use GitHub Pages for hosting the static documentation website. GitHub Pages is free, easy to setup, and naturally integrates nicely with GitHub repositories.
Testing
pytest is the go-to testing framework in the Python community, and therefore it was an easy choice for the project template. It also has a rich plugin ecosystem. In the project template, we decided to use pytest-cov for measuring coverage and pytest-github-actions-annotate-failures for visually displaying CI (continuous integration) test failures in the GitHub UI.
When building an open source package, it’s impossible to know what kind of projects will end up depending on it. For example, the dependent may not have the newest Python version in use. Thus, it’s also important to test the package against older Python versions. We decided to use 3.7 as the lowest supported version. At the time of writing this, 3.7 is the oldest Python version which still gets security updates (EOL 6/2023, source). In practice, testing against a number of different Python versions is achieved by using a matrix strategy in the GitHub Actions workflow job which runs the tests.
Automating everything
Like many guides suggest, it’s beneficial to automate everything you can, and that’s also the principle we followed with the project template. Automation both protects against silly typos and helps in the maintenance of projects which are no longer receiving proper care from their maintainers.
As was discussed above, there’s a number of linters and auto-formatters included in the template. Additionally, there’s pre-commit configuration which helps in automating the usage of all those tools. Pre-commit helps in getting more green CI pipelines as it ensures that the lightweight CI checks are already run on the developer’s machine.
Like mentioned in the testing section, the CI runs tests against multiple different Python versions. Additionally, all the pre-commit checks are run as part of the CI workflow.
For the releases, we decided to build a two-step semi-automated process. The first step is triggered by running the draft release GitHub actions workflow manually. The person triggering the workflow only needs to provide the desired version number of the new release or, if they are too lazy to check what was the previous version, they could just mention that it’s e.g. a patch release:
This automatically updates the changelog and the version in the pyproject.toml, creates commits for the changes, pushes a tag pointing to the commit to be released, and finally, creates a draft GitHub release which also contains the changelog of the release.
After the draft release is created, the person doing the release can finalise the process by publishing the draft release from GitHub UI. This releases the new version to PyPI and updates the documentation website. Simple as that – no coding needed for releasing!
In addition to CI and release related automation, there are GitHub Actions workflows for syncing the project structure with the template (basically cruft update + pull request) and another one for updating the dependencies of the project (basically poetry lock + pull request). Both of them run on a cron schedule or alternatively can be manually triggered if needed. These two workflows ensure that the project will receive maintenance PRs every now and then even if it’s not being actively developed anymore.
Closing words
This blog post described what a modern Python package looks like in 2022. You’ll get all the bells and whistles out of the box by using wolt-python-package-cookiecutter to bootstrap your projects. As tooling and best practices tend to evolve over time, the template may look slightly different in a few years. Thanks to cruft and the automation described above, all the projects generated from our project template will receive pull requests as the template evolves. Keeping up with the best practices couldn’t be any easier.
If you’d like to see what an example project that utilises wolt-python-package-cookiecutter looks like, go check pytest-split out.
