Poetry provides a all in one solution to Python packaging. I want to focus on why I was quite hard on Poetry in my last post, specifically on its default version capping and solver quirks, and also a few other negative things. This is a followup to Should you have upper version bounds, which you should read before this post.[Read More]
Favorite posts and series
My books and workshops
pybind11 (python_example, cmake_example, scikit_build_example) • cibuildwheel • build • scikit-build (cmake, ninja) • boost-histogram • Hist • UHI • Scikit-HEP/cookie • Vector • CLI11 • Plumbum • GooFit • Particle • DecayLanguage • Conda-Forge ROOT • POVM • Jekyll-Indico • pytest GHA annotate-failures
Should You Use Upper Bound Version Constraints?
Bound version constraints (upper caps) are starting to show up in the Python ecosystem. This is causing real world problems with libraries following this recommendation, and is likely to continue to get worse; this practice does not scale to large numbers of libraries or large numbers of users. In this discussion I would like to explain why always providing an upper limit causes far more harm than good even for true SemVer libraries, why libraries that pin upper limits require more frequent updates rather than less, and why it is not scalable. After reading this, hopefully you will always consider every cap you add, you will know the (few) places where pinning an upper limit is reasonable, and will possibly even avoid using libraries that pin upper limits needlessly until the author updates them to remove these pins.
Also be warned, I pick on Poetry quite a bit. The rising popularity of Poetry is likely due to the simplicity of having one tool vs. many for packaging, but it happens to also have a special dependency solver, a new upper bound syntax, and a strong recommendation to always limit upper versions - in direct opposition to members of the Python core developer team and PyPA developers. Not all libraries with excessive version capping are Poetry projects (like TensorFlow), but many, many of them are. To be clear, Poetry doesn’t force version pinning on you, but it does push you really, really hard to always version cap, and it’s targeting new Python users that don’t know any better yet than to accept bad recommendations. And these affect the whole ecosystem, including users who do not use poetry, but want to depend on libraries that do! I do really like other aspects of Poetry, and would like to eventually help it build binary packages with Scikit-build (CMake) via a plugin, and I use it on some of my projects happily. If I don’t pick on Poetry enough for you, don’t worry, I have a follow-up post that picks on it in much more detail. Also, check out pdm, which gives many of the benefits of Poetry while following PEP standards.[Read More]
App vs Library
What is the difference between an app and a library? This seemingly simple question confuses some, and it turns out to be a harder question to answer than you might expect. While the actual distinction between these common terms will always be muddled in practice, I propose a specific definition to be used when considering dependencies. This distinction is important when discussing bound version constraints in the next post.[Read More]
Scikit Build Proposal
I’ve spent the last few years trying to make it easy for anyone to extend Python with compiled languages. I’ve worked on pybind11, a powerful C++ library that allows users to write advanced Python extensions using just C++11, used by some of the largest projects, SciPy, PyTorch, Google, LLVM, and tens of thousands of other libraries, down to very small extensions. I also work on cibuildwheel, which makes building binaries (called wheels) on continuous integration (CI) simple. It is again powerful enough to used by huge projects, like Scikit-learn, matplotlib, mypy; and is simple enough to be used by hundreds of other packages. Recently it was accepted into the Python Packaging Authority (PyPA). There is one missing piece, though, to complete this picture of compiled extensions that easy to use for small projects, and powerful enough for large projects: the build system. I believe the solution to that is scikit-build, and I’d like to work on it over the next three years.
Scikit-build is a tool for integrating a package with a CMake build system into
Python. You can utilize the vast collection of packages and projects using
CMake already, and you have access to modern building features, like
multithreaded builds, library discovery, superb compiler and IDE support, and
all sorts of extended tooling. Modern CMake is quite pleasant to write compared
to times past; I have written a book and training course on
it. We ship up-to-date
ninja wheels for all binary platforms.
I wrote a proposal for an NSF CSSI Elements project containing three parts. The first part will cover core development on Scikit-build to address the current shortcomings and to prepare it for a post-distutils (Python 3.12+) world. The second part would cover assisting libraries with a science use case in either transitioning to scikit-build (ideally from an existing CMake build system with Python bindings, but I can help mentor developers in writing bindings (ideally pybind11), setting up CI, and writing CMake code as well (see my book or workshop on Modern CMake, and I’m happy to help old scikit-build projects transition to better practices). As part of this, I would be building up the examples and documentation, leading into the third part of the proposal: A series of training events and training material, including plans for something alongside SciPy.
You can also see an outline at scikit-build/scikit-build/wiki or at the end of this post.
Thank you for all the projects! The proposal was submitted Dec 8, 2021; mid year we should find out if it was accepted![Read More]
🎡 cibuildwheel 2.2
Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process![Read More]
🎡 cibuildwheel 2.0
The cibuildwheel package has just had a major release with some fantastic
features. Python 2.7 and 3.5 support has been removed (and PyPy3.6), allowing
us to update to the latest manylinux and auditwheel versions, and support the
newly unified manylinux PyPy3.7 images. We now allow users to select
pypa/build as a build frontend. We now have a custom option to enable
pre-release Pythons (3.10 currently) for testing before they are ABI stable
(please don’t release wheels until that happens). Maybe most exciting,
cibuildwheel now supports configuration in
pyproject.toml, allowing you to be
even further isolated from dependence on your CI system; you can easily produce
Linux and Windows wheels locally (macOS still installs to system locations).
And, since my last post and introduction
post, cibuildwheel is now part of the PyPA!