Scikit Build Proposal

I’ve spent the last few years trying to make it easy for anyone to extend Python with compiled languages. I’ve worked on pybind11, a powerful C++ library that allows users to write advanced Python extensions using just C++11, used by some of the largest projects, SciPy, PyTorch, Google, LLVM, and tens of thousands of other libraries, down to very small extensions. I also work on cibuildwheel, which makes building binaries (called wheels) on continuous integration (CI) simple. It is again powerful enough to used by huge projects, like Scikit-learn, matplotlib, mypy; and is simple enough to be used by hundreds of other packages. Recently it was accepted into the Python Packaging Authority (PyPA). There is one missing piece, though, to complete this picture of compiled extensions that easy to use for small projects, and powerful enough for large projects: the build system. I believe the solution to that is scikit-build, and I’d like to work on it over the next three years, and I need your help.

Scikit-build is a tool for integrating a package with a CMake build system into Python. You can utilize the vast collection of packages and projects using CMake already, and you have access to modern building features, like multithreaded builds, library discovery, superb compiler and IDE support, and all sorts of extended tooling. Modern CMake is quite pleasant to write compared to times past; I have written a book and training course on it. We ship up-to-date cmake and ninja wheels for all binary platforms.

I’m writing a proposal for an NSF CSSI Elements project containing three parts. The first part will cover core development on Scikit-build to address the current shortcomings and to prepare it for a post-distutils (Python 3.12+) world. The second part would cover assisting libraries with a science use case in either transitioning to scikit-build (ideally from an existing CMake build system with Python bindings, but I can help mentor developers in writing bindings (ideally pybind11), setting up CI, and writing CMake code as well (see my book or workshop on Modern CMake, and I’m happy to help old scikit-build projects transition to better practices). As part of this, I would be building up the examples and documentation, leading into the third part of the proposal: A series of training events and training material, including plans for something alongside SciPy.

Do you have an interesting use case? I’d love to hear about it. Feel free to reach out to me on twitter @henryschreiner3 or by email, henryfs at princeton.edu. I’m looking projects that would be interested in trying out the new scikit-build during years two and three. Tangibly, I’m looking for collaboration letters and a 1-3 sentence description of how we will collaborate over the course of the three year project - I can help you try out scikit-build over the next three years! After the break, I will give a few more details about the problem and outline my specific plans in a bit more detail. You can also see an outline at scikit-build/scikit-build/wiki or at the end of this post. Deadline is Dec 8, 2021, so letters need to be completed by the 7th or sooner!

[Read More]

🎡 cibuildwheel 2.2

Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process!

[Read More]

🎡 cibuildwheel 2.0

The cibuildwheel package has just had a major release with some fantastic features. Python 2.7 and 3.5 support has been removed (and PyPy3.6), allowing us to update to the latest manylinux and auditwheel versions, and support the newly unified manylinux PyPy3.7 images. We now allow users to select pypa/build as a build frontend. We now have a custom option to enable pre-release Pythons (3.10 currently) for testing before they are ABI stable (please don’t release wheels until that happens). Maybe most exciting, cibuildwheel now supports configuration in pyproject.toml, allowing you to be even further isolated from dependence on your CI system; you can easily produce Linux and Windows wheels locally (macOS still installs to system locations). And, since my last post and introduction post, cibuildwheel is now part of the PyPA!

[Read More]

Announcing CLI11 2.0

CLI11, a powerful library for writing beautiful command line interfaces in C++11, has been updated to 2.0. A lot of deprecated things have been removed, and there was a lot of cleanup under-the-hood; making CLI11 simpler. A few defaults have changed slightly, like better TOML support by default.

CLI11 does a better job than ever understanding any sort of container you provide - complex numbers are natively supported, along with atomic types. A long requested feature, simple version flags, has been added. Subcommands are more customizable. And there have been quite a few bugfixes for rare issues.

[Read More]

Setup an Apple Silicon Mac

I recently got an M1 mac, and I’ll be cataloging my experience with using it for scientific software development. I’ll be returning to update this page periodically, and will eventually have a focused recommendation for Apple Silicon setup, similar to my Intel setup.

[Read More]
macos 

🎡 cibuildwheel 1.8.0 and 1.9.0

cibuildwheel has just had two back-to-back releases, two weeks apart, representing several months of hard work and some exciting few features! I will be covering both releases at once, so we will discuss Apple Silicon support, architecture emulation on Linux, integrated PEP 621 Requires-Python support, the native GitHub Action, extended build and test controls, and more!

If you are following the releases, 1.7.0 came out last November (2020), and included the fantastic output folding feature, which makes logs much easier to read on CI systems that support folding, and makes it much easier to see how long each step takes. The 1.7.x series also included the addition of the working examples section of the documentation, which tracks some known projects using cibuildwheel, such as scikit-learn, Matlotlib, and MyPy; it is a great place to go to look into how other projects have integrated cibuildwheel into their workflow.

I have an general overview post as well. Now let’s look at what’s new! Update: cibuildwheel is now an official package of the PyPA!

[Read More]