pybind11 2.6.0

pybind11 logo

I am pleased to announce the release of pybind11 2.6.0! This is the largest release since 2.2 (released over three years ago). I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! The focus of this release was stability, packaging, and supporting more platforms, though there are a lot of small features and useful additions, covered by newly expanded docs.

Stability

One of the headline features is a brand new CI system, testing over 50 different configurations. This has enabled expanded support for compilers, such as NVCC, PGI, and lots of versions of Clang, GCC, and MSVC, now tested from C++11 to (the newly supported) C++20. Many warnings and bugs were fixed across the spectrum. PyPy support has been updated to the 7.3.x series, and fully supports pypy3 for the first time. New Python 3.9 support includes a workaround for a segfault in CPython 3.9.0 (pybind11 developers have fixed the bug in CPython, and 3.9.1 will work with older versions of pybind11 too; conda-forge also provides the patch). every configuration supported by cibuildwheel is supportd.

CMake build system

Another headline feature is the completely reworked build systems; first we will cover CMake. The focus was to support more situations and modern CMake tools in a backward compatible way, while maintaining support for CMake 3.4+. Most significantly, the generated Config/Target files now no longer depend on the generator platform, allowing a configuration to be used on a different machine. This enables, for the first time, the CMake files to be distributed in the Python package available on PyPI via pip!

You now simply use CMAKE_CXX_STANDARD or compile features - pybind11 will automatically require at least C++11 via compile features. You can use CMAKE_INTERPROCEDURAL_OPTIMIZATION and CMAKE_CXX_VISIBILITY_PRESET and pybind11 will get out of the way and let CMake dictate those features. CUDA as a language is fully supported now. And the Python discovery is much smarter, finding virtual environments and conda.

A huge new feature is support for CMake’s modern FindPython tooling; you can now simply write:

find_package(Python COMPONENTS Interpreter Development)
add_subdirectory(extern/pybind11)

and pybind11 will respect the Python you found, and will use the new variables. You can force the new search by instead setting PYBIND11_FINDPYTHON ON before including pybind11. This does requires CMake 3.12, with 3.15+ better and 3.18.2+ best, especially for PyPy (you also should use PyPy 7.3.2+).

You can now disable Python discovery entirely by setting PYBIND11_NOPYTHON TRUE, and then do it yourself with the modular targets provided. This is ideal for working in a large existing framework, or using Scikit-build’s helper file.

There is much more to explore, as well, like finer-grained targets and new helper functions.

Setup helpers

Not to be outdone, the setuptools packaging has been completely reworked and expanded to better support both simple and advanced setuptools projects. Before 2.6.0, pybind11 extensions that wanted to be widely available and avoid requiring CMake had to follow the python example repo. The basic ideas in that repo have now been integrated into a pybind11.setup_helpers module, and can be used any way that you would like to use pybind11. A lot of long-standing issues with the example have been fixed; if you have ever seen a setup.py with the (incorrect) statement that “has_flag” is part of Python 3.6+, you know that someone was copying this example. You can also be pretty sure that they have an incorrect has_flag function, as well, that will generate temporary files in your working directory on Windows. These and other bugs have been fixed for setup_helpers; it properly supports Windows compilers, C++17, etc.

Using it is a breeze. Just use PEP 518’s pyproject.toml file to list pybind11, or if you use the submodule form instead, just do:

DIR = os.path.abspath(os.path.dirname(__file__))

sys.path.append(os.path.join(DIR, "extern", "pybind11"))
from pybind11.setup_helpers import Pybind11Extension  # noqa: E402

(better yet, use pathlib if you are Python 3 only). Building an extension is simple:

ext_modules = [
    Pybind11Extension(
        "myext",
        sorted(SRC_FILES),
        include_dirs=INCLUDE_DIRS,
        cxx_std=11,
        extra_compile_args=["/d2FH4-"] if sys.platform.startswith("win32") else [],
    )
]

This looks just like a standard Extension (and is a subclass of it), but will set up the flags needed to compile pybind11 extensions. It also has a cxx_std setting that selects a C++ standard to use. There is also a pybind11_include argument that can disable the inclusion of the pybind11 directory (the setup_helpers file is completely standalone, and can even be copied into your project or even your setup.py).

Note that above I added a normal addition for an extension; the extra_compile_args. All normal additions are supported. (If you are curious, the one I illustrate above will keep extensions that are built on MSVC 2019 from adding an extra runtime dependency that some older redistributable downloads do not have).

If you want to leave off the cxx_std and instead use the highest available standard, then you need to also add the build_ext from setup_helpers (currently, that is all this does, though in the future more functionality could be added, like MinGW support):

from pybind11.setup_helpers import build_ext

setup(
    ...
    build_cmd = {"build_ext": build_ext},
)

There is also a parallel compiler utility, which can dramatically speed up the compile performance of your extension on multicore hardware:

from pybind11.setup_helpers import ParallelCompile

ParallelCompile("NPY_NUM_BUILD_JOBS").install()

setup(...

The variable name you choose when constructing it allows users to control the number of threads (the one here matches NumPy, which has similar functionality). You can also control the default and max number of threads via keyword arguments.

New features

Besides stability and bugfixes, and major building improvements, there are a significant collection of exciting new features.

Keyword-only and positional-only arguments are now supported on any version of Python. Use them like this:

.def("my_func", &func, "p1"_a, py::pos_only(), "n1"_a, py::kw_only(), "k1"_a)

This will define a function with one positional only argument, a normal argument n1, and a keyword-only argument k1. This can help you design tight, clean interfaces that are easy to maintain. A bug with .none() has also been fixed here.

You can now use py::is_final() to force a class to be uninheritable (CPython only). You can use py::preprend() to insert a function at the top of the call stack instead of last. There’s also now a way to get the type object from a registered C++ type with py::type::of<T>(), or you can get the type from a Python object with py::type::of(ob).

Support for code correctness

In several ways, pybind11 now helps you write more correct code. If you forget to call __init__ on a pybind11 class, an error is thrown (since the C++ memory is not initialized correctly without it). Conversions to python types, like py::bytes from general types like py::object now throw if the conversion is invalid, rather than having to check py::isinstance first.

Minor features

Lots of small changes were made to better support C++20 or be clearer/cleaner. There no longer is a nested submodule in the repo (great for use as a submodule from other places, like GitLab). Check out the full list, there are bugs finally fixed after several years in some cases! The docs have also been expanded and feature several new pages and sections.

New maintainer team

The largest change might not be in the code at all. A new collection of some of the best contributors were pulled in and made part of the maintenance team. There are more than a half-dozen of us, and we cover a wide range of backgrounds and interests, from High Energy Physics to Google. We would also like to thank Google for a generous donation to help our large number of CI jobs to run fully in parallel.

There are more exciting improvements to come! Hopefully releases will be a little more regular in the future with more patch releases and more frequent releases. Please continue to help us catch issues early by testing with the master branch! We can sometimes pull or change a feature on the master branch, but it should be quite stable, as it has to pass our 50+ checks.

comments powered by Disqus