Favorite posts and series

C++ 11 14 17 20macOS Setup (AS)Azure DevOps (Python Wheels) • Conda-Forge ROOTCLI11GooFitHistPython BindingsPython 2→33.8SSH

My books and workshops

Modern CMakeCMake WorkshopComputational Physics Class • Python CPU, GPU, Compiled minicourses • Level Up Your Python

My projects

pybind11cibuildwheelbuildboost-histogramHistVectorCLI11PlumbumGooFitParticleDecayLanguageConda-Forge ROOTPOVMJekyll-IndicoPyTest GHA annotate-failures

My sites



Azure DevOps: Introduction

Continuous Integration (CI) is fantastic for software development and deployment. One of the newest entries into the CI market1 is Microsoft’s Azure DevOps. Their Open Source support is impressive; it is likely part of the recent push2 by Microsoft to be more Open Source friendly. Open Source projects get 10 parallel builds, unlimited build minutes, 6 hour job timeouts, and incredibly fast jobs on macOS, Linux, and Windows, all via a single platform. Quite a few major projects3 have been moving to Azure since the initial release in December 2018. The configuration of DevOps is second only to GitLab CI in ease of use and possibly the most expressive system available. The multiple pipeline support also scales well to complicated procedures.

This is the first in a series of posts covering an introduction to setting up projects in Azure DevOps, developed to update the testing and releasing of Python packages for Scikit-HEP, a project for a coherent High Energy Physics Python analysis toolset. The second post covers release pipelines, and the third covers building binary Python packages using DevOps.

Note: I now highly recommend GitHub Actions, which is almost “Azure 2.0”, if you are interested in setting up CI. The language is very similar, although simplified, with some non-backward compatible bugfixes (such as multiline expressions will error if any line files, instead of just on the last line in Azure). You can read my tutorials on GitHub Actions on the Scikit-HEP developer pages.

[Read More]

ROOT on Conda Forge

Linux and macOS packages for Python 2.7, 3.6, 3.7, and 3.8

For High Energy Physics, the go-to framework for big data analysis has been CERN’s ROOT framework. ROOT is a massive C++ library that even predates the STL in some areas. It is1 also a JIT C++ interpreter called Cling, probably the best in the business. If you have heard of the Xeus C++ Kernel for Jupyter, that is built on top of Cling. ROOT has everything a HEP physicist could want: math, plotting, histograms, tuple and tree structures, a very powerful file format for IO, machine learning, Python bindings, and more. It also does things like dictionary generation and arbitrary class serialization (other large frameworks like Qt have similar generation tools).

You may already be guessing one of the most common problems for ROOT. It is huge and difficult to install – if you build from source, that’s a several hour task on a single core. It has gotten much better in the last 6 years, and there are several places you can find ROOT, but there are still areas where it is challenging. This is especially true for Python; ROOT is linked to your distro’s Python (both python2 and python3 if your distro supports it, as of ROOT 6.22); but the common rule for using Python is “don’t touch your system Python” - so modern Python users should be in a virtual environment, and for that ROOT requires the system site-packages option be enabled, which is not always ideal. And, if you use the Anaconda Python distribution, which is the most popular scientific distribution of Python and massively successful for ML frameworks, the general rule even for people who build ROOT themselves has been: don’t. But now, you can get a fully featured ROOT binary package for macOS or Linux, Python 2.7, 3.6, 3.7, or 3.8 from Conda-Forge, the most popular Anaconda community channel! Many more HEP recipes have now been added to Conda-Forge, as well! ROOT now also provides a conda docker image, too!

[Read More]

ROOT Install Options

New Conda Forge package of ROOT for Unix and more options

For particle physicists, ROOT is one of the most important toolkits around. It is a huge suite of tools that predates the C++ standard library, and has almost anything a particle physicist could want. It has driven developments in other areas too. ROOT’s current C++ interpreter, CLING, is the most powerful C++ interpreter available and is used by the Xeus project for Jupyter. The Python work has helped PyPy, with CPPYY also coming from ROOT. However, due to the size, complexity, and age of some parts of ROOT, it can be a bit challenging to install; and it is even more challenging when you want it to talk to Python. I would like to point to the brand-new Conda-Forge ROOT package for Linux and macOS, and point out a few other options for macOS installs. Note for Windows users: Due to the fact that ROOT expects the type long to match the system pointer size, 64-bit Windows cannot be supported for quite some time. While you can use it in 32 bit form, this is generally impossible to connect to Python, which usually will be a 64-bit build.

[Read More]

Histogram Speeds in Python

Let’s compare several ways of making Histograms. I’m going to assume you would like to end up with a nice OO histogram interface, so all the 2D methods will fill a Physt histogram. We will be using a 2 x 1,000,000 element array and filling a 2D histogram, or 10,000,000 elemends in a 1D histogram. Binnings are regular.

1D 10,000,000 item histogram

Example KNL MBP X24
Numpy: histogram 704 ms 147 ms 114 ms
Numpy: bincount 432 ms 110 ms 117 ms
fast-histogram 337 ms 45.9 ms 45.7 ms
Numba 312 ms 58.8 ms 60.7 ms

2D 1,000,000 item histogram

Example KNL MBP X24
Physt 1.21 s 293 ms 246 ms
Numpy: histogram2d 456 ms 114 ms 88.3 ms
Numpy: add.at 247 ms 62.7 ms 49.7 ms
Numpy: bincount 81.7 ms 23.3 ms 20.3 ms
fast-histogram 53.7 ms 10.4 ms 7.31 ms
fast-hist threaded 0.5 (6) 62.5 ms 9.78 ms (6) 15.4 ms
fast-hist threaded (m) 62.3 ms 4.89 ms 3.71 ms
Numba 41.8 ms 10.2 ms 9.73 ms
Numba threaded (6) 49.2 ms 4.23 ms (6) 4.12 ms
Cython 112 ms 12.2 ms 11.2 ms
Cython threaded (6) 128 ms 5.68 ms (8) 4.89 ms
PyBind11 sequential 93.9 ms 9.20 ms 17.8 ms
PyBind11 OpenMP atomic 4.06 ms 6.87 ms 1.91 ms
PyBind11 C++11 atomic (32) 10.7 ms 7.08 ms (48) 2.65 ms
PyBind11 C++11 merge (32) 23.0 ms 6.03 ms (48) 4.79 ms
PyBind11 OpenMP merge 8.74 ms 5.04 ms 1.79 ms
[Read More]