Azure DevOps: Python Wheels

This is the third post in a series about Azure DevOps. This one is about making Python wheels. If you want to play nice with Python users, or you have a complex build, this will make your package far more accessible to users. They are faster to install and to use and more secure. We will quickly cover making universal wheels, then we will move on to fully compiled binaries, including C++14, manylinux2010, and other hot topics. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. The results of this tutorial can be seen in the boost-histogram repository, under the .ci folder.

[Read More]

Azure DevOps: Releases

This is the second post in a series about Azure DevOps. This one is about release pipelines; if you use Azure to build packages (like binaries, etc.), how do you push them to a final endpoint? In this example, we will be building a simple pure Python package, and pushing the result to Test-PyPI. You can adapt it to your situation, however. The third post will cover building Python binaries. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. Several of the projects in SciKit-HEP are using release pipelines, include boost-histogram and Particle.

[Read More]

Azure DevOps: Introduction

Continuous Integration (CI) is fantastic for software development and deployment. One of the newest entries into the CI market1 is Microsoft’s Azure DevOps. Their Open Source support is impressive; it is likely part of the recent push2 by Microsoft to be more Open Source friendly. Open Source projects get 10 parallel builds, unlimited build minutes, 6 hour job timeouts, and incredibly fast jobs on macOS, Linux, and Windows, all via a single platform. Quite a few major projects3 have been moving to Azure since the initial release in December 2018. The configuration of DevOps is second only to GitLab CI in ease of use and possibly the most expressive system available. The multiple pipeline support also scales well to complicated procedures.

This is the first in a series of posts covering an introduction to setting up projects in Azure DevOps, developed to update the testing and releasing of Python packages for Scikit-HEP, a project for a coherent High Energy Physics Python analysis toolset. The second post covers release pipelines, and the third covers building binary Python packages using DevOps.

[Read More]

ROOT on Conda Forge

Linux and macOS packages for Python 2.7, 3.6, and 3.7

For High Energy Physics, the go-to framework for big data analysis has been CERN’s ROOT framework. ROOT is a massive C++ library that even predates the STL in some areas. It is1 also a JIT C++ interpreter called Cling, probably the best in the business. If you have heard of the Xeus C++ Kernel for Jupyter, that is built on top of Cling. ROOT has everything a HEP physicist could want: math, plotting, histograms, tuple and tree structures, a very powerful file format for IO, machine learning, Python bindings, and more. It also does things like dictionary generation and arbitrary class serialization (other large frameworks like Qt have similar generation tools).

You may already be guessing one of the most common problems for ROOT. It is huge and difficult to install – if you build from source, that’s a several hour task on a single core. It has gotten much better in the last 5 years, and there are several places you can find ROOT, but there are still areas where it is challenging. This is especially true for Python; ROOT is linked to just one version of Python, and the one you get with pre-built ROOT can often be the wrong one. And, if you use the Anaconda Python distribution, which is the most popular scientific distribution of Python and massively successful for ML frameworks, the general rule even for people who build ROOT themselves has been: don’t. But now, you can get a fully featured ROOT binary package for macOS or Linux, Python 2.7, 3.6, or 3.7, from Conda-Forge, the most popular Anacanda community channel!

[Read More]

ROOT Install Options

New Conda Forge package of ROOT for Unix and more options

For particle physicists, ROOT is one of the most important toolkits around. It is a huge suite of tools that predates the C++ standard library, and has almost anything a particle physicist could want. It has driven developments in other areas too. ROOT’s current C++ interpreter, CLING, is the most powerful C++ interpreter available and is used by the Xeus project for Jupyter. The Python work has helped PyPy, with CPPYY also coming from ROOT. However, due to the size, complexity, and age of some parts of ROOT, it can be a bit challenging to install; and it is even more challenging when you want it to talk to Python. I would like to point to the brand-new Conda-Forge ROOT package for Linux and macOS, and point out a few other options for macOS installs. Note for Windows users: Due to the fact that ROOT expects the type long to match the system pointer size, 64-bit Windows cannot be supported for quite some time. While you can use it in 32 bit form, this is generally impossible to connect to Python, which usually will be a 64-bit build.

[Read More]

Histogram Speeds in Python

Let’s compare several ways of making Histograms. I’m going to assume you would like to end up with a nice OO histogram interface, so all the 2D methods will fill a Physt histogram. We will be using a 2 x 1,000,000 element array and filling a 2D histogram, or 10,000,000 elemends in a 1D histogram. Binnings are regular.

1D 10,000,000 item histogram

Example KNL MBP X24
Numpy: histogram 704 ms 147 ms 114 ms
Numpy: bincount 432 ms 110 ms 117 ms
fast-histogram 337 ms 45.9 ms 45.7 ms
Numba 312 ms 58.8 ms 60.7 ms

2D 1,000,000 item histogram

Example KNL MBP X24
Physt 1.21 s 293 ms 246 ms
Numpy: histogram2d 456 ms 114 ms 88.3 ms
Numpy: 247 ms 62.7 ms 49.7 ms
Numpy: bincount 81.7 ms 23.3 ms 20.3 ms
fast-histogram 53.7 ms 10.4 ms 7.31 ms
fast-hist threaded 0.5 (6) 62.5 ms 9.78 ms (6) 15.4 ms
fast-hist threaded (m) 62.3 ms 4.89 ms 3.71 ms
Numba 41.8 ms 10.2 ms 9.73 ms
Numba threaded (6) 49.2 ms 4.23 ms (6) 4.12 ms
Cython 112 ms 12.2 ms 11.2 ms
Cython threaded (6) 128 ms 5.68 ms (8) 4.89 ms
PyBind11 sequential 93.9 ms 9.20 ms 17.8 ms
PyBind11 OpenMP atomic 4.06 ms 6.87 ms 1.91 ms
PyBind11 C++11 atomic (32) 10.7 ms 7.08 ms (48) 2.65 ms
PyBind11 C++11 merge (32) 23.0 ms 6.03 ms (48) 4.79 ms
PyBind11 OpenMP merge 8.74 ms 5.04 ms 1.79 ms
[Read More]