Bazel aims to provide a hermetic environment for your build & test operations (“actions”) to run in. However by default Bazel does not provide a hermetic C / C++ (CC) toolchain. If you would like to learn more about this and why it is important, then I suggest you read this article by Thulio Ferraz Assis at Aspect.

Alongside their article, they developed a solution to enable a hermetic GCC compiler and associated Linux sysroot package. Their solution is great if you want to use GCC. But if you instead want to use a hermetic llvm-clang compiler toolchain, then we are going to have to make some modifications.

By the end of this article you will be able to generate your own Linux sysroot package and combine it together with llvm-clang in order to provide a hermetic CC toolchain capable of cross-compilation.

Link to source code: https://github.com/scasagrande/toolchains_llvm_sysroot

Please leave your questions and comments under the discussions tab on the repository.

toolchains_llvm ruleset Link to heading

To start with, we are going to use the toolchains_llvm ruleset in order to fetch and setup the actual toolchain. This ruleset allows you to specify a Bazel target that contains your sysroot package files.

Before we set it up, you’ll need to choose if you’re going to use llvm-libc++ or if you are going to use libstdc++. The former is the default for the ruleset (when doing native compilation builds), it is the common libc++ implementation for macOS builds, and is included as part of the standard llvm packages. The latter, libstdc++, is more common for Linux builds. In both cases we will still require a separate sysroot package in order to provide a hermetic CC toolchain environment. Here I will be using libstdc++ for our Linux target platform builds, and libc++ for our macOS target platform builds.

Without a sysroot package, you might use this ruleset to define your CC toolchain as follows. This will enable a CC toolchain for Linux and macOS on both x86_64/amd64 and aarch64/arm64.

MODULE.bazel

 1bazel_dep(name = "toolchains_llvm", version = "1.1.2", dev_dependency = True)
 2
 3# Configure and register the toolchain.
 4llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm", dev_dependency = True)
 5llvm.toolchain(
 6    name = "llvm_toolchain",
 7    llvm_version = "15.0.2",
 8    stdlib = {
 9        "linux-x86_64": "stdc++",
10        "linux-aarch64": "stdc++",
11    }
12)
13
14use_repo(llvm, "llvm_toolchain")
15
16register_toolchains("@llvm_toolchain//:all", dev_dependency = True)

or for projects using the legacy WORKSPACE system:

WORKSPACE.bazel

 1load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 2
 3http_archive(
 4    name = "toolchains_llvm",
 5    sha256 = "e91c4361f99011a54814e1afbe5c436e0d329871146a3cd58c23a2b4afb50737",
 6    strip_prefix = "toolchains_llvm-1.0.0",
 7    url = "https://github.com/bazel-contrib/toolchains_llvm/releases/download/1.0.0/toolchains_llvm-1.0.0.tar.gz",
 8)
 9
10load("@toolchains_llvm//toolchain:deps.bzl", "bazel_toolchain_dependencies")
11
12bazel_toolchain_dependencies()
13
14load("@toolchains_llvm//toolchain:rules.bzl", "llvm_toolchain")
15
16llvm_toolchain(
17    name = "llvm_toolchain",
18    llvm_version = "15.0.2",
19    stdlib = {
20        "linux-x86_64": "stdc++",
21        "linux-aarch64": "stdc++",
22    }
23)
24
25load("@llvm_toolchain//:toolchains.bzl", "llvm_register_toolchains")
26
27llvm_register_toolchains()

Sysroot package generation Link to heading

Link to source code: https://github.com/scasagrande/toolchains_llvm_sysroot

Now we need to provide a sysroot package in order to ensure that the build is using a consistent version of our system libraries. As a bonus, we will generate a matching package for both x86_64 and aarch64 CPU architectures.

To programmatically construct these packages, I started with the sysroot generation code in the gcc-toolchain repo. Although it’s designed to build a sysroot package for that specific toolchain ruleset, we can modify it to meet the needs of toolchains_llvm. It’s also already set up to enable the generation of sysroot packages for both x86_64 and aarch64.

With this starting point, I made the following changes:

• Updated the following system libraries to target a Red Hat Enterprise Linux (RHEL) 8 environment
  • Kernel 4.18
  • glibc 2.28
  • libstdc++ 10.3
• Updated toolchains used to versions that support building those above libraries
• Removed a bunch of the files that we aren’t going to need, such as binaries
• Moved the files around, renamed some folders, and placed everything under a standard /usr layout

With these changes, you’ll have the basics that you need to provide a hermetic CC toolchain to Bazel.

In my case, I also decided to expand the sysroot package with openssl and cyrus-sasl, copied from a RHEL-UBI 8 container image. Openssl is a very common build dependency and is useful to have in your sysroot. For those of us targeting a RHEL 8 production environment, it can be very helpful to have this specific copy of openssl for FIPS compliance. But we’re not going to get into that today, and instead just focus on the fact that we are including these optional libraries in our sysroot package.

So now lets go ahead and build it. These commands will build our two sysroot packages with these extra ssl libraries bundled in, and output them into our current directory.

$ ./build.sh x86_64 . ssl
$ ./build.sh aarch64 . ssl

If you want to omit these ssl related packages, then replace ssl with base.

After some processing time, you will be left with 2 tar.xz files. If you have been following along, the x86_64 file should be approximately 55MB, and the aarch64 file approximately 35MB.

Using the sysroot package Link to heading

First we need to make the tar.xz files available to Bazel. You’ll need to make your sysroot package available somewhere for your team to download, such as in Artifactory. How you do that is up to you. For testing, you can put in a fake URL and use the --override_repository= CLI argument built into Bazel.

MODULE.bazel

 1http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 2
 3http_archive(
 4    name = "sysroot_linux_x86_64_2_28",
 5    url = "https://example.com/sysroot-x86_64-ssl.tar.xz",
 6    sha256 = "ABCD1234",
 7    build_file = "//external:BUILD.sysroot.bazel"
 8)
 9
10http_archive(
11    name = "sysroot_linux_aarch64_2_28",
12    url = "https://example.com/sysroot-aarch64-ssl.tar.xz",
13    sha256 = "1234ABCD",
14    build_file = "//external:BUILD.sysroot.bazel"
15)

WORKSPACE.bazel

 1load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 2
 3http_archive(
 4    name = "sysroot_linux_x86_64_2_28",
 5    url = "https://example.com/sysroot-x86_64-ssl.tar.xz",
 6    sha256 = "ABCD1234",
 7    build_file = "//external:BUILD.sysroot.bazel"
 8)
 9
10http_archive(
11    name = "sysroot_linux_aarch64_2_28",
12    url = "https://example.com/sysroot-aarch64-ssl.tar.xz",
13    sha256 = "1234ABCD",
14    build_file = "//external:BUILD.sysroot.bazel"
15)

external/BUILD.sysroot.bazel

1filegroup(
2    name = "sysroot",
3    srcs = glob(["**"]),
4    visibility = ["//visibility:public"]
5)

And then we just need to update our toolchain definition to use these new sysroot package when building and targeting linux x86_64 and aarch64.

MODULE.bazel

 1bazel_dep(name = "toolchains_llvm", version = "1.1.2", dev_dependency = True)
 2
 3llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm", dev_dependency = True)
 4llvm.toolchain(
 5    llvm_version = "15.0.2",
 6    stdlib = {
 7        "linux-x86_64": "stdc++",
 8        "linux-aarch64": "stdc++",
 9    },
10)
11llvm.sysroot(
12    name = "llvm_toolchain",
13    label = "@@sysroot_linux_x86_64_2_28//:sysroot",
14    targets = ["linux-x86_64"],
15)
16llvm.sysroot(
17    name = "llvm_toolchain",
18    label = "@@sysroot_linux_aarch64_2_28//:sysroot",
19    targets = ["linux-aarch64"],
20)
21use_repo(llvm, "llvm_toolchain")
22
23register_toolchains("@llvm_toolchain//:all", dev_dependency = True)

WORKSPACE.bazel

 1llvm_toolchain(
 2    name = "llvm_toolchain",
 3    llvm_version = "15.0.2",
 4    stdlib = {
 5        "linux-x86_64": "stdc++",
 6        "linux-aarch64": "stdc++",
 7    }
 8    sysroot = {
 9        "linux-x86_64": "@sysroot_linux_x86_64_2_28//:sysroot",
10        "linux-aarch64": "@sysroot_linux_aarch64_2_28//:sysroot",
11    }
12)

Optionally, we can test our build without having uploaded the tar.xz. Start with unpacking the tar file into a folder on your filesystem. Next make a BUILD.bazel file located at the root of this extracted folder (that is, beside the extracted usr/ folder) with the contents from your BUILD.sysroot.bazel file.

Finally, to execute your test, run the following if you are building on a Linux x86_64 machine:

$ bazel build --override_repository=sysroot_linux_x86_64_2_28=/path/to/sysroot-x86_64-ssl //...

Result Link to heading

In the end you should now be able to build and test targeting linux x86_64 or aarch64, both with native builds or cross compilation, including from macOS!

But does that mean that you now have a fully hermetic CC build? Well, that is going to depend on the rest of your project. For example, if you have rust dependencies fetched via rules_rust , some of them may include build.rs files that perform their own CC compilation that breaks out of the Bazel sandbox, such as rdkafka-sys, openssl-sys, or sasl2-sys. In the future I will cover how best to deal with these cases, while also using our new sysroot packages.

Feedback Link to heading

Please leave your questions and comments in this discussions thread on the repository.

I can also be reached on the Bazel slack community under my full name.