run_binary() a py_binary() fails on a pure system (NixOS)

[hzeller]

A binary build with py_binary() will not run with run_binary() on a pure system (NixOS), but will work when using a genrule and explicitly choosing the Python toolchain coming from rules_python:

Example:

py_binary(
    name = "hello",
    srcs = ["hello.py"],
    main = "hello.py",
)

run_binary(
    name = "hello_run_binary",
    outs = ["hello_run_binary.out"],
    args = ["$(location hello_run_binary.out)"],
    tool = ":hello",
)

genrule(
    name = "hello_genrule",
    outs = ["hello_genrule.out"],
    cmd = "$(PYTHON3) $(location :hello) $@",
    toolchains = ["@rules_python//python:current_py_toolchain"],
    tools = [":hello"],
)

The MODULE.bazel gets latest rules_python and also registers the toolchain

bazel_dep(name = "bazel_skylib", version = "1.9.0")
bazel_dep(name = "rules_python", version = "1.8.3")

# Registering toolchain to leave no doubt
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.11",
    is_default = True,
)

Running these targets with bazel 7 or bazel 8 on a pure system:

bazel build :hello_genrule    # works
bazel build :hello_run_binary # fails with "env: 'python3': No such file or directory"

Important is it to run on a pure hermetic systems such as NixOS, it seems to (accidentally ?) be working on impure systems such as Ubuntu.

Here is the full example with MODULE.bazel, BUILD and hello.py as well with a shell.nix

https://github.com/hzeller/python-run_binary-issue

Not marked as bug right now as I don't know if working as intended, and one has to always use genrule(), or if run_binary() is supposed to work, but for whatever reason it is not using the proper toolchain.

Context

I attempt to make the glib BZR module work by replacing run_binary() with genrule(), but in the review, the maintainer suspects this might actually be an issue in rules_python instead:

bazelbuild/bazel-central-registry#7467

[rickeylev]

This is because the system_python bootstrap is being used, which requires python3 to be available on the system to do initial bootstrap.

Set config_settings={"@rules_python///python/config_settings:bootstrap_impl=script"} (or the equiv in your bazelrc) on the binary, and it'll switch to a shell based bootstrap. Note that it requires bash.

[hzeller]

So to make running binaries more robust without special config settings in .bazelrc it sounds like using genrule() with the toolchain argument instead of run_binary() is the way to go ?

[hzeller]

Even when setting the config setting on the binary (I suppose you mean the py_binary()) and with bash in path:

py_binary(
    name = "hello",
    srcs = ["hello.py"],
    main = "hello.py",
    config_settings={
        "@rules_python//python/config_settings:bootstrap_impl" : "script"
    }
)

... this fails

$ bazel build :hello_run_binary --verbose_failures
Starting local Bazel server (8.5.0- (@non-git)) and connecting to it...
INFO: Invocation ID: ce4f1c0d-47a8-450c-9c64-f3ffcae2db58
INFO: Analyzed target //:hello_run_binary (81 packages loaded, 2897 targets configured).
ERROR: /home/hzeller/src/my/python-run_binary-issue/BUILD:13:11: RunBinary hello_run_binary.out failed: (Exit 127): hello failed: error executing RunBinary command (from target //:hello_run_binary) 
  (cd /home/hzeller/.cache/bazel/_bazel_hzeller/0fff3e273ca8fa912137b004d1a9fbfb/sandbox/processwrapper-sandbox/1/execroot/_main && \
  exec env - \
  bazel-out/k8-opt-exec-ST-e998660a2af8/bin/hello bazel-out/k8-opt/bin/hello_run_binary.out)
# Configuration: 5ff5d8df9857f5146f18faf0e81aff8664343fe7d4c7323ec392005f5b9e7807
# Execution platform: @@platforms//host:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
bazel-out/k8-opt-exec-ST-e998660a2af8/bin/hello: line 312: exec: env: not found
Target //:hello_run_binary failed to build
INFO: Elapsed time: 3.448s, Critical Path: 0.21s
INFO: 11 processes: 11 internal.
ERROR: Build did NOT complete successfully

[rickeylev]

So to make running binaries more robust without special config settings in .bazelrc it sounds like using genrule() with the toolchain argument instead of run_binary() is the way to go ?

It's not more robust, no. What it's doing is assuming the output of py_binary is itself a python file, which can be passed directly to a Python interpreter. This is only incidentally true with the system_python bootstrap, and isn't true with the shell bootstrap. In general, the output of a Bazel executable rule is some self-executable program, so trying to use it as the input for another program is unsound. Eventually, we're going to change the default to the script bootstrap.

... line 312: exec: env: not found

I'm not familiar with Nix, but it sounds like it doesn't have things like coreutils available? But that doesn't quite track because the first line of the bootstrap is /usr/bin/env, so how did it get all the way to the end? Is /usr/bin not on PATH on Nix?

You can have a look at https://github.com/bazel-contrib/rules_python/blob/main/python/private/stage1_bootstrap_template.sh to see the system utilities that the script bootstrap currently requires.

I'd be open to PRs to make it more portable. Just note that, since we don't have a Nix CI, we can't prevent regressions.

You might also be interested in #2500 (build a native binary to bootstrap instead).

[hzeller]

It's not more robust, no. What it's doing is assuming the output of py_binary is itself a python file, which can be passed directly to a Python interpreter.

So maybe instead of wrapping that python script in a py_binary() whose output might not be a python binary, it would be better to directly invoke the Python file with the $(PYTHON3) interpreter (in a genrule(), then)
After all, the scripts in the glib project are not huge python applications, just scripts to emit defines and such.

This is only incidentally true with the system_python bootstrap, and isn't true with the shell bootstrap. In general, the output of a Bazel executable rule is some self-executable program, so trying to use it as the input for another program is unsound. Eventually, we're going to change the default to the script bootstrap.

When you do, make sure to use /bin/sh not bash as bash is not required on a Posix system, but /bin/sh is.

... line 312: exec: env: not found

I'm not familiar with Nix, but it sounds like it doesn't have things like coreutils available? But that doesn't quite track because the first line of the bootstrap is /usr/bin/env, so how did it get all the way to the end? Is /usr/bin not on PATH on Nix?

Nix only provides the Posix required binaries at the Posix required paths.
That is /bin/sh as only binary in /bin and /usr/bin/env as only binary in /usr/bin.

Core-utils are available but are reached with the PATH. However, since bazel does not pass through the PATH that doesn't help. So if bash is installed, you can find it with /usr/bin/env bash, but in bazel only if use_default_shell_env.
Core utils such as cat and awk are probably available, but bash or python are of course already special tools that you can never assume to be on any system. To be hermetic in bazel, it is good to use a module that provides core tools.

I'd be open to PRs to make it more portable. Just note that, since we don't have a Nix CI, we can't prevent regressions.

I think we should have a Nix CI on BZR, bazel's leaky implementation of hermeticity is really put to a test on Nix and it can highlight many of the shortcomings. A large amount of developers use Nix these days as it is a very good way of creating a hermetic system.

You might also be interested in #2500 (build a native binary to bootstrap instead).

[rickeylev]

use genrule with $python3 if its a single self-contained py file

Yes, that is perfectly fine, and probably about the lowest overhead way to do it.

env is available at /usr/bin, but that isn't on PATH in strict bazel shell

Thanks! That should be a simple fix in the shell bootstrap, then. We can just add some command calls and logic to try env and fallback to /usr/bin/env.

make sure to use /bin/sh not bash

Unfortunately, the bootstrap has a bunch of bashisms in it, namely arrays. I tried rewriting it to posix shell a few months ago and it was quite annoying and I lost momentum. Happy to review PRs, though.