Skip to content

[CI] Refactor Dockerfiles and installation scripts#18796

Merged
tqchen merged 1 commit intoapache:mainfrom
mshr-h:docker-cleanup
Feb 19, 2026
Merged

[CI] Refactor Dockerfiles and installation scripts#18796
tqchen merged 1 commit intoapache:mainfrom
mshr-h:docker-cleanup

Conversation

@mshr-h
Copy link
Contributor

@mshr-h mshr-h commented Feb 18, 2026

  • Removed Rust toolchain from all CI Docker images (no longer needed)
  • Deleted 8 obsolete scripts: Ubuntu 20.04-specific installers, old LLVM/CMake/Golang/sbt scripts
  • New download-and-verify utility: reusable curl + checksum verification helper, used across install scripts
  • CMake & sccache: switched from source/cargo builds to prebuilt binaries with checksum verification
  • LLVM: consolidated to one script, updated from Focal/LLVM 10-12 to Jammy/LLVM 15-17
  • APT key management: migrated from deprecated apt-key add to signed-by keyrings (LLVM, Vulkan, ROCm, Node.js)
  • Version upgrades: Emscripten 4.0.23, Node.js 22.x, TensorFlow 2.19.0, NumPy 1.26.*, Apache RAT 0.17, Wasmtime v41.0.3
  • Dockerfile maintainance: ENV KEY=value syntax, trap cleanup EXIT patterns, cmake --build instead of make, Python support narrowed to 3.10-3.11

The image builds are significantly faster now.

  • ci_arm: 37m -> 13m
  • ci_cpu: 44m -> 15m
  • ci_gpu: 53m -> 28m

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @mshr-h, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refines the Continuous Integration setup by upgrading numerous core dependencies and standardizing installation practices across various Docker environments. The changes aim to improve the reliability, security, and maintainability of the CI pipelines by centralizing download verification, updating GPG key handling, and leveraging pre-built binaries where appropriate. This cleanup also removes outdated Ubuntu 20.04 specific configurations, streamlining the overall CI infrastructure.

Highlights

  • CI Dockerfile Consolidation and Upgrades: Multiple CI Dockerfiles (ci_arm, ci_cpu, ci_gpu, ci_lint, ci_wasm) have been updated to streamline dependency installations, remove deprecated scripts, and upgrade various tools and libraries. This includes a shift from source-based CMake installation to binary, removal of Rust and Golang environment setups, and standardization of environment variable assignments.
  • New download_and_verify.sh Utility: A new utility script, download_and_verify.sh, was introduced to centralize and standardize the process of downloading files and verifying their checksums within the CI environment. This script replaces manual curl and sha256sum/sha512sum commands across several installation scripts.
  • Dependency Version Bumps and Installation Method Changes: Key dependencies such as LLVM, CMake, Emscripten, Apache RAT, sccache, wasmtime, TensorFlow, and NumPy have been upgraded to newer versions. For CMake, sccache, and wasmtime, the installation method was switched from building from source or using package managers to downloading pre-built binaries for improved efficiency and reliability.
  • Standardized GPG Key Handling: The method for adding GPG keys for various APT repositories (NVIDIA CUDA, LLVM, Node.js, ROCm, Vulkan) has been updated to use gpg --dearmor and store keys in /usr/share/keyrings, aligning with modern Debian/Ubuntu practices for secure package management.
  • Removal of Ubuntu 20.04 Specific Scripts: Several installation scripts specifically tailored for Ubuntu 20.04 (ubuntu2004_install_core.sh, ubuntu2004_install_python.sh, ubuntu2004_install_python_package.sh, ubuntu2004_install_redis.sh) have been removed, indicating a potential consolidation or deprecation of Ubuntu 20.04 support in the CI.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • docker/Dockerfile.ci_arm
    • Copied download_and_verify.sh to /usr/local/bin.
    • Switched from ubuntu_install_cmake_source.sh to ubuntu_install_cmake_binary.sh.
    • Removed Rust environment setup.
    • Standardized ENV PATH and ENV PYTHONNOUSERSITE assignments.
  • docker/Dockerfile.ci_cpu
    • Copied download_and_verify.sh to /usr/local/bin.
    • Switched from ubuntu_install_cmake_source.sh to ubuntu_install_cmake_binary.sh.
    • Removed Rust and Golang environment setups.
    • Standardized ENV TVM_VENV, ENV PATH, and ENV PYTHONNOUSERSITE assignments.
  • docker/Dockerfile.ci_gpu
    • Copied download_and_verify.sh to /usr/local/bin.
    • Updated NVIDIA GPG key handling to use gpg --dearmor.
    • Switched from ubuntu_install_cmake_source.sh to ubuntu_install_cmake_binary.sh.
    • Removed Rust environment setup.
    • Standardized ENV TVM_VENV, ENV PATH, ENV PYTHONNOUSERSITE, ENV NVIDIA_DRIVER_CAPABILITIES, ENV CPLUS_INCLUDE_PATH, and ENV C_INCLUDE_PATH assignments.
  • docker/Dockerfile.ci_lint
    • Copied download_and_verify.sh to /usr/local/bin.
    • Removed Rust environment setup.
    • Standardized ENV TVM_VENV, ENV PATH, and ENV PYTHONNOUSERSITE assignments.
  • docker/Dockerfile.ci_wasm
    • Copied download_and_verify.sh to /usr/local/bin.
    • Switched from ubuntu_install_cmake_source.sh to ubuntu_install_cmake_binary.sh.
    • Removed Rust environment setup.
    • Standardized ENV TVM_VENV, ENV PATH, and ENV PYTHONNOUSERSITE assignments.
  • docker/install/ubuntu2004_install_core.sh
    • Removed file.
  • docker/install/ubuntu2004_install_python.sh
    • Removed file.
  • docker/install/ubuntu2004_install_python_package.sh
    • Removed file.
  • docker/install/ubuntu2004_install_redis.sh
    • Removed file.
  • docker/install/ubuntu2204_install_llvm.sh
    • Updated GPG key handling for LLVM repository to use gpg --dearmor.
    • Simplified LLVM package installation to only include llvm-17 and clang-17 related packages.
  • docker/install/ubuntu_install_aprofile_aem.sh
    • Changed trap cleanup 0 to trap cleanup EXIT.
    • Used download-and-verify for GCC and FVP downloads.
  • docker/install/ubuntu_install_clang_format.sh
    • Updated GPG key handling for LLVM repository to use gpg --dearmor.
    • Switched clang-format version to 17.
  • docker/install/ubuntu_install_cmake_binary.sh
    • Added new script to install CMake from pre-built binaries, supporting x86_64 and aarch64, and utilizing download-and-verify.
  • docker/install/ubuntu_install_cmake_source.sh
    • Removed file.
  • docker/install/ubuntu_install_dnnl.sh
    • Added cleanup function with trap cleanup EXIT.
    • Used download-and-verify for DNNL archive download.
  • docker/install/ubuntu_install_emscripten.sh
    • Upgraded Emscripten SDK version from 3.1.51 to 5.0.1.
  • docker/install/ubuntu_install_golang.sh
    • Removed file.
  • docker/install/ubuntu_install_googletest.sh
    • Used download-and-verify for Google Test archive download.
  • docker/install/ubuntu_install_llvm.sh
    • Removed file.
  • docker/install/ubuntu_install_nodejs.sh
    • Updated GPG key handling for Node.js repository to use gpg --dearmor.
  • docker/install/ubuntu_install_python_package.sh
    • Updated numpy package version from 1.21.* to 1.26.*.
  • docker/install/ubuntu_install_rat.sh
    • Updated Apache RAT version from 0.12 to 0.17.
    • Used download-and-verify for Apache RAT archive download.
  • docker/install/ubuntu_install_rocm.sh
    • Updated GPG key handling for ROCm repository to use gpg --dearmor.
  • docker/install/ubuntu_install_rust.sh
    • Renamed to docker/utils/download_and_verify.sh.
    • Refactored into a generic script for downloading files and verifying checksums, removing Rust-specific installation logic.
  • docker/install/ubuntu_install_sccache.sh
    • Switched from cargo install to downloading pre-built sccache binaries (v0.14.0).
    • Removed Rust-related environment variable setup and chmod for /opt/rust.
  • docker/install/ubuntu_install_tensorflow.sh
    • Upgraded TensorFlow version from 2.18.0 to 2.19.0 and simplified installation by removing explicit Keras and ml_dtypes installations.
  • docker/install/ubuntu_install_tensorflow_aarch64.sh
    • Updated numpy and tensorflow package versions to align with ubuntu_install_tensorflow.sh.
  • docker/install/ubuntu_install_tflite.sh
    • Updated Flatbuffers build commands to use cmake --build and cmake --install with parallel execution.
    • Updated TFLite build command to use cmake --build with parallel execution.
  • docker/install/ubuntu_install_vulkan.sh
    • Updated GPG key handling for Vulkan repository to use gpg --dearmor.
  • docker/install/ubuntu_install_wasmtime.sh
    • Switched from curl | bash installation to downloading pre-built wasmtime binaries (v41.0.3).
    • Removed Rust-related environment variable setup and rustup target add command.
  • tests/conda/build-environment.yaml
    • Updated llvmdev dependency from >=11 to =17.
  • tests/lint/git-clang-format.sh
    • Updated clang-format version check and error messages from 15 to 17.
  • tests/scripts/task_config_build_gpu.sh
    • Updated USE_LLVM configuration to use llvm-config-17.
  • tests/scripts/task_config_build_wasm.sh
    • Updated USE_LLVM configuration to use llvm-config-17.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/main.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant and valuable set of upgrades and cleanups to the CI infrastructure. The changes, such as upgrading dependencies, replacing apt-key with more secure methods, and switching to pre-built binaries for tools like CMake and sccache, are excellent improvements that will enhance security, build speed, and maintainability. The introduction of the download_and_verify.sh script is a great example of applying DRY principles. I have a few minor suggestions to improve consistency and robustness in some of the shell scripts.

@mshr-h mshr-h force-pushed the docker-cleanup branch 5 times, most recently from e3040bf to d13ecf1 Compare February 19, 2026 09:41
@mshr-h mshr-h changed the title [CI] Further upgrade and cleanup CI [CI] Refactor Dockerfiles and installation scripts Feb 19, 2026
@mshr-h mshr-h marked this pull request as ready for review February 19, 2026 10:20
@mshr-h
Copy link
Contributor Author

mshr-h commented Feb 19, 2026

cc @tqchen @yongwww

@tqchen tqchen merged commit 72f5185 into apache:main Feb 19, 2026
10 checks passed
@tqchen
Copy link
Member

tqchen commented Feb 19, 2026

would be nice if we can followup with uv

@mshr-h mshr-h deleted the docker-cleanup branch February 19, 2026 15:50
@mshr-h
Copy link
Contributor Author

mshr-h commented Feb 19, 2026

for sure! it's my next step 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments