Skip to content

[fix](paimon) install paimon-cpp arrow static deps into isolated dir #60730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xylaaaaa wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[fix](paimon) install paimon-cpp arrow static deps into isolated dir #60730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6b757a0
[fix](paimon) install paimon-cpp arrow static deps into isolated dir
xylaaaaa Feb 13, 2026
5b82128
[chore](paimon) link prior thirdparty PR #60296
xylaaaaa Feb 13, 2026
6687f97
lower parallel
morningman Feb 13, 2026
d6c7652
add log
morningman Feb 13, 2026
c13a9ed
test
morningman Feb 14, 2026
8121303
add tmp dir
morningman Feb 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions .github/workflows/build-thirdparty.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,18 @@ jobs:
export CMAKE_POLICY_VERSION_MINIMUM=3.10
export CUSTOM_CMAKE="/usr/local/bin/cmake"

export TMPDIR=/home/runner/work/doris/doris/.tmp
export TMP=$TMPDIR
export TEMP=$TMPDIR
mkdir -p $TMPDIR

df -h
echo $TMPDIR
echo $RUNNER_TEMP

cd thirdparty
./build-thirdparty.sh -j "$(nproc)"
#./build-thirdparty.sh -j "$(nproc)"
./build-thirdparty.sh -j 2

build_macos:
name: Build Third Party Libraries (macOS)
Expand Down Expand Up @@ -195,7 +205,12 @@ jobs:
export CMAKE_POLICY_VERSION_MINIMUM=3.10
export CUSTOM_CMAKE="/usr/local/bin/cmake"

df -h
echo $TMPDIR
echo $RUNNER_TEMP

cd thirdparty
#./build-thirdparty.sh -j "$(nproc)"
./build-thirdparty.sh -j "$(nproc)"

build_macos_arm64:
Expand Down Expand Up @@ -255,6 +270,11 @@ jobs:
export CMAKE_POLICY_VERSION_MINIMUM=3.10
export CUSTOM_CMAKE="/usr/local/bin/cmake"

df -h
echo $TMPDIR
echo $RUNNER_TEMP

cd thirdparty
./build-thirdparty.sh -j "$(nproc)"
#./build-thirdparty.sh -j "$(nproc)"
./build-thirdparty.sh -j 2

18 changes: 18 additions & 0 deletions thirdparty/build-thirdparty.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2028,6 +2028,21 @@ build_paimon_cpp() {
# These libraries are built but not installed by default
echo "Installing paimon-cpp internal dependencies..."

# Install paimon-cpp Arrow deps used by paimon parquet static libs.
# Keep them in an isolated directory to avoid clashing with Doris Arrow.
local paimon_deps_dir="${TP_INSTALL_DIR}/paimon-cpp/lib64/paimon_deps"
mkdir -p "${paimon_deps_dir}"
for paimon_arrow_dep in \
libarrow.a \
libarrow_filesystem.a \
libarrow_dataset.a \
libarrow_acero.a \
libparquet.a; do
if [ -f "arrow_ep-install/lib/${paimon_arrow_dep}" ]; then
Copy link

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The paths hardcode both the destination libdir (lib64) and the Arrow EP libdir (arrow_ep-install/lib). This makes the logic brittle if the build outputs to lib64 (or a different layout) in some environments/configurations. Consider deriving the lib directory from existing build variables (or probing arrow_ep-install/lib64 vs .../lib) and using a single computed arrow_lib_dir/paimon_lib_dir to avoid repeated assumptions.

Copilot uses AI. Check for mistakes.
cp -v "arrow_ep-install/lib/${paimon_arrow_dep}" "${paimon_deps_dir}/${paimon_arrow_dep}"
Copy link

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop silently skips missing static libs, which can turn into harder-to-debug link failures later (especially if Arrow’s built targets differ by version/options). If these archives are required, fail fast when any expected file is missing (e.g., emit an error and return non-zero). If they’re optional, at least log an explicit warning per missing library so the build output explains what happened.

Suggested change
cp -v "arrow_ep-install/lib/${paimon_arrow_dep}" "${paimon_deps_dir}/${paimon_arrow_dep}"
cp -v "arrow_ep-install/lib/${paimon_arrow_dep}" "${paimon_deps_dir}/${paimon_arrow_dep}"
else
echo "Warning: expected Arrow static library 'arrow_ep-install/lib/${paimon_arrow_dep}' not found; paimon-cpp static linking may fail if this archive is required." >&2

Copilot uses AI. Check for mistakes.
fi
done

# Install roaring_bitmap, renamed to avoid conflict with Doris's croaringbitmap
if [ -f "release/libroaring_bitmap.a" ]; then
cp -v "release/libroaring_bitmap.a" "${TP_INSTALL_DIR}/lib64/libroaring_bitmap_paimon.a"
Comment on lines +2033 to 2048
Copy link

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The paths hardcode both the destination libdir (lib64) and the Arrow EP libdir (arrow_ep-install/lib). This makes the logic brittle if the build outputs to lib64 (or a different layout) in some environments/configurations. Consider deriving the lib directory from existing build variables (or probing arrow_ep-install/lib64 vs .../lib) and using a single computed arrow_lib_dir/paimon_lib_dir to avoid repeated assumptions.

Suggested change
local paimon_deps_dir="${TP_INSTALL_DIR}/paimon-cpp/lib64/paimon_deps"
mkdir -p "${paimon_deps_dir}"
for paimon_arrow_dep in \
libarrow.a \
libarrow_filesystem.a \
libarrow_dataset.a \
libarrow_acero.a \
libparquet.a; do
if [ -f "arrow_ep-install/lib/${paimon_arrow_dep}" ]; then
cp -v "arrow_ep-install/lib/${paimon_arrow_dep}" "${paimon_deps_dir}/${paimon_arrow_dep}"
fi
done
# Install roaring_bitmap, renamed to avoid conflict with Doris's croaringbitmap
if [ -f "release/libroaring_bitmap.a" ]; then
cp -v "release/libroaring_bitmap.a" "${TP_INSTALL_DIR}/lib64/libroaring_bitmap_paimon.a"
# Determine paimon-cpp library directory (lib vs lib64)
local paimon_lib_root="${TP_INSTALL_DIR}/paimon-cpp"
local paimon_lib_dir="${paimon_lib_root}/lib"
if [ ! -d "${paimon_lib_dir}" ]; then
paimon_lib_dir="${paimon_lib_root}/lib64"
fi
local paimon_deps_dir="${paimon_lib_dir}/paimon_deps"
mkdir -p "${paimon_deps_dir}"
# Determine Arrow EP library directory (lib vs lib64)
local arrow_lib_dir="arrow_ep-install/lib"
if [ ! -d "${arrow_lib_dir}" ]; then
arrow_lib_dir="arrow_ep-install/lib64"
fi
for paimon_arrow_dep in \
libarrow.a \
libarrow_filesystem.a \
libarrow_dataset.a \
libarrow_acero.a \
libparquet.a; do
if [ -f "${arrow_lib_dir}/${paimon_arrow_dep}" ]; then
cp -v "${arrow_lib_dir}/${paimon_arrow_dep}" "${paimon_deps_dir}/${paimon_arrow_dep}"
fi
done
# Install roaring_bitmap, renamed to avoid conflict with Doris's croaringbitmap
# Determine top-level TP lib directory (lib vs lib64)
local tp_lib_dir="${TP_INSTALL_DIR}/lib"
if [ ! -d "${tp_lib_dir}" ]; then
tp_lib_dir="${TP_INSTALL_DIR}/lib64"
fi
if [ -f "release/libroaring_bitmap.a" ]; then
cp -v "release/libroaring_bitmap.a" "${tp_lib_dir}/libroaring_bitmap_paimon.a"

Copilot uses AI. Check for mistakes.
Expand Down Expand Up @@ -2259,6 +2274,9 @@ for package in "${packages[@]}"; do
${command}
cd "${TP_DIR}"
cleanup_package_source "${package}"
echo "debug after clean: ${package}"
df -h
du -sh "${TP_DIR}"
fi
done

Expand Down
Loading