Skip to content

[thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH#60752

Open
SwingURM wants to merge 1 commit intoapache:masterfrom
SwingURM:thirdparty/fix-pkg-config
Open

[thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH#60752
SwingURM wants to merge 1 commit intoapache:masterfrom
SwingURM:thirdparty/fix-pkg-config

Conversation

@SwingURM
Copy link

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
When compiling hyperscan, the container's behavior is to give up compiling hscollider because it cannot find pcre.

[ ~]# pkg-config --exists libpcre && echo "success"
[ ~]# env | grep PKG
PKG_CONFIG_PATH=/root/doris/thirdparty/installed/lib64/pkgconfig

On my local machine, PKG_CONFIG_PATH (which is set in the script) does not prevent pkg-config from searching system libs. Thus, CMake's pkg_check_modules finds pcre and tries to compile hscollider, but fails due to mixing system's libc (linker errors).

~/ldb_toolchain/bin via  v3.14.2 
❯ ./pkg-config --exists libpcre && echo "find pcre"
find pcre

~/ldb_toolchain/bin via  v3.14.2 
❯ env | grep PKG                                   
PKG_CONFIG_PATH=/home/iamswing/doris/thirdparty/installed/lib64/pkgconfig

However, on my local machine,
Error:

src/hyperscan-5.4.2/doris_build on  master [!?] via  v15.2.1-gcc via  v4.2.3
❯ /home/iamswing/ldb_toolchain/bin/clang++ -D_HAS_AUTO_PTR_ETC=0 -O2 -std=c++11 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -DNDEBUG -fno-omit-frame-pointer -fvisibility=hidden -Wvla -Wpointer-arith -Wno-self-assign -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-bitwise-instead-of-logical -Wno-ignored-attributes -Wno-redundant-move -Wweak-vtables -Wmissing-declarations  -g -DNDEBUG  tools/hscollider/CMakeFiles/hscollider.dir/Corpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/FileCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/ColliderCorporaParser.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/NfaGeneratedCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GraphTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GroundTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/UltimateTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/args.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/limit.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/pcre_util.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/Thread.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/main.cpp.o -o bin/hscollider  lib/libhs.a  -L/usr/lib  -lpcre  lib/libdatabaseutil.a  lib/libexpressionutil.a  lib/libcorpusomatic.a  lib/libcrosscompileutil.a  -lpthread  -rdynamic -v
clang version 20.1.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/iamswing/ldb_toolchain/tmp/gentoo/llvm/bin
System configuration file directory: /home/iamswing/ldb_toolchain/bin/../etc/clang
User configuration file directory: /home/iamswing/.config/clang
Selected GCC installation: /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/home/iamswing/ldb_toolchain/bin/x86_64-pc-linux-gnu-ld" --sysroot=/home/iamswing/ldb_toolchain/bin/.. --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -export-dynamic -pie -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o bin/hscollider /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o /home/iamswing/ldb_toolchain/bin/../usr/lib/crti.o /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/crtbeginS.o -L/usr/lib -L/home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15 -L/home/iamswing/ldb_toolchain/bin/../lib -L/home/iamswing/ldb_toolchain/bin/../usr/lib tools/hscollider/CMakeFiles/hscollider.dir/Corpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/FileCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/ColliderCorporaParser.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/NfaGeneratedCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GraphTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GroundTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/UltimateTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/args.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/limit.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/pcre_util.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/Thread.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/main.cpp.o lib/libhs.a -lpcre lib/libdatabaseutil.a lib/libexpressionutil.a lib/libcorpusomatic.a lib/libcrosscompileutil.a -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/crtendS.o /home/iamswing/ldb_toolchain/bin/../usr/lib/crtn.o
x86_64-pc-linux-gnu-ld: error: undefined symbol: __libc_csu_fini
>>> referenced by /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o:(_start)

x86_64-pc-linux-gnu-ld: error: undefined symbol: __libc_csu_init
>>> referenced by /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o:(_start)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Solution

Use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH. PKG_CONFIG_PATH appends to the default search path, causing CMake to find the system's pcre. Using PKG_CONFIG_LIBDIR overrides the default search path and only searches in the specified directory. All relevant .pc files are present in this path.

Modifying CMake's related variables may also solve this problem.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • [☑️] Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • [☑️] No.
    • Yes.
  • Does this need documentation?

    • [☑️] No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 13, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@SwingURM SwingURM changed the title [thirdparty] use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH [thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH Feb 13, 2026
@SwingURM
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28471 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17652	4467	4301	4301
q2	q3	10654	768	515	515
q4	4686	351	249	249
q5	7556	1210	1017	1017
q6	171	174	144	144
q7	766	825	671	671
q8	9294	1457	1311	1311
q9	5123	4751	4682	4682
q10	6840	1867	1624	1624
q11	486	263	253	253
q12	695	561	467	467
q13	17762	4215	3389	3389
q14	231	231	215	215
q15	963	789	799	789
q16	747	709	659	659
q17	709	857	438	438
q18	6021	5423	5143	5143
q19	1163	993	602	602
q20	515	496	374	374
q21	4479	1827	1389	1389
q22	343	288	239	239
Total cold run time: 96856 ms
Total hot run time: 28471 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4397	4348	4342	4342
q2	q3	1773	2165	1706	1706
q4	824	1144	813	813
q5	4033	4318	4321	4318
q6	174	179	142	142
q7	1703	1602	1492	1492
q8	2406	2653	2491	2491
q9	7445	7352	7357	7352
q10	2713	2825	2455	2455
q11	515	431	426	426
q12	493	581	434	434
q13	3990	4605	3691	3691
q14	277	310	290	290
q15	868	818	846	818
q16	771	836	799	799
q17	1196	1540	1297	1297
q18	7063	6813	6756	6756
q19	894	909	841	841
q20	2065	2163	2127	2127
q21	3959	3472	3277	3277
q22	457	448	412	412
Total cold run time: 48016 ms
Total hot run time: 46279 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184583 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

query5	5029	658	537	537
query6	319	219	198	198
query7	4221	487	265	265
query8	346	244	234	234
query9	8694	2765	2791	2765
query10	541	388	353	353
query11	16931	17408	17313	17313
query12	230	128	130	128
query13	1380	481	351	351
query14	7335	3446	3243	3243
query14_1	2952	3015	2986	2986
query15	215	202	182	182
query16	1013	491	478	478
query17	1245	816	673	673
query18	2854	487	357	357
query19	223	229	206	206
query20	151	133	144	133
query21	213	149	123	123
query22	5438	5102	4837	4837
query23	17231	16824	16588	16588
query23_1	16603	16707	16681	16681
query24	7171	1705	1271	1271
query24_1	1261	1260	1274	1260
query25	550	485	443	443
query26	1234	273	159	159
query27	2748	489	289	289
query28	4443	1875	1949	1875
query29	794	595	493	493
query30	309	241	218	218
query31	877	717	636	636
query32	86	73	76	73
query33	518	345	285	285
query34	939	944	576	576
query35	644	679	585	585
query36	1078	1082	926	926
query37	132	98	86	86
query38	2955	2913	2915	2913
query39	884	894	849	849
query39_1	834	834	831	831
query40	224	151	133	133
query41	67	58	56	56
query42	106	103	104	103
query43	376	397	357	357
query44	
query45	197	192	190	190
query46	938	1028	628	628
query47	2137	2133	2135	2133
query48	313	334	222	222
query49	636	472	384	384
query50	716	282	220	220
query51	4096	4170	4077	4077
query52	107	113	101	101
query53	337	368	305	305
query54	297	269	270	269
query55	89	84	79	79
query56	313	307	294	294
query57	1338	1331	1247	1247
query58	294	273	272	272
query59	2525	2645	2619	2619
query60	355	333	316	316
query61	143	146	142	142
query62	624	590	550	550
query63	330	287	286	286
query64	4861	1266	987	987
query65	
query66	1381	496	359	359
query67	16343	16468	16333	16333
query68	
query69	398	307	294	294
query70	990	991	861	861
query71	337	316	299	299
query72	2797	2698	2434	2434
query73	550	588	327	327
query74	9938	9941	9757	9757
query75	2996	2854	2526	2526
query76	2297	1128	706	706
query77	380	387	312	312
query78	11201	11453	10663	10663
query79	2383	856	612	612
query80	1772	662	576	576
query81	550	285	249	249
query82	997	160	124	124
query83	345	270	244	244
query84	248	124	103	103
query85	912	482	430	430
query86	420	310	288	288
query87	3218	3092	2962	2962
query88	3614	2683	2671	2671
query89	451	381	366	366
query90	1955	185	172	172
query91	159	158	131	131
query92	74	74	77	74
query93	1046	888	507	507
query94	624	326	290	290
query95	581	340	376	340
query96	654	550	234	234
query97	2478	2491	2437	2437
query98	220	209	220	209
query99	995	971	891	891
Total cold run time: 256564 ms
Total hot run time: 184583 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.69% (19523/37050)
Line Coverage 36.25% (182046/502248)
Region Coverage 32.62% (141273/433120)
Branch Coverage 33.64% (61220/181963)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.80% (26065/36303)
Line Coverage 54.49% (272998/500999)
Region Coverage 52.03% (227640/437497)
Branch Coverage 53.46% (97659/182667)

@SwingURM
Copy link
Author

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants