Skip to content
Closed

~ #2325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
221 commits
Select commit Hold shift + click to select a range
5a920fa
WIP for FileSource connector
ddelnano Jan 24, 2025
5438415
Working FileSource connector with occassional segfaults from race con…
ddelnano Jan 29, 2025
1023689
Add pxlog module, FileSource function and the associated tests
ddelnano Jan 29, 2025
7606f37
Add file source mutation support to query broker
ddelnano Jan 29, 2025
8f2ae0e
Add file source controller to metadata service
ddelnano Jan 30, 2025
9d62c51
Integrate file source controller into metadata service
ddelnano Jan 30, 2025
08356f4
Register metadata service's FileSource RPC methods. Add missing mutat…
ddelnano Jan 30, 2025
b4da693
Add GetFileSourceStatus UDTF
ddelnano Jan 30, 2025
0b2c17c
Remove debug message
ddelnano Jan 31, 2025
d10362b
Ensure FileSourceManager is passed through to avoid segfault
ddelnano Jan 31, 2025
c6ff273
Add pxl support for deleting file sources
ddelnano Jan 31, 2025
6f73901
Fix mutation executor bug that prevented UI from showing table result…
ddelnano Jan 31, 2025
4c73e71
FileSourceConnector and PEM changes to get the system working
ddelnano Jan 31, 2025
2c91ee2
Update go protos
ddelnano Jan 31, 2025
c9f452a
Linting changes
ddelnano Jan 31, 2025
7269101
Move Table::Cursor to a non nested class and introduce base and sub c…
ddelnano Feb 4, 2025
62930a6
Add bare bones HotOnlyTable impl
ddelnano Feb 5, 2025
af28ec3
WIP implementing HotOnlyTable
ddelnano Feb 5, 2025
64293fc
Fix StoreWithRowAccounting.PopFront bug (returning unalloc'ed memory)…
ddelnano Feb 6, 2025
d9528c9
Update standalone PEM's file source manager to work with HotOnlyTable
ddelnano Feb 7, 2025
0bfb559
Fix use after free issue detected by ASAN
ddelnano Feb 7, 2025
4b0552b
Determine accurate file position by clearing error state (eof)
ddelnano Feb 7, 2025
c434c9b
Use vector auto generated file in standalone pem client script
ddelnano Feb 7, 2025
22970aa
Add stirling file source connector e2e test
ddelnano Feb 7, 2025
c2c8259
Linting changes
ddelnano Feb 7, 2025
903a64a
Refactor file parsing to make it easier to support other formats
ddelnano Feb 7, 2025
4565751
Add stream_status table to track throughput of file sources
ddelnano Feb 7, 2025
0bf70cf
Add px-log-generator for use with Pixie log source
ddelnano Feb 11, 2025
e5ab803
Add initial version of sink_results table within OTelExportSinkNode
ddelnano Feb 12, 2025
d9a90b6
Add mutation id to Relation data types and make accessible from compi…
ddelnano Feb 14, 2025
ebca756
Add context field to Operator plan proto used for determining if stre…
ddelnano Feb 17, 2025
0b569d3
Add pxl support to populate the mutation_id context field in the plan…
ddelnano Feb 18, 2025
38d957f
Ensure gazelle knows about the file source carnot protos. Linting fixes
ddelnano Feb 18, 2025
e56c69a
Add DataFrame argument for opting into stream results for non mutatio…
ddelnano Feb 18, 2025
74ed8e5
Fix PEM segfault issue on sink_results insert. Need to figure out 'co…
ddelnano Feb 19, 2025
61f5423
Add time_ column to sink_results table. This fixed the horrible query…
ddelnano Feb 20, 2025
52eb376
Fix issue querying kelvin sink_results table
ddelnano Feb 24, 2025
e4c071c
Add upid column to sink_results table
ddelnano Feb 25, 2025
43d67d4
Opt MemorySources into sink_results data collection when mutation pre…
ddelnano Feb 25, 2025
bb7866e
Rename file source connector test names, simplify file source connect…
ddelnano Feb 26, 2025
395a4cb
Opt pxlog module into triggering mutations from web UI
ddelnano Feb 26, 2025
f02544e
Trigger pxlog mutations during export scripts
ddelnano Feb 26, 2025
e0a924b
Fix issue with execution stats in MemorySources
ddelnano Feb 26, 2025
cd2ac1d
Changes to get first iteration of pipeline flow graph working
ddelnano Feb 27, 2025
1d92299
Remove fake OperatorType's from planpb. Move protocol table identific…
ddelnano Feb 27, 2025
d58a34f
Initial version of px/pipeline_flow_graph
ddelnano Feb 27, 2025
19b1af4
Fix issue where file rotation breaks file source connector
ddelnano Feb 28, 2025
a5afbee
Remove extraneous logs
ddelnano Feb 28, 2025
8314380
Revert unnecssary changes and add extra information to analyze log
ddelnano Feb 28, 2025
a77bb9d
Handle unstructured files in file source connector
ddelnano Mar 5, 2025
ac0d014
Implement exec node support for OTel log export
ddelnano Mar 6, 2025
9d5e616
Add log support to OTelExportSinkIR
ddelnano Mar 6, 2025
adc80bd
Fix misspelling
ddelnano Mar 6, 2025
9ec76ad
Add px.otel.log pxl module
ddelnano Mar 6, 2025
444b762
Ensure query broker CompileMutationRequests have otel and plugin conf…
ddelnano Mar 6, 2025
5a365ab
Update pipeline status to support bpftrace
ddelnano Apr 1, 2025
04cf9a6
Enhance FileSourceConnector to support unstructured files
ddelnano Apr 18, 2025
dd05ec8
Add tracepoint id string columen to GetTracepointStatus UDTF
ddelnano Apr 18, 2025
310eb74
Fix bug in HotOnlyTable store
ddelnano Apr 18, 2025
f834e8d
Update DEVELOPMENT.md
entlein Apr 27, 2025
e001302
Convert array and object JSON data values to strings so nested fields…
ddelnano Apr 29, 2025
601ce9f
build prep for local dev
mebegu Apr 29, 2025
8ca941b
make file source persistent table
mebegu Apr 29, 2025
958f5b6
Do not truncate string columns for file sources
ddelnano Apr 29, 2025
bb26cb6
Merge branch 'ddelnano/pipeline-support-wip' into pipeline-support-wip
mebegu Apr 29, 2025
0d1c0e0
Add uuid column to file source data tables
ddelnano Apr 30, 2025
9fad0d3
Merge branch 'ddelnano/pipeline-support-wip' into pipeline-support-wip
mebegu May 2, 2025
08c5c9e
Update DEVELOPMENT.md
entlein Apr 27, 2025
e40aed6
Update DEVELOPMENT.md
entlein Apr 27, 2025
17b9d5b
adding the build cache entry for bazel
entlein Apr 27, 2025
652265e
adding the compile mode into vizier skaffold
entlein Apr 27, 2025
ef467a8
these should be the most important steps
entlein Apr 27, 2025
6730d7e
more text
entlein Apr 27, 2025
1124c1f
comments unaligned fixed
entlein Apr 27, 2025
889de79
should be all now
entlein Apr 27, 2025
4036120
should be all now
entlein Apr 27, 2025
6ee0c1b
review development.md, and add extra comments
mebegu May 2, 2025
b01f03e
chore: cosmetic beautification
entlein May 3, 2025
eb762e3
chore: comment out the cache dir for bazel and explain how to use it …
entlein May 4, 2025
2e52fa0
Merge branch 'pixie-io:main' into main
entlein May 7, 2025
e9858af
PR resolve: seperating the 24.04 specifics from the overall description
entlein May 7, 2025
37ca63d
PR resolve: seperating the 24.04 specifics from the overall description
entlein May 7, 2025
aa134b9
PR resolve: referencing upstream doc for cli install and cleaning up …
entlein May 7, 2025
a6568b4
PR resolve: markdown numbering got confused
entlein May 7, 2025
859063a
feature: moving the minikube ubuntu dependencies into chef rather tha…
entlein May 7, 2025
1c1bb8c
Update DEVELOPMENT.md
entlein May 7, 2025
9f31d5f
Update DEVELOPMENT.md
entlein May 7, 2025
61ffd7e
Update DEVELOPMENT.md
entlein May 7, 2025
f7fab71
Fixing the numbering and removing empty quotes
entlein May 7, 2025
77d82f0
newline added
entlein May 7, 2025
840dc97
adding a file to document how an SRE would work with Pixie while the …
entlein May 7, 2025
46de0bd
I thought i already commited this
entlein May 7, 2025
3d95116
Fixed -R for recursive setgid bit
entlein May 7, 2025
085e27b
Adding the missing kernel header warning explanation for minikube
entlein May 7, 2025
c56fc28
test if this is a feasible approach to overwrite the skaffold produce…
entlein May 7, 2025
4a94d60
more replacements
entlein May 7, 2025
6b325cb
chore: amend text to resolve PR comments: highlight that minikube is …
entlein May 8, 2025
e078ae7
chore: revert skaffold_visizer but add comments
entlein May 8, 2025
94dd78b
chore: amend text to resolve PR comments: highlight that minikube is …
entlein May 8, 2025
29b5314
Update tools/chef/cookbooks/px_dev/recipes/linux.rb
entlein May 8, 2025
e3588fa
Update DEVELOPMENT.md
entlein May 8, 2025
c616e66
Update DEVELOPMENT.md
entlein May 8, 2025
c846a4d
chore: apply Dom s suggested edit
entlein May 8, 2025
11e0519
chore: adding the compilation mode explanation to the docs
entlein May 8, 2025
c124e0d
Remove trailing whitespace linux.rb
entlein May 8, 2025
a4674d7
Remove trailing whitespace from skaffold_vizier.yaml
entlein May 8, 2025
5b407ff
linting: removing lots of whitespaces at the EOL
entlein May 8, 2025
fd7d4ea
linting: file must end on a newline
entlein May 8, 2025
ef5e22e
Update .bazelrc
entlein May 9, 2025
5f7e4e7
Merge pull request #5 from k8sstormcenter/entlein-patch-2
entlein May 9, 2025
dfb198b
Merge pipeline-support-wip
ddelnano May 9, 2025
9ab6ecf
feature: testing dockerhub
entlein May 12, 2025
04b3e5f
feature: testing dockerhub
entlein May 12, 2025
726abbc
feature: testing dockerhub
entlein May 12, 2025
40e7191
feature: testing dockerhub
entlein May 12, 2025
90f1202
feature: testing dockerhub
entlein May 12, 2025
b750d89
feature: testing dockerhub
entlein May 12, 2025
75e1cf2
feature: testing dockerhub
entlein May 12, 2025
a52ae38
feature: testing dockerhub
entlein May 12, 2025
b4d31f9
feature: vizier persistent now working with our image
entlein May 12, 2025
3dfa7a6
feature: vizier persistent now working with our image
entlein May 12, 2025
cef03b8
feature: vizier etcd now working with our image
entlein May 12, 2025
0e4d43c
feature: vizier etcd now working with our image
entlein May 12, 2025
8a3d8e2
feature: vizier persistent ap now working with our image
entlein May 12, 2025
a54fd5e
feature: vizier persistent ap now working with our image
entlein May 12, 2025
698cb0d
feature: adding helm installer
entlein May 12, 2025
0d1ff45
merge
entlein May 12, 2025
0e2567a
Merge pull request #6 from k8sstormcenter/doc/usingpixiedev
entlein May 12, 2025
562d496
Merge pull request #8 from k8sstormcenter/pipeline-support-wip-merged
mebegu May 12, 2025
2cc76b1
Merge branch 'pipeline-support-wip'
mebegu May 12, 2025
e09ab3e
Merge pull request #3 from k8sstormcenter/pipeline-support-wip
mebegu May 12, 2025
145bcf5
docu: adding how to remove the hsts blocker for chrome and the syntax…
entlein Jun 5, 2025
bdf3e6f
Update README.md
entlein Jun 5, 2025
2b7963a
Merge branch 'main' of github.com:pixie-io/pixie
entlein Sep 10, 2025
20ad8d1
Update clickhouse to log to stdout. Stil having issues with test timeout
ddelnano Sep 6, 2025
3285d25
Initial start to working clickhouse_source_node. It queries all the d…
ddelnano Sep 7, 2025
6d6fe70
Add time based filtering and use batch size from plan node
ddelnano Sep 7, 2025
6bf5a41
Add partition column, get clickhouse_source_node_test working and fai…
ddelnano Oct 3, 2025
2fef3e8
Update carnot_executable to start a clickhouse container and make cha…
ddelnano Oct 3, 2025
7ded9f8
Get the carnot_executable version working e2e
ddelnano Oct 3, 2025
456219b
Add clickhouse config to compiler and logical state protos
ddelnano Oct 6, 2025
0ad9cea
Add clickhouse export support. IR and node tests passing
ddelnano Oct 6, 2025
5208617
Add px.CreateClickHouseSchemas UDTF, integrate in standalone_pem and …
ddelnano Oct 20, 2025
a5b02e3
First working version of px.CreateClickHouseSchemas UDTF
ddelnano Oct 21, 2025
14023b2
Add more loggig to row_batch
ddelnano Oct 21, 2025
831fe3f
Get carnot_executable working with unmodified stirling http_events table
ddelnano Oct 21, 2025
e508bdd
Remove upid (UINT128) and px_info column omission
ddelnano Oct 22, 2025
0bb09a2
Get all clickhouse tests passing and carnot_executable adhoc test
ddelnano Oct 22, 2025
d4fa954
Add use_if_not_exists for px.CreateClickHouseSchemas and test ClickHo…
ddelnano Oct 30, 2025
f1530c6
Merge branch 'ddelnano/add-clickhouse-source-node' into k8sstormcente…
ddelnano Oct 30, 2025
efcce8e
Remove hard coded clickhouse details in favor of DataFrame clickhouse…
ddelnano Oct 30, 2025
ed4472e
Add test to verify that ClickHouseSourceNodeIR correctly keeps non de…
ddelnano Nov 2, 2025
a674e1a
Fix bug where column indicies were mismatched with child Map operator…
ddelnano Nov 2, 2025
853684d
Fix clickhouse export sink bugs. Use compiler state for clickhouse dsn
ddelnano Nov 9, 2025
b791dc4
Use k8sstormceneter plugin repo
ddelnano Nov 9, 2025
c1c2558
Add adaptive export vizier service code. Verify it works adhoc
ddelnano Nov 9, 2025
6e1ee73
Ensure that '.beta' suffixed DataTables are handled properly since Cl…
ddelnano Nov 9, 2025
c7620c3
Get adaptive_export running on k8s without adaptivity piece
ddelnano Nov 9, 2025
c9bc972
Test adaptive_export end to end
ddelnano Nov 10, 2025
23da8a8
Add placeholder value for later sed
ddelnano Nov 10, 2025
2394108
Merge pull request #18 from ddelnano/k8sstormcenter/add-clickhouse-so…
entlein Nov 10, 2025
f277ede
docu: adding how to remove the hsts blocker for chrome and the syntax…
entlein Jun 5, 2025
3492af4
Update README.md
entlein Jun 5, 2025
4db5413
Merge branch 'main' of github.com:pixie-io/pixie
entlein Sep 10, 2025
6050e17
Merge branch 'ddelnano/add-clickhouse-source-node' into k8sstormcente…
ddelnano Oct 30, 2025
a4239f3
Remove hard coded clickhouse details in favor of DataFrame clickhouse…
ddelnano Oct 30, 2025
ae967ff
Add test to verify that ClickHouseSourceNodeIR correctly keeps non de…
ddelnano Nov 2, 2025
60edeeb
Fix bug where column indicies were mismatched with child Map operator…
ddelnano Nov 2, 2025
f35be2d
Fix clickhouse export sink bugs. Use compiler state for clickhouse dsn
ddelnano Nov 9, 2025
b326816
Use k8sstormceneter plugin repo
ddelnano Nov 9, 2025
b2a7190
Add adaptive export vizier service code. Verify it works adhoc
ddelnano Nov 9, 2025
b824922
Ensure that '.beta' suffixed DataTables are handled properly since Cl…
ddelnano Nov 9, 2025
3210633
Get adaptive_export running on k8s without adaptivity piece
ddelnano Nov 9, 2025
e39ef30
Test adaptive_export end to end
ddelnano Nov 10, 2025
0f066fd
Add placeholder value for later sed
ddelnano Nov 10, 2025
74bfa8f
Merge pull request #18 from ddelnano/k8sstormcenter/add-clickhouse-so…
entlein Nov 10, 2025
50f0cdc
Remove file source, sink results and other vestigial changes
ddelnano Feb 4, 2026
37cb31b
Use correct runner name and don't use remote exec
ddelnano Feb 4, 2026
e7dcabb
Use correct runners. Disable remote execution
ddelnano Feb 4, 2026
5ac34ea
Merge pull request #21 from k8sstormcenter/use-correct-runners
ddelnano Feb 4, 2026
928b51d
Fix clang tidy issues
ddelnano Feb 5, 2026
7be770c
Don't swallow helpful error context
ddelnano Feb 5, 2026
ac0860c
Fix magic_enum
ddelnano Feb 5, 2026
589b55b
Add extra debugging info
ddelnano Feb 5, 2026
9e217f9
Trigger
ddelnano Feb 5, 2026
f0f75f7
Fix openssl_trace_bpf_test and protobuf compilation issues and gcc bu…
ddelnano Feb 5, 2026
9014ccf
Fix another compilation issue
ddelnano Feb 6, 2026
10b2c89
Remove unnecssary debug logging
ddelnano Feb 6, 2026
a8ae46a
Try to get release builds working: replace pixie-io/pixie repo refere…
ddelnano Feb 7, 2026
2942e09
Use correct runners
ddelnano Feb 7, 2026
a40b233
Use correct runners
ddelnano Feb 7, 2026
f840fe4
docker login with buildbot token prior to image push
ddelnano Feb 7, 2026
832d293
Use packages:write token for docker login
ddelnano Feb 7, 2026
fc9190e
Handle the case where there isn't a preivous tag
ddelnano Feb 7, 2026
4d5a272
Add debug info for ory dockertest package connection issues
ddelnano Feb 7, 2026
29ae270
Fix linker error related to libstdc++.a
ddelnano Feb 7, 2026
f8ff1bc
Debugging info for matrix gen, remove clang-tidy temporarily
ddelnano Feb 7, 2026
1fb878a
Trigger
ddelnano Feb 7, 2026
6715e4e
Trigger again
ddelnano Feb 7, 2026
e6316cc
Add docker debugging info. Remove expensive clang-tidy build temporarily
ddelnano Feb 7, 2026
dbe93f0
Trigger again
ddelnano Feb 7, 2026
dbad698
Use newer rules_docker with puller fixes
ddelnano Feb 8, 2026
ea5e272
Add packages: writes permission for operator and cloud
ddelnano Feb 8, 2026
3af8369
Ensure that the pusher binary is not built for the target platform
ddelnano Feb 8, 2026
7b4206b
Log into ghcr.io for cloud and operator jobs
ddelnano Feb 8, 2026
69a14ac
Remove straggling gcs mirror reference
ddelnano Feb 8, 2026
aeaff86
Disable failing bpf tests
ddelnano Feb 8, 2026
602fdc5
Add x86_64_sysroot to cloud_build_release.sh
ddelnano Feb 8, 2026
f225e31
Replace docker client with crane to see if it fixes network issues
ddelnano Feb 9, 2026
9412950
Upgrade docker buildx to fix deprecated client version failure
ddelnano Feb 9, 2026
9693219
Upgrade docker buildx to fix deprecated client version failure
ddelnano Feb 9, 2026
01d9983
Only supply from index when previous image is available
ddelnano Feb 9, 2026
7627654
Reuse builder if it was created before
ddelnano Feb 9, 2026
f997a8f
Do not fail if license generation fails
ddelnano Feb 9, 2026
f2a4f70
Remove clang-tidy from job
ddelnano Feb 10, 2026
5f2dc72
Move check earlier to prevent bootstrapping problem
ddelnano Feb 10, 2026
b4ba215
Use correct worker
ddelnano Feb 11, 2026
37cf378
Use correct email for k8sstormcenter-buildbot user
ddelnano Feb 11, 2026
54d2a3a
Merge branch 'main' into ddelnano/remove-unnecessary-changes
ddelnano Feb 11, 2026
c6a6f1f
Merge pull request #23 from k8sstormcenter/ddelnano/remove-unnecessar…
ddelnano Feb 11, 2026
0e15ab9
Increase MaxWait to aid dockertest dockerd access
ddelnano Feb 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
# Global bazelrc file, see https://docs.bazel.build/versions/master/guide.html#bazelrc.

# Use local Cache directory if building on a VM:
# On Chef VM, create a directory and comment in the following line:
# build --disk_cache=</tmp/bazel/> # Optional for multi-user cache: Make this directory owned by a group name e.g. "bazelcache"

# Use strict action env to prevent leaks of env vars.
build --incompatible_strict_action_env

# Use cache
# build --disk_cache=/tmp/bazel/cache # must not be merged dev only settng

# Only pass through GH_API_KEY for stamped builds.
# This is still not ideal as it still busts the cache of stamped builds.
build:stamp --stamp
Expand Down
84 changes: 49 additions & 35 deletions .github/workflows/build_and_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,36 +35,36 @@ jobs:
with:
image-base-name: "dev_image_with_extras"
ref: ${{ needs.env-protect-setup.outputs.ref }}
clang-tidy:
runs-on: oracle-16cpu-64gb-x86-64
needs: [authorize, env-protect-setup, get-dev-image]
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
ref: ${{ needs.env-protect-setup.outputs.ref }}
- name: Add pwd to git safe dir
run: git config --global --add safe.directory `pwd`
- name: get bazel config
uses: ./.github/actions/bazelrc
with:
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- name: Save Diff Info
run: ./ci/save_diff_info.sh
- name: Run Clang Tidy
shell: bash
run: |
diff_file="diff_origin_main_cc"
if [[ "${{ github.event_name }}" == "push" ]] || [[ "${{ github.event_name }}" == "schedule" ]]; then
diff_file="diff_head_cc"
fi
./ci/run_clang_tidy.sh -f "${diff_file}"
# clang-tidy:
# runs-on: oracle-vm-16cpu-64gb-x86-64
# needs: [authorize, env-protect-setup, get-dev-image]
# container:
# image: ${{ needs.get-dev-image.outputs.image-with-tag }}
# steps:
# - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
# with:
# fetch-depth: 0
# ref: ${{ needs.env-protect-setup.outputs.ref }}
# - name: Add pwd to git safe dir
# run: git config --global --add safe.directory `pwd`
# - name: get bazel config
# uses: ./.github/actions/bazelrc
# with:
# BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
# - name: Save Diff Info
# run: ./ci/save_diff_info.sh
# - name: Run Clang Tidy
# shell: bash
# run: |
# diff_file="diff_origin_main_cc"
# if [[ "${{ github.event_name }}" == "push" ]] || [[ "${{ github.event_name }}" == "schedule" ]]; then
# diff_file="diff_head_cc"
# fi
# ./ci/run_clang_tidy.sh -f "${diff_file}"
code-coverage:
if: github.event_name == 'push'
needs: [authorize, env-protect-setup, get-dev-image]
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
steps:
Expand All @@ -85,10 +85,10 @@ jobs:
run: |
# Github actions container runner creates a docker network without IPv6 support. We enable it manually.
sysctl -w net.ipv6.conf.lo.disable_ipv6=0
./ci/collect_coverage.sh -u -b main -c "$(git rev-parse HEAD)" -r pixie-io/pixie
./ci/collect_coverage.sh -u -b main -c "$(git rev-parse HEAD)" -r k8sstormcenter/pixie
generate-matrix:
needs: [authorize, env-protect-setup, get-dev-image]
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
outputs:
Expand Down Expand Up @@ -120,7 +120,7 @@ jobs:
bazel_tests_*
build-and-test:
needs: [authorize, env-protect-setup, get-dev-image, generate-matrix]
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
permissions:
contents: read
actions: read
Expand Down Expand Up @@ -154,12 +154,27 @@ jobs:
${{ matrix.args }} \
--target_pattern_file=target_files/${{ matrix.buildables }} \
2> >(tee bazel_stderr)
- name: Debug Docker networking
if: ${{ matrix.tests }}
shell: bash
run: |
docker info
docker network inspect bridge
docker run --rm postgres:13.3 echo "pull works"
CID=$(docker run -d --rm -e POSTGRES_PASSWORD=secret postgres:13.3)
sleep 2
docker inspect "$CID" --format '{{json .NetworkSettings}}' | jq '{Gateway: .Gateway, IPAddress: .IPAddress, Bridge: .Bridge, Networks: .Networks}'
docker stop "$CID"
- name: Test ${{ matrix.name }}
if: ${{ matrix.tests }}
shell: bash
run: |
# Github actions container runner creates a docker network without IPv6 support. We enable it manually.
sysctl -w net.ipv6.conf.lo.disable_ipv6=0

# Our qemu builds require unprivileged user namespaces to run.
sysctl -w kernel.unprivileged_userns_clone=1
sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
./scripts/bazel_ignore_codes.sh test ${{ matrix.args }} --target_pattern_file=target_files/${{ matrix.tests }} \
2> >(tee bazel_stderr)
- name: Parse junit reports
Expand All @@ -181,19 +196,18 @@ jobs:
shell: bash
build-and-test-status:
if: always()
needs: [build-and-test, clang-tidy, generate-matrix]
needs: [build-and-test, generate-matrix]
runs-on: ubuntu-latest
steps:
- if: needs.build-and-test.result == 'success' && needs.clang-tidy.result == 'success'
- if: needs.build-and-test.result == 'success'
run: echo "Build and Test complete ✓"
- if: >
needs.generate-matrix.result == 'success' && needs.clang-tidy.result == 'success'
needs.generate-matrix.result == 'success'
&& needs.build-and-test.result == 'skipped'
run: echo "Build and Test skipped no matrix configs generated ✓"
- if: >
!(needs.build-and-test.result == 'success' && needs.clang-tidy.result == 'success') &&
!(needs.build-and-test.result == 'success') &&
!(needs.generate-matrix.result == 'success' &&
needs.clang-tidy.result == 'success' &&
needs.build-and-test.result == 'skipped')
run: |
echo "Build and Test failed"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cacher.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
with:
image-base-name: "dev_image"
populate-caches:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: get-dev-image
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
Expand Down
25 changes: 4 additions & 21 deletions .github/workflows/cli_release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
image-base-name: "dev_image_with_extras"
build-release:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: get-dev-image
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
Expand All @@ -42,18 +42,13 @@ jobs:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_NUMBER: ${{ github.run_attempt }}
JOB_NAME: ${{ github.job }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand All @@ -66,13 +61,6 @@ jobs:
with:
name: linux-artifacts
path: artifacts/
- name: Update GCS Manifest
env:
ARTIFACT_MANIFEST_BUCKET: "pixie-dev-public"
# Use the old style versions file instead of the new updates for the gcs manifest.
MANIFEST_UPDATES: ""
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
run: ./ci/update_artifact_manifest.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: artifact-upload-log
Expand Down Expand Up @@ -131,18 +119,13 @@ jobs:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Add pwd to git safe dir
run: |
git config --global --add safe.directory `pwd`
- name: Upload signed CLI
env:
REF: ${{ github.event.ref }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
ARTIFACT_UPLOAD_LOG: "artifact_uploads.json"
shell: bash
run: |
Expand Down Expand Up @@ -188,7 +171,7 @@ jobs:
--notes $'Pixie CLI Release:\n'"${changelog}"
gh release upload "${TAG_NAME}" linux-artifacts/* macos-artifacts/*
update-gh-artifacts-manifest:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: [get-dev-image, create-github-release]
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
Expand Down Expand Up @@ -217,8 +200,8 @@ jobs:
env:
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name 'k8sstormcenter-buildbot'
git config --global user.email 'info@fusioncore.ai'
git config --global user.signingkey "${BUILDBOT_GPG_KEY_ID}"
git config --global commit.gpgsign true
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/cloud_release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ jobs:
image-base-name: "dev_image_with_extras"
build-release:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
steps:
Expand All @@ -30,15 +33,13 @@ jobs:
with:
download_toplevel: 'true'
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Import GPG key
env:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- name: Login to GHCR
run: echo "${{ github.token }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
Expand All @@ -47,7 +48,6 @@ jobs:
GH_API_KEY: ${{ secrets.GITHUB_TOKEN }}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
shell: bash
run: |
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/codeql.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
with:
category: "/language:go"
analyze-python:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
permissions:
actions: read
contents: read
Expand All @@ -42,7 +42,7 @@ jobs:
with:
category: "/language:python"
analyze-javascript:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
permissions:
actions: read
contents: read
Expand Down
29 changes: 11 additions & 18 deletions .github/workflows/operator_release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ jobs:
image-base-name: "dev_image_with_extras"
build-release:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
env:
Expand All @@ -33,23 +36,20 @@ jobs:
with:
download_toplevel: 'true'
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Import GPG key
env:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- name: Login to GHCR
run: echo "${{ github.token }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
BUILD_NUMBER: ${{ github.run_attempt }}
JOB_NAME: ${{ github.job }}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
GH_REPO: ${{ github.repository }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
shell: bash
Expand All @@ -60,13 +60,6 @@ jobs:
mkdir -p "${ARTIFACTS_DIR}"
./ci/save_version_info.sh
./ci/operator_build_release.sh
- name: Update GCS Manifest
env:
ARTIFACT_MANIFEST_BUCKET: "pixie-dev-public"
# Use the old style versions file instead of the new updates for the gcs manifest.
MANIFEST_UPDATES: ""
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
run: ./ci/update_artifact_manifest.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: manifest-updates
Expand Down Expand Up @@ -127,8 +120,8 @@ jobs:
env:
GIT_SSH_COMMAND: "ssh -i /tmp/ssh.key"
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name 'k8sstormcenter-buildbot'
git config --global user.email 'info@fusioncore.ai'
- name: Push Helm YAML to gh-pages
shell: bash
env:
Expand All @@ -142,7 +135,7 @@ jobs:
git commit -s -m "Release Helm chart ${VERSION}"
git push origin "gh-pages"
update-gh-artifacts-manifest:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: [get-dev-image, create-github-release]
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
Expand Down Expand Up @@ -171,8 +164,8 @@ jobs:
env:
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name 'k8sstormcenter-buildbot'
git config --global user.email 'info@fusioncore.ai'
git config --global user.signingkey "${BUILDBOT_GPG_KEY_ID}"
git config --global commit.gpgsign true
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_genfiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
with:
image-base-name: "dev_image"
run-genfiles:
runs-on: oracle-8cpu-32gb-x86-64
runs-on: oracle-vm-16cpu-64gb-x86-64
needs: get-dev-image
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
Expand Down
Loading
Loading