feat: Add script for adding components from image packages#15753
feat: Add script for adding components from image packages#15753liunan-ms wants to merge 2 commits intotomls/base/mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new repo utility script to sync missing components from a Kiwi image .packages output into base/comps/components.toml, aiming to keep the component list complete and alphabetically ordered.
Changes:
- Introduces
scripts/add-missing-components-from-image.shto detect missing components from a.packagesfile. - Appends missing component stanzas to
base/comps/components.tomland re-sorts the file contents.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Get the list of missing components | ||
| MISSING_COMPONENTS=$( | ||
| for comp in $(dnf repoquery -q -y --srpm $(cat "$PACKAGES_FILE" | sed -e "s/|.*//") --queryformat '%{name}\n' | sort | uniq); do | ||
| azldev comp list "$comp" 2>&1 | ||
| done | grep "component not found" | sed -e "s/component not found: //" | sort | uniq |
There was a problem hiding this comment.
With set -euo pipefail, this missing-component pipeline will abort the whole script in common “no missing components” cases: (1) azldev comp list is likely to exit non-zero when a component is missing, and (2) grep exits 1 when it finds no matches. Both non-zero statuses will trip -e/pipefail before MISSING_COMPONENTS is checked. Consider explicitly tolerating these expected failures (e.g., azldev ... || true, grep ... || true) or, better, detect missing components via exit codes instead of grepping stderr text.
| # Get the list of missing components | |
| MISSING_COMPONENTS=$( | |
| for comp in $(dnf repoquery -q -y --srpm $(cat "$PACKAGES_FILE" | sed -e "s/|.*//") --queryformat '%{name}\n' | sort | uniq); do | |
| azldev comp list "$comp" 2>&1 | |
| done | grep "component not found" | sed -e "s/component not found: //" | sort | uniq | |
| # Get the list of missing components by checking azldev exit codes | |
| MISSING_COMPONENTS=$( | |
| for comp in $(dnf repoquery -q -y --srpm $(cat "$PACKAGES_FILE" | sed -e "s/|.*//") --queryformat '%{name}\n' | sort | uniq); do | |
| if ! azldev comp list "$comp" >/dev/null 2>&1; then | |
| echo "$comp" | |
| fi | |
| done | sort | uniq |
| # Check if component already exists in the file | ||
| if grep -q "^\[components\.$comp\]$" "$COMPONENTS_TOML"; then | ||
| echo "Skipping $comp - already exists in components.toml" | ||
| continue | ||
| fi | ||
|
|
||
| # Create the component entry | ||
| ENTRY="[components.$comp]" | ||
|
|
||
| # Find the correct alphabetical position and insert | ||
| # We'll append to the file and then sort it properly | ||
| echo "" >> "$COMPONENTS_TOML" | ||
| echo "$ENTRY" >> "$COMPONENTS_TOML" | ||
| echo "Added: $comp" |
There was a problem hiding this comment.
Component names that contain characters not valid in TOML bare keys (e.g. python3.14 or perl-Text-Tabs+Wrap) need quoting in table headers; base/comps/components.toml already uses [components.'python3.14']/[components.'perl-Text-Tabs+Wrap']. This script always writes [components.$comp] and also searches with a regex pattern that won’t match the quoted form (and treats . as a wildcard), so it can create invalid TOML and/or duplicates. Please add logic to emit the correct quoted header when needed and use fixed-string matching for existence checks (including the quoted form).
| for comp in $(dnf repoquery -q -y --srpm $(cat "$PACKAGES_FILE" | sed -e "s/|.*//") --queryformat '%{name}\n' | sort | uniq); do | ||
| azldev comp list "$comp" 2>&1 |
There was a problem hiding this comment.
dnf repoquery ... $(cat "$PACKAGES_FILE" | sed ...) expands the entire packages list into the command line, which can hit ARG_MAX on larger images and is sensitive to word-splitting/globbing. Prefer feeding the package list via stdin/xargs (dnf repoquery ... --srpm --queryformat ... $(...) -> xargs -a/while read) and avoid the UUOC (sed ... "$PACKAGES_FILE").
| # Rewrite the file | ||
| { | ||
| echo "$HEADER" | ||
| echo "$COMPONENTS" | ||
| } > "$COMPONENTS_TOML" |
There was a problem hiding this comment.
Rewriting components.toml by redirecting directly to the same path (} > "$COMPONENTS_TOML") is not atomic; an interruption or error can leave the file truncated/corrupted. It’s safer to write to a temp file in the same directory and mv it into place after successful generation.
|
|
||
| # Add each missing component to components.toml | ||
| for comp in $MISSING_COMPONENTS; do | ||
| # Check if component already exists in the file |
There was a problem hiding this comment.
This won't be correct, as the components may be defined in different files; that's why my original command line I'd shared uses azldev comp list with -O json and a | jq ... pipe to properly query. That would also obviate the need for the more complex parsing below.
|
I think this script would probably be simpler if done in python instead of bash... |
|
can you rebase this into one commit with a commit message describing what/why/how? |
4 participants
This PR introduces a new utility script to automate the process of adding missing components from a kiwi image's
.packagesfile into thecomponents.tomlconfiguration. This streamlines the workflow for ensuring all required components are tracked and properly ordered.scripts/add-missing-components-from-image.shdetects missing components from a kiwi image's.packagesfile and inserts them intobase/comps/components.toml, maintaining alphabetical order.Usage:
./scripts/add-missing-components-from-image.sh [packages-file]