Skip to content

Conversation

@climbfuji
Copy link
Collaborator

Handle and test xmllint versions that return a zero exit code even if validation fails

  1. In scripts/parse_tools/xml_tools.py, handle xmllint versions that return a zero exit code even if the validation fails. Simply logic, remove unused code, and update test/unit_tests/test_sdf.py accordingly
  2. Create test/unit_tests/xmllint_wrapper/xmllint wrapper that emulates the behavior of "bad" xmllint which returns a zero exit code even if the validation fails, and test in GitHub actions

Notes.

  1. This PR is based on code suggested by @gold2718 in a PR that got merged but later reverted due to problems with (unrelated) regular expression changes - see Add unit tests for SDF reading climbfuji/ccpp-framework#5. Credits to @gold2718 for this contribution.
  2. call_command was only used in one place (validate XML files) and contained overly complicated logic to deal with optional arguments that were unnecessary. With the need to parse stdout/stderr from the subprocess in the validate_xml function, there would have been even more code duplication and unnecessary logic.
  3. When Add capability to parse nested suites in capgen #691 was merged, xmllint became a hard requirement, therefore we can remove logic that allows silent failures of the XML validation step when xmllint cannot be found (silent failures in general are a bad idea imo).

User interface changes?: No

Fixes #700

Testing:
unit tests: pytest is now run in GitHub actions with both "good" and "bad" xmllint versions
manual testing: ran Python doctest on the updated xml_tools.py file, ran pytest locally on my laptop with "good" and "bad" xmllint

…behavior of bad xmllint which returns a zero exit code even if the validation fails
…urn a zero exit code even if the validation fails; simply logic and update test/unit_tests/test_sdf.py accordingly
res = validate_xml_file(file, 'suite', schema_version, logger)
if not res:
raise CCPPError(f"Invalid suite definition file, '{sdf}'")
raise CCPPError(f"Invalid suite definition file, '{file}'")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated bug fix uncovered while working on this PR

@climbfuji climbfuji marked this pull request as ready for review December 12, 2025 14:06
@climbfuji climbfuji self-assigned this Dec 12, 2025
@mkavulich
Copy link
Collaborator

@climbfuji I'm a bit confused about how this new test helps the issue. I thought your intention was to run a test on a known bad XML, and if that returns a zero exit code, use that to determine that the xmllint version is bad? With this test it seems like we're just creating a test that will never fail with a non-zero exit code, unless I'm misunderstanding something.

@climbfuji
Copy link
Collaborator Author

@climbfuji I'm a bit confused about how this new test helps the issue. I thought your intention was to run a test on a known bad XML, and if that returns a zero exit code, use that to determine that the xmllint version is bad? With this test it seems like we're just creating a test that will never fail with a non-zero exit code, unless I'm misunderstanding something.

No, the intention is to create a fake xmllint version that behaves like the "bad" version that @gold2718 somehow ended up with and then make sure that this version can also be used and produces the correct results. The fake xmllint version is the newly added xmllint wrapper.

Copy link
Collaborator

@peverwhee peverwhee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @climbfuji !

Copy link
Member

@dustinswales dustinswales left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look good to me. Thanks!

@mkavulich
Copy link
Collaborator

@climbfuji I assume we want to wait for @gold2718 to review this one before merging?

@climbfuji
Copy link
Collaborator Author

@climbfuji I assume we want to wait for @gold2718 to review this one before merging?

Yes, this is only five days old. Thanks for checking.

@climbfuji climbfuji requested review from a team as code owners December 17, 2025 21:23
Copy link
Collaborator

@gold2718 gold2718 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good (i.e., way better than my version), thanks!

@mkavulich mkavulich merged commit 6c661ac into NCAR:develop Dec 18, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle xmllint versions that return zero even if validation fails

5 participants