Fix ImportError when importing SUPERVISOR_COMMS outside task context#61630
Open
andreahlert wants to merge 2 commits intoapache:mainfrom
Open
Fix ImportError when importing SUPERVISOR_COMMS outside task context#61630andreahlert wants to merge 2 commits intoapache:mainfrom
andreahlert wants to merge 2 commits intoapache:mainfrom
Conversation
SUPERVISOR_COMMS was declared as a bare type annotation without assignment, which does not create an actual module attribute in Python. This caused ImportError when Variable.get() was called at the top level of a DAG file (outside task execution context), such as during dag.test(). Initialize SUPERVISOR_COMMS to None so the import always succeeds, add None guards before .send() calls in _set_variable, _delete_variable, and ExecutionAPISecretsBackend methods, and provide helpful error messages suggesting alternatives like environment variables or Jinja templates. Closes: apache#51816
Contributor
There was a problem hiding this comment.
This fix looks reasonable in isolation, but several provider tests appear to have relied on the previous permissive behavior when SUPERVISOR_COMMS was absent or not initialized. By normalizing it to None, execution-time paths (particularly around connection resolution) are now enforced and fail loudly, which is causing widespread provider CI failures. It seems those tests did not anticipate SUPERVISOR_COMMS being present but None and therefore entering these stricter code paths. It appears that provider tests might have to be loosened (or maybe connection resolution migh have to check for SUPERVISOR_COMMS) to accomodate this change but I am not going to make such a sweeping suggestion unilaterally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #51816
SUPERVISOR_COMMSwas declared as a bare type annotation (SUPERVISOR_COMMS: CommsDecoder[...]) intask_runner.py, which does not create an actual module attribute in Python. This causedImportErrorwhenVariable.get()was called at the top level of a DAG file (outside task execution context), such as duringdag.test().Changes
SUPERVISOR_COMMStoNoneso the import always succeeds. The existing check inensure_secrets_backend_loaded()already handlesis not None, so the fallback logic is unaffected.Noneguards before.send()calls in_set_variable(),_delete_variable(), and allExecutionAPISecretsBackendmethods to preventAttributeErroronNone.Variable.get()fails outside task context, suggesting alternatives (environment variables, Jinja templates, moving the call inside a task).Variable.get/set/deleteoutside task execution context.Files changed
task_runner.pySUPERVISOR_COMMS = None(was bare annotation)context.pyexecution_api.pyreturn Noneguardstest_variables.pyTest plan
test_variables.pytests pass (no regression)TestVariableOutsideTaskContexttests passVariable.get()with env var works outside task contextVariable.get()without env var raises helpful error outside task contextVariable.set()raises clear error outside task contextVariable.delete()raises clear error outside task context