Add Pause and Paused states

[tarun-google]

Add Pause and Paused states.

Note: we have to update the documentation once feature is released completely https://docs.cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.JobState

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @tarun-google, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the Apache Beam Python SDK by introducing new PAUSING and PAUSED job states for Dataflow jobs. This change allows the Dataflow runner to correctly interpret and react to these new states, providing more detailed lifecycle management for pipelines. The update includes modifications to the core PipelineState enum, the Dataflow runner's state handling, and the underlying Dataflow API message definitions, along with corresponding unit tests to ensure proper functionality.

Highlights

• New Pipeline States: Introduced PAUSING and PAUSED states to the Beam Python SDK's PipelineState enum, enabling more granular control over job lifecycle.
• Dataflow Runner Integration: Integrated the new JOB_STATE_PAUSING and JOB_STATE_PAUSED from the Dataflow API into the DataflowRunner's job state monitoring and mapping logic.
• API Message Definitions: Updated the Dataflow API client messages to include the definitions and documentation for JOB_STATE_PAUSING and JOB_STATE_PAUSED.
• Unit Testing: Added comprehensive unit tests to verify the correct handling and mapping of the new PAUSING and PAUSED job states within the Dataflow runner.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
  • Updated job state checking to include 'JOB_STATE_PAUSED' and 'JOB_STATE_PAUSING' as non-terminal states.
  • Mapped new Dataflow API states 'JOB_STATE_PAUSING' and 'JOB_STATE_PAUSED' to Beam's 'PipelineState' enum.
• sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
  • Added new test cases for 'wait_until_finish' to verify handling of 'PAUSING' and 'PAUSED' job states.
  • Extended 'test_api_jobstate_to_pipeline_state' to assert correct mapping for 'PAUSING' and 'PAUSED' states.
• sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
  • Defined 'JOB_STATE_PAUSING' and 'JOB_STATE_PAUSED' enum values within 'ExecutionStageStateValueValuesEnum' and 'CurrentStateValueValuesEnum'.
  • Added documentation comments for the newly introduced job states.
• sdks/python/apache_beam/runners/runner.py
  • Introduced 'PAUSING' and 'PAUSED' states to the 'PipelineState' enum.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[tarun-google]

Run Python_Transforms PreCommit 3.10

[github-actions]

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[claudevdm]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for PAUSE and PAUSED job states in the Dataflow runner. The changes correctly update the state polling logic, state mappings, and the PipelineState enum. New tests are included to cover these additions. The changes are well-implemented, but I've identified an opportunity to refactor some duplicated test code to improve maintainability.

[claudevdm]

Were these changes auto-generated by

beam/sdks/python/apache_beam/runners/dataflow/internal/clients/README.txt

Line 4 in cea7923

gen_client \

?

[tarun-google]

is this agreeable? script did this

[tarun-google]

I do not think storing the secret and id here is required. when we run from local it gets gcloud creds. If you think other users might need it we can manually fix it.

[claudevdm]

@Abacn can you please also take a look? I'm not familiar with these autogenerated files but I assume we just trust what is generated?

[Abacn]

Yeah, Beam Python SDK vendors a Dataflow v1beta3 client generated by (long-deprecated) Google apitools. The tooling may have some weird behaviors.

[tarun-google]

Run Python PreCommit 3.10