feat: Add suspend/resume evaluation sample with trajectory evaluators #397
+293
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
January 12, 2026 07:34
This commit adds a simple test agent and supporting infrastructure to test the suspend/resume pattern for RPA process invocations during evaluations. Changes: 1. Updated pyproject.toml to use local uipath-python editable install 2. Created test-suspend-resume-agent sample with: - Simple graph with one node that calls interrupt(InvokeProcess) - Proper checkpointer configuration for interrupt support - Evaluation set for testing suspend/resume flow - README with testing instructions The test agent demonstrates the minimal setup needed to test suspend/resume: - Uses LangGraph's interrupt() to suspend execution - Creates InvokeProcess request with proper structure - Returns state that preserves the __interrupt__ field This will be used to validate the changes in uipath-python that handle: - SUSPENDED status detection - Trigger extraction and pass-through - Resume functionality Testing: cd samples/test-suspend-resume-agent uv run --directory ../.. uipath eval graph evaluations/test_suspend_resume.json
Fixed the evaluation set to match the expected schema: - Added version: "1.0" and id field - Changed evaluationCriteria to evaluationCriterias (plural) - Added evaluatorRefs array - Initialized agent project with uipath init The evaluation setup is now complete and can run successfully. Agent properly suspends with interrupt(InvokeProcess) pattern.
Added detailed logging to clearly show: - When the agent node starts execution - When interrupt() is about to be called (suspend point) - When execution resumes after interrupt - Query and request details at each step This makes it much easier to understand and debug the suspend/resume pattern during development and testing.
Updated README to explain the enhanced logging system: - Shows example output at each stage (execution start, suspension, trigger detection, pass-through, resume) - Documents the logging symbols (🔴 for suspend, 🟢 for resume, ✅ for success) - Provides clear lifecycle explanation with log examples - Explains what happens at each step and what to look for This makes it easy for developers to understand what's happening during suspend/resume testing and troubleshooting.
Added three testing methods to demonstrate and verify suspend/resume:
1. **Python Test Script** (test_suspend_resume.py):
- Automated test showing complete cycle
- Uses LangGraph API directly
- Demonstrates checkpoint inspection
- Shows resume with Command()
- Verifies state transitions
2. **CLI Test Script** (test_cli_suspend_resume.sh):
- Tests using uipath CLI commands
- Captures output with triggers
- Extracts trigger information
- Shows all resume methods
3. **Comprehensive Guide** (SUSPEND_RESUME_GUIDE.md):
- Explains all testing methods
- Documents the suspend/resume flow
- Shows Orchestrator integration patterns
- Includes troubleshooting section
- Provides production deployment guidance
Test Output Example:
```
STEP 1: Initial execution - agent will SUSPEND
🔴 AGENT NODE: About to call interrupt() - SUSPENDING EXECUTION
✅ First execution result: {'__interrupt__': [...]}
STEP 2: Checking agent state after suspension
🔴 Found 1 pending task(s)
Interrupt value: InvokeProcess(name='TestProcess', ...)
STEP 3: RESUMING execution
🟢 Resume execution result: {'result': 'Process completed'}
✅ Test completed!
```
This makes it easy to understand and test the suspend/resume pattern
both locally and when integrated with UiPath Orchestrator.
Add comprehensive test-suspend-resume-agent sample demonstrating: - Agent suspend/resume pattern with interrupt() - InvokeProcess trigger creation for RPA orchestration - Evaluation framework integration with suspend detection - Trajectory and contains-based evaluators Includes: - graph.py: LangGraph agent with interrupt() for suspension - Evaluation sets for both suspend and completion testing - Two evaluators: trajectory (LLM-based) and contains (log-based) - Architecture diagram showing execution flow - Complete documentation and test scripts Evaluators run only on completed executions, not during suspension.
- Add type annotation for dict[str, Any] in InvokeProcess - Remove unsupported input/output parameters from StateGraph - Add type: ignore comments for LangGraph internal types
- Rename test-suspend-resume-agent -> tool-calling-suspend-resume - Remove unnecessary files: - AGENTS.md, CLAUDE.md (duplicates from .agent/) - SUSPEND_RESUME_GUIDE.md (excessive documentation) - test scripts (test_suspend_resume.py, test_cli_suspend_resume.sh) - architecture.drawio, agent.mermaid (diagrams not needed) - langgraph.json, uv.lock (local dev files) - Update README.md to be concise and focused - Update pyproject.toml name and description - Update graph.py docstring
Chibionos
force-pushed
the
feat/test-suspend-resume-evals
branch
from
bc752e9 to
cf5c5b9
Compare
added 7 commits
January 12, 2026 07:36
- Import InvokeProcess from uipath.platform.common instead of custom implementation - Capture return value from interrupt() to get actual process output - Use process output in result instead of hardcoded string - Add logging for process output upon resumption - Follow established pattern from multi-agent samples
…mple - Remove duplicate .agent/ reference documentation - Remove empty bindings.json configuration - Remove auto-generated entry-points.json - Remove redundant test_with_evaluators.json eval set - Keep only essential files: graph.py, pyproject.toml, uipath.json, README.md, and evaluations/
- Add langgraph.json to define graph entrypoint - Configure uipath.json with runtime options - Add test_simple.json for basic suspend testing without auth - Enable sample to run evaluations properly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds comprehensive test-suspend-resume-agent sample demonstrating agent suspend/resume pattern with evaluation framework integration.
Changes
interrupt()for RPA process invocationKey Features
✅ Agent suspends via
interrupt()with InvokeProcess payload✅ Runtime detects SUSPENDED status and extracts triggers
✅ Evaluators run only on completed executions
✅ Complete documentation and test scripts
Architecture
See
architecture.drawiofor detailed component relationships and execution flow.Testing