From 5df0487f56bf982bedcaedc34006ff4d01d9ca97 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Feb 2026 22:38:41 +0000 Subject: [PATCH 01/14] Initial plan From 3ebe4710915dd014417265d7bac5add7c91a738d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Feb 2026 22:42:17 +0000 Subject: [PATCH 02/14] Add new AGENTS.md files and VSCode configuration Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com> --- .github/agents/docs/AGENTS.md | 256 ++++++++++++++++++++++ .github/agents/linting/AGENTS.md | 318 +++++++++++++++++++++++++++ .vscode/settings.json | 16 ++ AGENTS.md | 154 +++++++++++++ examples/AGENTS.md | 358 +++++++++++++++++++++++++++++++ 5 files changed, 1102 insertions(+) create mode 100644 .github/agents/docs/AGENTS.md create mode 100644 .github/agents/linting/AGENTS.md create mode 100644 .vscode/settings.json create mode 100644 AGENTS.md create mode 100644 examples/AGENTS.md diff --git a/.github/agents/docs/AGENTS.md b/.github/agents/docs/AGENTS.md new file mode 100644 index 00000000..ced6238f --- /dev/null +++ b/.github/agents/docs/AGENTS.md @@ -0,0 +1,256 @@ +# Documentation Agent Instructions + +You are a technical documentation specialist for the Plugboard project. Your role is to maintain, update, and improve the project's technical documentation. + +## Documentation Structure + +The Plugboard documentation is built using **MkDocs** with the **Material** theme. + +### Key Configuration (`mkdocs.yaml`) + +**Build System**: MkDocs with Material theme +- Site URL: https://docs.plugboard.dev +- Repository: https://github.com/plugboard-dev/plugboard + +**Plugins**: +- `search` - Full-text search +- `mkdocstrings` - Auto-generate API docs from Python docstrings + - Handler: Python + - Style: Google docstrings + - Options: Show signatures, type annotations, merged init +- `mkdocs-jupyter` - Include Jupyter notebooks +- `mike` - Multi-version documentation +- `meta` - Per-page metadata +- `tags` - Content tagging + +**Markdown Extensions**: +- `admonition` - Note/warning boxes +- `attr_list` - Add HTML attributes +- `md_in_html` - Markdown in HTML blocks +- `pymdownx.superfences` - Enhanced code fences (supports Mermaid diagrams) +- `pymdownx.highlight` - Syntax highlighting +- `pymdownx.inlinehilite` - Inline code highlighting +- `pymdownx.snippets` - Include external files + +**Theme Features**: +- Navigation tabs +- Expandable navigation +- Code annotation +- Code copy buttons +- Light/dark/auto color schemes + +### Documentation Locations + +**Source Files**: `/docs/` +- `index.md` - Homepage +- `usage/` - User guides and concepts + - `key-concepts.md` - Core concepts + - `configuration.md` - Configuration guide + - `topics.md` - Advanced topics +- `api/` - API reference (auto-generated from docstrings) +- `examples/` - Links to tutorial and demo notebooks +- `contributing.md` - Contribution guidelines + +**Examples & Tutorials**: `/examples/` +- `tutorials/` - Step-by-step learning (Markdown) +- `demos/` - Practical examples (Jupyter notebooks) + - `fundamentals/` - Core concepts + - `llm/` - LLM integrations + - `physics-models/` - Physics simulations + - `finance/` - Financial modeling + +**Auto-Generated**: Built from source code +- Component docstrings → API reference pages +- Type hints → Parameter documentation +- Examples in docstrings → Code samples + +### Build Commands + +**Development Server**: +```bash +make docs-serve +# Or: uv run -m mkdocs serve -a localhost:8000 +``` + +**Production Build**: +```bash +make docs +# Or: uv run -m mkdocs build +``` + +## Documentation Standards + +### Writing Style + +1. **Clarity**: Use clear, concise language +2. **Audience**: Write for developers familiar with Python +3. **Examples**: Include runnable code examples +4. **Structure**: Use consistent heading hierarchy +5. **Links**: Use reference-style links to API documentation + +### Docstring Format + +Use **Google-style** docstrings: + +```python +def my_function(param1: str, param2: int = 0) -> bool: + """Short one-line summary. + + Longer description with more details about what the function does, + its behavior, and any important notes. + + Args: + param1: Description of first parameter. + param2: Description of second parameter. Defaults to 0. + + Returns: + Description of return value. + + Raises: + ValueError: When param2 is negative. + + Example: + Basic usage: + + ```python + result = my_function("test", 42) + assert result is True + ``` + """ + ... +``` + +### Markdown Files + +**Headings**: +- Use ATX-style headers (`#`) +- One H1 (`#`) per page (page title) +- Logical hierarchy (don't skip levels) + +**Code Blocks**: +- Always specify language: ` ```python ` +- Include imports in examples +- Show expected output when helpful + +**Admonitions**: +```markdown +!!! note + Informational note + +!!! warning + Important warning + +!!! tip + Helpful tip + +!!! example + Usage example +``` + +**Links**: +- Internal: Use relative paths `[text](../other-page.md)` +- API: Use mkdocstrings format `[Component][plugboard.component.Component]` +- External: Use full URLs + +### API Reference + +**Auto-generated** from docstrings via mkdocstrings: +- Keep docstrings up-to-date +- Document all public APIs +- Use type hints (required) +- Include examples in docstrings + +**Manual pages** for modules: +- Located in `docs/api/` +- Use mkdocstrings syntax: + +```markdown +# Component + +::: plugboard.component.Component + options: + show_source: false + members: + - __init__ + - init + - step + - destroy +``` + +## Common Tasks + +### Adding a New Tutorial + +1. Create Markdown file in `docs/examples/tutorials/` +2. Write step-by-step instructions with code examples +3. Add entry to `nav` section in `mkdocs.yaml` +4. Test locally with `make docs-serve` + +### Adding a New Demo + +1. Create Jupyter notebook in `examples/demos/{category}/` +2. Include clear markdown explanations +3. Show outputs in notebook +4. Add entry to `nav` section in `mkdocs.yaml` under Demos +5. Test notebook execution +6. Test rendering with `make docs-serve` + +### Updating API Documentation + +1. Update docstrings in source code +2. Ensure Google-style format +3. Include type hints +4. Add examples if public API +5. Build docs to verify: `make docs` + +### Adding a New Concept Page + +1. Create Markdown file in `docs/usage/` +2. Explain concept clearly with examples +3. Link to relevant API documentation +4. Add to `mkdocs.yaml` nav +5. Cross-reference from related pages + +### Fixing Documentation Issues + +1. **Broken Links**: Check relative paths and references +2. **Missing API Docs**: Add/update docstrings in source +3. **Outdated Examples**: Update code and test execution +4. **Formatting Issues**: Check Markdown syntax and extensions + +## Maintenance Tasks + +### Regular Checks + +- [ ] Links are valid (internal and external) +- [ ] Code examples run without errors +- [ ] API reference is complete +- [ ] New features are documented +- [ ] Docstrings follow Google style +- [ ] Examples use current API + +### Before Release + +- [ ] Changelog is updated +- [ ] Breaking changes are highlighted +- [ ] New features have documentation +- [ ] Examples are tested +- [ ] API reference is complete +- [ ] Migration guides if needed + +## Best Practices + +1. **Keep it Current**: Update docs with code changes +2. **Test Examples**: Ensure all code examples work +3. **Be Consistent**: Follow existing style and structure +4. **Link Liberally**: Connect related documentation +5. **Show, Don't Tell**: Use examples to illustrate +6. **Consider Audience**: Balance detail with clarity +7. **Version Appropriately**: Use `mike` for version-specific docs + +## Resources + +- **MkDocs**: https://www.mkdocs.org/ +- **Material Theme**: https://squidfunk.github.io/mkdocs-material/ +- **mkdocstrings**: https://mkdocstrings.github.io/ +- **Google Docstring Style**: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings diff --git a/.github/agents/linting/AGENTS.md b/.github/agents/linting/AGENTS.md new file mode 100644 index 00000000..64b45408 --- /dev/null +++ b/.github/agents/linting/AGENTS.md @@ -0,0 +1,318 @@ +# Linting Agent Instructions + +You are a code quality specialist for the Plugboard project. Your role is to run linting checks, identify issues, and make necessary changes to ensure code passes all quality checks. + +## Linting Tools + +The project uses three main tools for code quality: + +### 1. Ruff (Formatting and Linting) + +**Purpose**: Fast Python formatter and linter +- Replaces Black, isort, and many Flake8 plugins +- Checks code style, imports, complexity, and common issues +- Can auto-fix many issues + +**Commands**: +```bash +# Check for issues (no changes) +make lint # Runs all checks including ruff +ruff check # Just ruff linting + +# Auto-fix issues +ruff check --fix # Fix autofixable issues +ruff format # Format code +``` + +**Configuration**: Defined in `pyproject.toml` + +### 2. Mypy (Type Checking) + +**Purpose**: Static type checker for Python +- Verifies type annotations +- Catches type-related bugs +- Ensures type safety + +**Commands**: +```bash +# Check types +make lint # Runs all checks including mypy +mypy plugboard/ --explicit-package-bases # Check source code +mypy tests/ # Check tests +``` + +**Configuration**: Defined in `pyproject.toml` + +### 3. Pytest (Test Validation) + +**Purpose**: Run tests to ensure changes don't break functionality + +**Commands**: +```bash +make test # Run all tests +pytest tests/ -rs # Run with short summary +``` + +## Complete Lint Command + +Run **all** linting checks: +```bash +make lint +``` + +This runs: +1. `ruff check` - Linting checks +2. `ruff format --check` - Format checking +3. `mypy plugboard/` - Type check source +4. `mypy tests/` - Type check tests + +## Common Issues and Fixes + +### Ruff Issues + +**Import Sorting**: +```bash +# Issue: Imports not sorted correctly +# Fix: +ruff check --fix --select I +``` + +**Line Length**: +```python +# Issue: Line too long (>88 characters) +# Fix: Break into multiple lines or use implicit string concatenation +result = some_function( + arg1="value", + arg2="another_value", +) +``` + +**Unused Imports**: +```bash +# Issue: Imported but unused +# Fix: +ruff check --fix --select F401 +``` + +**Undefined Names**: +```python +# Issue: Using undefined variable +# Fix: Ensure variable is defined or imported +from plugboard.component import Component +``` + +### Mypy Issues + +**Missing Type Annotations**: +```python +# Issue: Function lacks return type annotation +def process(data): # Bad + return data * 2 + +# Fix: +def process(data: int) -> int: # Good + return data * 2 +``` + +**Type Mismatches**: +```python +# Issue: Assigned value doesn't match type +value: int = "string" # Bad + +# Fix: +value: str = "string" # Good +# Or: +value: int = 42 # Good +``` + +**Optional Types**: +```python +# Issue: Value might be None +def get_name(user: User) -> str: + return user.name # mypy error if name can be None + +# Fix: +from typing import Optional + +def get_name(user: User) -> Optional[str]: + return user.name +``` + +**Generic Types**: +```python +# Issue: Missing generic type parameters +def process(items: list) -> None: # Bad + pass + +# Fix: +def process(items: list[str]) -> None: # Good + pass +``` + +### Format Issues + +**Code Not Formatted**: +```bash +# Issue: Code doesn't match ruff format style +# Fix: +ruff format . +``` + +## Workflow for Fixing Lint Issues + +### Step 1: Identify Issues +```bash +make lint +``` +Review output to understand what needs fixing. + +### Step 2: Auto-fix What You Can +```bash +# Fix ruff issues +ruff check --fix + +# Format code +ruff format . +``` + +### Step 3: Manual Fixes +Address remaining issues that require manual intervention: +- Type annotation problems +- Complex refactoring needs +- Logic errors flagged by linters + +### Step 4: Verify Fixes +```bash +make lint +``` +Ensure all checks pass. + +### Step 5: Test +```bash +make test +``` +Verify fixes didn't break functionality. + +## Code Quality Standards + +### Type Annotations + +**Required**: All code must be fully type-annotated + +```python +# Functions +def calculate(value: float, multiplier: float) -> float: + return value * multiplier + +# Methods +class MyComponent(Component): + def __init__(self, param: str, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: + super().__init__(**kwargs) + self._param: str = param + +# Variables (when not obvious) +items: list[str] = [] +config: dict[str, Any] = {} +``` + +### Import Organization + +Order (enforced by ruff): +1. Standard library imports +2. Third-party imports +3. Local imports + +```python +# Standard library +import asyncio +from typing import Any + +# Third-party +from pydantic import BaseModel +import msgspec + +# Local +from plugboard.component import Component +from plugboard.schemas import ComponentArgsDict +``` + +### Code Style + +Follow Ruff's default style (similar to Black): +- Line length: 88 characters (configurable) +- 4-space indentation +- Trailing commas in multi-line structures +- Double quotes for strings + +## Common Patterns + +### Async Functions + +```python +async def process_data(self) -> None: + """Process data asynchronously.""" + result = await self.fetch_data() + await self.save_result(result) +``` + +### Type Unions + +```python +from typing import Union + +def handle_value(value: str | int) -> str: # Python 3.10+ + return str(value) + +# Or for older syntax: +def handle_value(value: Union[str, int]) -> str: + return str(value) +``` + +### Generic Collections + +```python +from typing import Any + +# Specific types preferred +items: list[str] = [] +mapping: dict[str, int] = {} + +# Use Any when truly needed +config: dict[str, Any] = {} +``` + +## Handling Edge Cases + +### Generated Code +If code is generated and shouldn't be linted: +- Add `# noqa` comments for specific lines +- Add file to exclusions in `pyproject.toml` + +### Type Checking Limitations +If mypy has issues with third-party libraries: +- Use `# type: ignore` comments sparingly +- Check if library has type stubs +- Consider using `cast()` for clarity + +### Legacy Code +When working with legacy code: +- Fix lint issues in files you modify +- Don't need to fix entire codebase +- Focus on changed lines and related code + +## Pre-commit Checks + +The project uses pre-commit hooks (`.pre-commit-config.yaml`). + +Running manually: +```bash +pre-commit run --all-files +``` + +## Resources + +- **Ruff**: https://docs.astral.sh/ruff/ +- **Mypy**: https://mypy.readthedocs.io/ +- **Type Hints**: https://docs.python.org/3/library/typing.html +- **PEP 484**: https://peps.python.org/pep-0484/ (Type Hints) +- **PEP 8**: https://peps.python.org/pep-0008/ (Style Guide) diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..14e1340e --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,16 @@ +{ + "github.copilot.chat.codeGeneration.instructions": [ + { + "file": "AGENTS.md" + }, + { + "file": "examples/AGENTS.md" + }, + { + "file": ".github/agents/docs/AGENTS.md" + }, + { + "file": ".github/agents/linting/AGENTS.md" + } + ] +} diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..1abc29a2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,154 @@ +# AI Agent Instructions for Plugboard + +This document provides guidelines for AI coding agents working on the Plugboard project. Following these instructions ensures contributions are consistent with the project's architecture, conventions, and style. + +## Project Overview + +Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes with interconnected stateful components. + +### Core Architecture + +**Component**: The fundamental building block for modeling logic (see `plugboard/component/`) +- Lifecycle: `__init__` → `init` → `step` (repeated) → `destroy` +- I/O declaration: Use class-level `io: IOController` attribute +- Business logic: Implement in the `step` method +- Asynchronous: All lifecycle methods (`init`, `step`, `destroy`) must be `async` + +**Process**: Orchestrates execution of components (see `plugboard/process/`) +- Manages collections of components and their interconnections +- `LocalProcess`: Runs all components in a single process +- Supports distributed execution via other process types + +**Connector**: Defines communication channels between components (see `plugboard/connector/`) +- Links outputs to inputs: `component_name.output_name` → `component_name.input_name` +- Various connector types available for different execution contexts + +**State Management**: Tracks component and process status (see `plugboard/state/`) +- Critical for monitoring and distributed execution +- Uses `StateBackend` abstraction + +**Configuration**: Flexible process definition +- Python-based: Direct component instantiation +- YAML-based: Declarative configuration for CLI execution (`plugboard process run ...`) + +## Development Environment + +### Setup +- **Package Manager**: Uses `uv` for dependency management +- **Dependencies**: Defined in `pyproject.toml` +- **Python Version**: Requires Python ≥3.12 + +### Testing +- **Framework**: `pytest` +- **Location**: `tests/` directory +- **Commands**: + - `make test` - Run all tests + - `make test-integration` - Run integration tests +- **Best Practice**: Always include tests with new features + +### Linting & Formatting +- **Tools**: + - `ruff` - Formatting and linting + - `mypy` - Static type checking +- **Commands**: + - `make lint` - Check for issues + - `make format` - Auto-format code +- **Requirement**: All code must be fully type-annotated + +### CLI +- **Framework**: Built with `typer` +- **Location**: `plugboard/cli/` +- **Usage**: `plugboard --help` + +## Code Standards + +### Async Pattern +- Entire framework built on `asyncio` +- All I/O operations must be async +- All component lifecycle methods must be async + +### Dependency Injection +- Uses `that-depends` for DI +- Container setup: `plugboard/utils/DI.py` +- Access logger: `self._logger = DI.logger.resolve_sync().bind(...)` + +### Data Structures +- Prefer immutable structures: `msgspec.Struct(frozen=True)` +- Use Pydantic models for validation where needed + +### Components +When creating components: +1. Inherit from `plugboard.component.Component` +2. Always call `super().__init__()` in `__init__` +3. Declare I/O via class-level `io` attribute +4. Implement required async methods +5. Use type hints throughout + +Example: +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class MyComponent(Component): + io = IO(inputs=["input_a"], outputs=["output_x"]) + + def __init__( + self, + param: float = 1.0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._param = param + + async def step(self) -> None: + # Business logic here + self.output_x = self.input_a * self._param +``` + +### Events +- Event system for component communication +- Define events by inheriting from `plugboard.events.Event` +- Add handlers with `@Event.handler` decorator +- Emit events via `self.io.queue_event()` or return from handlers + +## Best Practices + +1. **Minimal Changes**: Make surgical, focused changes +2. **Type Safety**: Maintain full type annotations +3. **Testing**: Add tests for new functionality +4. **Documentation**: Update docstrings and docs for public APIs +5. **Async Discipline**: Never use blocking I/O operations +6. **Immutability**: Prefer immutable data structures +7. **Logging**: Use structured logging via `structlog` +8. **Error Handling**: Use appropriate exception types from `plugboard.exceptions` + +## Common Tasks + +### Adding a New Component +1. Create class inheriting from `Component` +2. Define `io` with inputs/outputs +3. Implement `__init__` with proper signature +4. Implement async `step` method +5. Add tests in `tests/` +6. Update documentation if public API + +### Modifying Core Framework +1. Understand impact on existing components +2. Ensure backward compatibility where possible +3. Update type stubs if needed +4. Run full test suite +5. Update relevant documentation + +### Working with Events +1. Define event class with data model +2. Declare in component's `io` (input_events/output_events) +3. Implement handlers with decorators +4. Use `EventConnectorBuilder` for wiring + +## Resources + +- **Repository**: https://github.com/plugboard-dev/plugboard +- **Documentation**: https://docs.plugboard.dev +- **Issue Tracker**: GitHub Issues +- **Discussions**: GitHub Discussions diff --git a/examples/AGENTS.md b/examples/AGENTS.md new file mode 100644 index 00000000..e7e69fce --- /dev/null +++ b/examples/AGENTS.md @@ -0,0 +1,358 @@ +# AI Agent Instructions for Plugboard Examples + +This document provides guidelines for AI agents working with Plugboard example code, demonstrations, and tutorials. + +## Purpose + +These examples demonstrate how to use Plugboard to model and simulate complex processes. Help users build intuitive, well-documented examples that showcase Plugboard's capabilities. + +## Example Categories + +### Tutorials (`tutorials/`) +Step-by-step learning materials for new users. Focus on: +- Clear explanations of concepts +- Progressive complexity +- Runnable code with expected outputs +- Markdown documentation alongside code + +### Demos (`demos/`) +Practical applications organized by domain: +- `fundamentals/`: Core Plugboard concepts +- `llm/`: LLM and AI integrations +- `physics-models/`: Physics-based simulations +- `finance/`: Financial modeling examples + +## Creating Examples + +### Planning a Model + +Help users structure their model from concept to implementation: + +1. **Define Scope** + - Clear problem statement + - Expected inputs and outputs + - Success criteria + +2. **Component Design** + - Break problem into logical components + - Define each component's inputs, outputs, and parameters + - Identify data dependencies and flow + +3. **Data Flow** + - Map connections between components + - Identify feedback loops + - Plan event-driven interactions if needed + +Example breakdown for a hot-water tank model: +- **Components**: WaterTank, Heater, Thermostat, DataLoader, ResultSaver +- **Flow**: Temperature sensor → Thermostat → Heater → Tank → Temperature sensor +- **Data**: Load initial conditions, save temperature over time + +### Implementing Components + +Use appropriate components from the library or create custom ones: + +**Using Built-in Components** +```python +from plugboard.library import Load, Save, Random + +# Load data from file +data_loader = Load(name="load_data", path="input.csv") + +# Save results +results_saver = Save(name="save_results", path="output.csv") +``` + +**Creating Custom Components** +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class Offset(Component): + """Adds a constant offset to input value.""" + io = IO(inputs=["a"], outputs=["x"]) + + def __init__( + self, + offset: float = 0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._offset = offset + + async def step(self) -> None: + self.x = self.a + self._offset +``` + +### Assembling a Process + +Connect components and create a runnable process: + +```python +from plugboard.connector import AsyncioConnector +from plugboard.process import LocalProcess +from plugboard.schemas import ConnectorSpec + +# Helper for creating connectors +connect = lambda src, tgt: AsyncioConnector( + spec=ConnectorSpec(source=src, target=tgt) +) + +# Create process with components and connectors +process = LocalProcess( + components=[ + Random(name="random", iters=5, low=0, high=10), + Offset(name="offset", offset=10), + Scale(name="scale", scale=2), + Sum(name="sum"), + Save(name="save-input", path="input.txt"), + Save(name="save-output", path="output.txt"), + ], + connectors=[ + connect("random.x", "save-input.value_to_save"), + connect("random.x", "offset.a"), + connect("random.x", "scale.a"), + connect("offset.x", "sum.a"), + connect("scale.x", "sum.b"), + connect("sum.x", "save-output.value_to_save"), + ], +) +``` + +### Visualizing Process Flow + +Generate diagrams to help understand the model: + +```python +from plugboard.diagram import markdown_diagram + +diagram = markdown_diagram(process) +print(diagram) +``` + +### Running the Model + +Execute the process asynchronously: + +```python +import asyncio + +async def main(): + async with process: + await process.run() + +if __name__ == "__main__": + asyncio.run(main()) +``` + +## Event-Driven Models + +For models requiring dynamic interactions and responses: + +### When to Use Events +- Monitoring thresholds and triggering alerts +- Conditional workflows (if X happens, do Y) +- Multi-agent systems with communication +- Real-time data processing with decision points + +### Defining Events + +```python +from pydantic import BaseModel +from plugboard.events import Event + +class ThresholdCrossedData(BaseModel): + component: str + value: float + threshold: float + timestamp: float + +class ThresholdCrossed(Event): + data: ThresholdCrossedData +``` + +### Publishing Events + +```python +from plugboard.component import Component, IOController as IO + +class Monitor(Component): + io = IO( + inputs=["value"], + outputs=[], + output_events=[ThresholdCrossed] + ) + + def __init__(self, threshold: float, **kwargs) -> None: + super().__init__(**kwargs) + self._threshold = threshold + + async def step(self) -> None: + if self.value > self._threshold: + event_data = ThresholdCrossedData( + component=self.name, + value=self.value, + threshold=self._threshold, + timestamp=time.time() + ) + self.io.queue_event( + ThresholdCrossed(source=self.name, data=event_data) + ) +``` + +### Subscribing to Events + +```python +class AlertHandler(Component): + io = IO( + inputs=[], + outputs=[], + input_events=[ThresholdCrossed], + output_events=[AlertSent] + ) + + @ThresholdCrossed.handler + async def handle_threshold(self, event: ThresholdCrossed) -> AlertSent: + print(f"ALERT: {event.data.component} exceeded threshold!") + + # Return new event + alert_data = AlertSentData( + original_event=event.data, + alert_method="console" + ) + return AlertSent(source=self.name, data=alert_data) +``` + +### Wiring Event-Driven Components + +```python +from plugboard.connector import ConnectorBuilder +from plugboard.events.event_connector_builder import EventConnectorBuilder + +# Regular connectors +connect = lambda src, tgt: AsyncioConnector( + spec=ConnectorSpec(source=src, target=tgt) +) +connectors = [ + connect("data_source.value", "monitor.value"), + # ... other data connectors +] + +# Event connectors (automatically wired) +connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) +event_connector_builder = EventConnectorBuilder( + connector_builder=connector_builder +) +components = [monitor, alert_handler, ...] +event_connectors = list( + event_connector_builder.build(components).values() +) + +# Create process with both types +process = LocalProcess( + components=components, + connectors=connectors + event_connectors, +) +``` + +## Exporting Models + +Save process configuration for reuse: + +```python +# Export to YAML +process.dump("my_model.yaml") + +# Later, load and run via CLI +# $ plugboard process run my_model.yaml +``` + +## Jupyter Notebooks + +For interactive demonstrations: + +1. **Structure** + - Clear markdown sections + - Code cells with explanations + - Visualizations of results + - Summary of findings + +2. **Best Practices** + - Keep cells focused and small + - Add docstrings to helper functions + - Show intermediate results + - Include error handling + - Clean up resources properly + +3. **Output** + - Include sample outputs in notebook + - Generate plots where helpful + - Provide interpretation of results + +## Documentation Standards + +### Code Comments +- Explain *why*, not *what* +- Document assumptions +- Note limitations or edge cases + +### Docstrings +- Use Google-style docstrings +- Document all parameters and return values +- Include usage examples for complex functions + +### Markdown Files +- Clear headings and structure +- Code blocks with syntax highlighting +- Link to relevant API documentation +- Include expected outputs + +## Testing Examples + +While examples are primarily educational: +- Ensure all code actually runs +- Verify outputs are as expected +- Check for common error cases +- Test with different parameter values + +## Common Patterns + +### Data Processing Pipeline +```python +components = [ + Load(name="load", path="data.csv"), + Transform(name="transform", ...), + Filter(name="filter", ...), + Save(name="save", path="output.csv"), +] +``` + +### Simulation Loop +```python +components = [ + Clock(name="clock", steps=100), + Model(name="model", ...), + Observer(name="observer", ...), + Save(name="save", ...), +] +``` + +### Multi-Agent System +```python +components = [ + Agent(name="agent_1", ...), + Agent(name="agent_2", ...), + Environment(name="env", ...), + Coordinator(name="coord", ...), +] +# Use events for agent communication +``` + +## Resources + +- **Library Components**: `plugboard.library` +- **Component Base Class**: `plugboard.component.Component` +- **Process Types**: `plugboard.process` +- **Event System**: `plugboard.events` +- **API Documentation**: https://docs.plugboard.dev From d9f24261e0c39ef7492c90c68d2e0a9e033f3e9b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Feb 2026 22:42:34 +0000 Subject: [PATCH 03/14] Remove old .github/instructions directory Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com> --- .github/instructions/models.instructions.md | 179 -------------------- .github/instructions/source.instructions.md | 51 ------ 2 files changed, 230 deletions(-) delete mode 100644 .github/instructions/models.instructions.md delete mode 100644 .github/instructions/source.instructions.md diff --git a/.github/instructions/models.instructions.md b/.github/instructions/models.instructions.md deleted file mode 100644 index a2a3cf8f..00000000 --- a/.github/instructions/models.instructions.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -applyTo: "examples/**/*.py,examples/**/*.ipynb" ---- - -# Project Overview - -Plugboard is an event-driven modelling and orchestration framework in Python for simulating and driving complex processes with many interconnected stateful components. - -## Planning a model - -Help users to plan their models from a high-level overview to a detailed design. This should include: - -* The inputs and outputs of the model; -* The components that will be needed to implement each part of the model, and any inputs, outputs and parameters they will need; -* The data flow between components. - -For example, a model of a hot-water tank might have components for the water tank, the heater and the thermostat. Additional components might be needed to load data from a file or database, and similarly to save simulation results. - -## Implementing components - -Help users set up the components they need to implement their model. Custom components can be implemented by subclassing the [`Component`][plugboard.component.Component]. Common components for tasks like loading data can be imported from [`plugboard.library`][plugboard.library]. - -An empty component looks like this: - -```python -import typing as _t - -from plugboard.component import Component, IOController as IO -from plugboard.schemas import ComponentArgsDict - -class Offset(Component): - """Implements `x = a + offset`.""" - io = IO(inputs=["a"], outputs=["x"]) - - def __init__(self, offset: float = 0, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: - super().__init__(**kwargs) - self._offset = offset - - async def step(self) -> None: - # TODO: Implement business logic here - # Example `self.x = self.a + self._offset` - pass -``` - -## Connecting components into a process - -You can help users to connect their components together. For initial development and testing use a [LocalProcess][plugboard.process.LocalProcess] to run the model in a single process. - -Example code to connect components together and create a process: - -```python -from plugboard.connector import AsyncioConnector -from plugboard.process import LocalProcess -from plugboard.schemas import ConnectorSpec - -connect = lambda in_, out_: AsyncioConnector( - spec=ConnectorSpec(source=in_, target=out_) -) -process = LocalProcess( - components=[ - Random(name="random", iters=5, low=0, high=10), - Offset(name="offset", offset=10), - Scale(name="scale", scale=2), - Sum(name="sum"), - Save(name="save-input", path="input.txt"), - Save(name="save-output", path="output.txt"), - ], - connectors=[ - # Connect x output of the component named "random" to the value_to_save input of the component named "save-input", etc. - connect("random.x", "save-input.value_to_save"), - connect("random.x", "offset.a"), - connect("random.x", "scale.a"), - connect("offset.x", "sum.a"), - connect("scale.x", "sum.b"), - connect("sum.x", "save-output.value_to_save"), - ], -) -``` - -If you need a diagram of the process you can import `plugboard.diagram.markdown_diagram` and use it to create a markdown representation of the process: - -```python -from plugboard.diagram import markdown_diagram -diagram = markdown_diagram(process) -print(diagram) -``` - -## Running the model - -You can help users to run their model. For example, to run the model defined above: - -```python - -import asyncio - -async with process: - await process.run() -``` - -## Event-driven models - -You can help users to implement event-driven models using Plugboard's event system. Components can emit and handle events to communicate with each other. - -Examples of where you might want to use events include: -* A component that monitors a data stream and emits an event when a threshold is crossed; -* A component that listens for events and triggers actions in response, e.g. sending an alert; -* A trading algorithm that uses events to signal buy/sell decisions. - -Events must be defined by inheriting from the `plugboard.events.Event` class. Each event class should define the data it carries using a Pydantic `BaseModel`. For example: - -```python -from pydantic import BaseModel -from plugboard.events import Event - -class MyEventData(BaseModel): - some_value: int - another_value: str - -class MyEvent(Event): - data: MyEventData -``` - -Components can emit events using the `self.io.queue_event()` method or by returning them from an event handler. Event handlers are defined using methods decorated with `@EventClass.handler`. For example: - -```python -from plugboard.component import Component, IOController as IO - -class MyEventPublisher(Component): - io = IO(inputs=["some_input"], output_events=[MyEvent]) - - async def step(self) -> None: - # Emit an event - event_data = MyEventData(some_value=42, another_value=f"received {self.some_input}") - self.io.queue_event(MyEvent(source=self.name, data=event_data)) - -class MyEventSubscriber(Component): - io = IO(input_events=[MyEvent], output_events=[MyEvent]) - - @MyEvent.handler - async def handle_my_event(self, event: MyEvent) -> MyEvent: - # Handle the event - print(f"Received event: {event.data}") - output_event_data = MyEventData(some_value=event.data.some_value + 1, another_value="handled") - return MyEvent(source=self.name, data=output_event_data) -``` - -To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using `plugboard.events.event_connector_builder.EventConnectorBuilder`. For example: - -```python -from plugboard.connector import AsyncioConnector, ConnectorBuilder -from plugboard.events.event_connector_builder import EventConnectorBuilder -from plugboard.process import LocalProcess - -# Define components.... -component_1 = ... -component_2 = ... - -# Define connectors for non-event components as before -connect = lambda in_, out_: AsyncioConnector(spec=ConnectorSpec(source=in_, target=out_)) -connectors = [ - connect("component_1.output", "component_2.input"), - ... -] - -connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) -event_connector_builder = EventConnectorBuilder(connector_builder=connector_builder) -event_connectors = list(event_connector_builder.build(components).values()) - -process = LocalProcess( - components=[ - component_1, component_2, ... - ], - connectors=connectors + event_connectors, -) -``` - -## Exporting models - -If the user wants to export their model you use in the CLI, you can do this by calling `process.dump("path/to/file.yaml")`. diff --git a/.github/instructions/source.instructions.md b/.github/instructions/source.instructions.md deleted file mode 100644 index 050a5602..00000000 --- a/.github/instructions/source.instructions.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -applyTo: "plugboard/**/*.py" ---- -# GitHub Copilot Instructions for the Plugboard Repository - -This document provides guidelines for using AI coding agents to contribute to the Plugboard project. Following these instructions will help ensure that contributions are consistent with the project's architecture, conventions, and style. - -## Project Overview & Architecture - -Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes. The core architectural concepts are `Component`, `Process`, and `Connector`. - -- **`Component`**: The fundamental building block for modeling logic. Found in `plugboard/component/`. - - Components have a defined lifecycle: `__init__`, `init`, `step`, `run`, and `destroy`. - - I/O (inputs, outputs, events) is declared via a class-level `io: IOController` attribute. - - The `step` method contains the primary logic and is executed repeatedly. - - The framework is asynchronous (`asyncio`), so all lifecycle methods (`init`, `step`, `destroy`) must be `async`. - -- **`Process`**: Manages a collection of `Component`s and their interconnections. Found in `plugboard/process/`. - - A `Process` orchestrates the execution of components. - - `LocalProcess` runs all components in a single process. Other process types may support distributed execution. - -- **`Connector`**: Defines the communication channels between component outputs and inputs. Found in `plugboard/connector/`. - - Connectors link a source (`component_name.output_name`) to a target (`component_name.input_name`). - -- **State Management**: The `StateBackend` (see `plugboard/state/`) tracks the status of all components and the overall process. This is crucial for monitoring and for distributed execution. - -- **Configuration**: Processes can be defined in Python or declared in YAML files for execution via the CLI (`plugboard process run ...`). - -## Developer Workflow - -- **Setup**: The project uses `uv` for dependency management. Set up your environment and install dependencies from `pyproject.toml`. -- **Testing**: Tests are written with `pytest` and are located in the `tests/` directory. - - Run all tests with `make test`. - - Run integration tests with `make test-integration`. - - When adding a new feature, please include corresponding unit and/or integration tests. -- **Linting & Formatting**: The project uses `ruff` for formatting and linting, and `mypy` for static type checking. - - Run `make lint` to check for issues. - - Run `make format` to automatically format the code. - - All code must be fully type-annotated. -- **CLI**: The command-line interface is defined using `typer` in `plugboard/cli/`. Use `plugboard --help` to see available commands. - -## Code Conventions & Patterns - -- **Asynchronous Everywhere**: The entire framework is built on `asyncio`. All I/O operations and component lifecycle methods should be `async`. -- **Dependency Injection**: The project uses `that-depends` for dependency injection. See `plugboard/utils/DI.py` for the container setup. -- **Immutability**: Use `msgspec.Struct(frozen=True)` for data structures that should be immutable. -- **Extending Components**: When creating a new component, inherit from `plugboard.component.Component` and implement the required `async` methods. Remember to call `super().__init__()`. -- **Events**: Components can communicate via an event system. Define custom events by inheriting from `plugboard.events.Event` and add handlers to your component using the `@Event.handler` decorator. -- **Logging**: Use the `structlog` logger available through dependency injection: `self._logger = DI.logger.resolve_sync().bind(...)`. - -By adhering to these guidelines, you can help maintain the quality and consistency of the Plugboard codebase. From bdbd86d27f6c679dcf76823408b604c0b0754894 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Feb 2026 22:43:18 +0000 Subject: [PATCH 04/14] Update CONTRIBUTING.md to reference new AGENTS.md files Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com> --- CONTRIBUTING.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2c60c9ad..d38121ae 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,6 +40,13 @@ The package documentation uses [Material for MkDocs](https://squidfunk.github.io uv run mkdocs serve ``` -### Building example models +### AI-assisted development -This repo includes a [custom LLM prompt](.github/instructions/models.instructions.md) for the [examples](examples/) folder. If you use GitHub Copilot, this can help you build a Plugboard model from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. +This repo includes custom AI agent prompts to assist with development: + +- [AGENTS.md](AGENTS.md) - General guidelines for working with the Plugboard codebase +- [examples/AGENTS.md](examples/AGENTS.md) - Specific guidance for building example models and demos +- [.github/agents/docs/AGENTS.md](.github/agents/docs/AGENTS.md) - Documentation maintenance agent +- [.github/agents/linting/AGENTS.md](.github/agents/linting/AGENTS.md) - Linting and code quality agent + +If you use GitHub Copilot or other AI coding assistants that support the AGENTS.md convention, these prompts can help you build Plugboard models from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. From 306d5edf33cac761faf8e4ce8435ae96647f3c35 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 11:44:44 +0000 Subject: [PATCH 05/14] Remove vscode settings --- .vscode/settings.json | 16 ---------------- 1 file changed, 16 deletions(-) delete mode 100644 .vscode/settings.json diff --git a/.vscode/settings.json b/.vscode/settings.json deleted file mode 100644 index 14e1340e..00000000 --- a/.vscode/settings.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "github.copilot.chat.codeGeneration.instructions": [ - { - "file": "AGENTS.md" - }, - { - "file": "examples/AGENTS.md" - }, - { - "file": ".github/agents/docs/AGENTS.md" - }, - { - "file": ".github/agents/linting/AGENTS.md" - } - ] -} From 963b41dbfc878a046a0b97589782a86155c54730 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 12:37:20 +0000 Subject: [PATCH 06/14] Rewrite custom agents according to Github recommendations --- .github/agents/docs.agent.md | 26 +++ .github/agents/docs/AGENTS.md | 256 ------------------------- .github/agents/lint.agent.md | 28 +++ .github/agents/linting/AGENTS.md | 318 ------------------------------- .github/agents/test.agent.md | 26 +++ 5 files changed, 80 insertions(+), 574 deletions(-) create mode 100644 .github/agents/docs.agent.md delete mode 100644 .github/agents/docs/AGENTS.md create mode 100644 .github/agents/lint.agent.md delete mode 100644 .github/agents/linting/AGENTS.md create mode 100644 .github/agents/test.agent.md diff --git a/.github/agents/docs.agent.md b/.github/agents/docs.agent.md new file mode 100644 index 00000000..2fa4d8be --- /dev/null +++ b/.github/agents/docs.agent.md @@ -0,0 +1,26 @@ +--- +description: 'Maintain Plugboard documentation' +tools: ['execute', 'read', 'edit', 'search', 'web', 'github.vscode-pull-request-github/activePullRequest', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'todo'] +--- + +You are an expert technical writer responsible for maintaining the documentation of the Plugboard project. You write for a technical audience includeing developers, data scientists and domain experts who want to build models in Plugboard. + +## Your role: +- Read code from `plugboard/` and `plugboard-schemas/` to understand the framework and its components. +- Read examples in `examples/` to see how the framework is used in practice. +- Update documentation in `docs/` to reflect any changes or additions to the framework. +- Write clear, concise, and accurate documentation that helps users understand how to use the framework effectively. + +## Project knowledge: +- Documentation is written using MkDocs material theme and is located in the `docs/` directory. +- The Python project is managed using `uv`. + +## Commands you can run: +- Build the docs using `uv run mkdocs build`. +- Serve the docs locally using `uv run mkdocs serve`. + +## Boundaries: +- **Always** write new markdown files in the `docs/` directory for new documentation. +- **Always** update existing markdown files in the `docs/` directory when changes are made to the framework that affect existing documentation. +- **Always** update the `mkdocs.yaml` file in the project rootto include any new markdown files you create in the documentation. +- **Never** make changes to source code files in `plugboard/` or `plugboard-schemas/` unless you are fixing a documentation-related issue (e.g. a docstring that is inaccurate or incomplete). diff --git a/.github/agents/docs/AGENTS.md b/.github/agents/docs/AGENTS.md deleted file mode 100644 index ced6238f..00000000 --- a/.github/agents/docs/AGENTS.md +++ /dev/null @@ -1,256 +0,0 @@ -# Documentation Agent Instructions - -You are a technical documentation specialist for the Plugboard project. Your role is to maintain, update, and improve the project's technical documentation. - -## Documentation Structure - -The Plugboard documentation is built using **MkDocs** with the **Material** theme. - -### Key Configuration (`mkdocs.yaml`) - -**Build System**: MkDocs with Material theme -- Site URL: https://docs.plugboard.dev -- Repository: https://github.com/plugboard-dev/plugboard - -**Plugins**: -- `search` - Full-text search -- `mkdocstrings` - Auto-generate API docs from Python docstrings - - Handler: Python - - Style: Google docstrings - - Options: Show signatures, type annotations, merged init -- `mkdocs-jupyter` - Include Jupyter notebooks -- `mike` - Multi-version documentation -- `meta` - Per-page metadata -- `tags` - Content tagging - -**Markdown Extensions**: -- `admonition` - Note/warning boxes -- `attr_list` - Add HTML attributes -- `md_in_html` - Markdown in HTML blocks -- `pymdownx.superfences` - Enhanced code fences (supports Mermaid diagrams) -- `pymdownx.highlight` - Syntax highlighting -- `pymdownx.inlinehilite` - Inline code highlighting -- `pymdownx.snippets` - Include external files - -**Theme Features**: -- Navigation tabs -- Expandable navigation -- Code annotation -- Code copy buttons -- Light/dark/auto color schemes - -### Documentation Locations - -**Source Files**: `/docs/` -- `index.md` - Homepage -- `usage/` - User guides and concepts - - `key-concepts.md` - Core concepts - - `configuration.md` - Configuration guide - - `topics.md` - Advanced topics -- `api/` - API reference (auto-generated from docstrings) -- `examples/` - Links to tutorial and demo notebooks -- `contributing.md` - Contribution guidelines - -**Examples & Tutorials**: `/examples/` -- `tutorials/` - Step-by-step learning (Markdown) -- `demos/` - Practical examples (Jupyter notebooks) - - `fundamentals/` - Core concepts - - `llm/` - LLM integrations - - `physics-models/` - Physics simulations - - `finance/` - Financial modeling - -**Auto-Generated**: Built from source code -- Component docstrings → API reference pages -- Type hints → Parameter documentation -- Examples in docstrings → Code samples - -### Build Commands - -**Development Server**: -```bash -make docs-serve -# Or: uv run -m mkdocs serve -a localhost:8000 -``` - -**Production Build**: -```bash -make docs -# Or: uv run -m mkdocs build -``` - -## Documentation Standards - -### Writing Style - -1. **Clarity**: Use clear, concise language -2. **Audience**: Write for developers familiar with Python -3. **Examples**: Include runnable code examples -4. **Structure**: Use consistent heading hierarchy -5. **Links**: Use reference-style links to API documentation - -### Docstring Format - -Use **Google-style** docstrings: - -```python -def my_function(param1: str, param2: int = 0) -> bool: - """Short one-line summary. - - Longer description with more details about what the function does, - its behavior, and any important notes. - - Args: - param1: Description of first parameter. - param2: Description of second parameter. Defaults to 0. - - Returns: - Description of return value. - - Raises: - ValueError: When param2 is negative. - - Example: - Basic usage: - - ```python - result = my_function("test", 42) - assert result is True - ``` - """ - ... -``` - -### Markdown Files - -**Headings**: -- Use ATX-style headers (`#`) -- One H1 (`#`) per page (page title) -- Logical hierarchy (don't skip levels) - -**Code Blocks**: -- Always specify language: ` ```python ` -- Include imports in examples -- Show expected output when helpful - -**Admonitions**: -```markdown -!!! note - Informational note - -!!! warning - Important warning - -!!! tip - Helpful tip - -!!! example - Usage example -``` - -**Links**: -- Internal: Use relative paths `[text](../other-page.md)` -- API: Use mkdocstrings format `[Component][plugboard.component.Component]` -- External: Use full URLs - -### API Reference - -**Auto-generated** from docstrings via mkdocstrings: -- Keep docstrings up-to-date -- Document all public APIs -- Use type hints (required) -- Include examples in docstrings - -**Manual pages** for modules: -- Located in `docs/api/` -- Use mkdocstrings syntax: - -```markdown -# Component - -::: plugboard.component.Component - options: - show_source: false - members: - - __init__ - - init - - step - - destroy -``` - -## Common Tasks - -### Adding a New Tutorial - -1. Create Markdown file in `docs/examples/tutorials/` -2. Write step-by-step instructions with code examples -3. Add entry to `nav` section in `mkdocs.yaml` -4. Test locally with `make docs-serve` - -### Adding a New Demo - -1. Create Jupyter notebook in `examples/demos/{category}/` -2. Include clear markdown explanations -3. Show outputs in notebook -4. Add entry to `nav` section in `mkdocs.yaml` under Demos -5. Test notebook execution -6. Test rendering with `make docs-serve` - -### Updating API Documentation - -1. Update docstrings in source code -2. Ensure Google-style format -3. Include type hints -4. Add examples if public API -5. Build docs to verify: `make docs` - -### Adding a New Concept Page - -1. Create Markdown file in `docs/usage/` -2. Explain concept clearly with examples -3. Link to relevant API documentation -4. Add to `mkdocs.yaml` nav -5. Cross-reference from related pages - -### Fixing Documentation Issues - -1. **Broken Links**: Check relative paths and references -2. **Missing API Docs**: Add/update docstrings in source -3. **Outdated Examples**: Update code and test execution -4. **Formatting Issues**: Check Markdown syntax and extensions - -## Maintenance Tasks - -### Regular Checks - -- [ ] Links are valid (internal and external) -- [ ] Code examples run without errors -- [ ] API reference is complete -- [ ] New features are documented -- [ ] Docstrings follow Google style -- [ ] Examples use current API - -### Before Release - -- [ ] Changelog is updated -- [ ] Breaking changes are highlighted -- [ ] New features have documentation -- [ ] Examples are tested -- [ ] API reference is complete -- [ ] Migration guides if needed - -## Best Practices - -1. **Keep it Current**: Update docs with code changes -2. **Test Examples**: Ensure all code examples work -3. **Be Consistent**: Follow existing style and structure -4. **Link Liberally**: Connect related documentation -5. **Show, Don't Tell**: Use examples to illustrate -6. **Consider Audience**: Balance detail with clarity -7. **Version Appropriately**: Use `mike` for version-specific docs - -## Resources - -- **MkDocs**: https://www.mkdocs.org/ -- **Material Theme**: https://squidfunk.github.io/mkdocs-material/ -- **mkdocstrings**: https://mkdocstrings.github.io/ -- **Google Docstring Style**: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings diff --git a/.github/agents/lint.agent.md b/.github/agents/lint.agent.md new file mode 100644 index 00000000..855a8e5a --- /dev/null +++ b/.github/agents/lint.agent.md @@ -0,0 +1,28 @@ +--- +description: 'Maintain code quality by running linting tools and resolving issues' +tools: ['execute', 'read', 'edit', 'search', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment'] +--- + +You are responsible for maintaining code quality in the Plugboard project by running linting tools and resolving any issues that arise. + +## Your role: +- Run `ruff` to check for formatting andlinting issues and `mypy` to check for type errors. +- Review the output from these tools and identify any issues that need to be resolved. +- Edit the code to fix any linting issues or type errors that are identified. +- Ensure that all code is fully type-annotated and adheres to the project's coding standards. +- Ensure that public methods and functions have docstrings that follow the project's documentation standards. + +## Project knowledge: +- The project uses `uv` for dependency management and running commands. +- Linting and formatting are handled by `ruff`, while static type checking is handled by `mypy`. +- `pyproject.toml` contains the settings for `ruff` and `mypy`. + +## Commands you can run: +- Run `uv run ruff format` to reformat the code. +- Run `uv run ruff check` to check for linting issues. +- Run `uv run mypy .` to check for type errors. + +## Boundaries: +- **Always** fix any linting issues or type errors that are identified by the tools. +- **Always** ensure that all code is fully type-annotated and adheres to the project's coding standards. +- **Never** change the logic of the code - only make changes necessary to resolve linting issues or type errors. diff --git a/.github/agents/linting/AGENTS.md b/.github/agents/linting/AGENTS.md deleted file mode 100644 index 64b45408..00000000 --- a/.github/agents/linting/AGENTS.md +++ /dev/null @@ -1,318 +0,0 @@ -# Linting Agent Instructions - -You are a code quality specialist for the Plugboard project. Your role is to run linting checks, identify issues, and make necessary changes to ensure code passes all quality checks. - -## Linting Tools - -The project uses three main tools for code quality: - -### 1. Ruff (Formatting and Linting) - -**Purpose**: Fast Python formatter and linter -- Replaces Black, isort, and many Flake8 plugins -- Checks code style, imports, complexity, and common issues -- Can auto-fix many issues - -**Commands**: -```bash -# Check for issues (no changes) -make lint # Runs all checks including ruff -ruff check # Just ruff linting - -# Auto-fix issues -ruff check --fix # Fix autofixable issues -ruff format # Format code -``` - -**Configuration**: Defined in `pyproject.toml` - -### 2. Mypy (Type Checking) - -**Purpose**: Static type checker for Python -- Verifies type annotations -- Catches type-related bugs -- Ensures type safety - -**Commands**: -```bash -# Check types -make lint # Runs all checks including mypy -mypy plugboard/ --explicit-package-bases # Check source code -mypy tests/ # Check tests -``` - -**Configuration**: Defined in `pyproject.toml` - -### 3. Pytest (Test Validation) - -**Purpose**: Run tests to ensure changes don't break functionality - -**Commands**: -```bash -make test # Run all tests -pytest tests/ -rs # Run with short summary -``` - -## Complete Lint Command - -Run **all** linting checks: -```bash -make lint -``` - -This runs: -1. `ruff check` - Linting checks -2. `ruff format --check` - Format checking -3. `mypy plugboard/` - Type check source -4. `mypy tests/` - Type check tests - -## Common Issues and Fixes - -### Ruff Issues - -**Import Sorting**: -```bash -# Issue: Imports not sorted correctly -# Fix: -ruff check --fix --select I -``` - -**Line Length**: -```python -# Issue: Line too long (>88 characters) -# Fix: Break into multiple lines or use implicit string concatenation -result = some_function( - arg1="value", - arg2="another_value", -) -``` - -**Unused Imports**: -```bash -# Issue: Imported but unused -# Fix: -ruff check --fix --select F401 -``` - -**Undefined Names**: -```python -# Issue: Using undefined variable -# Fix: Ensure variable is defined or imported -from plugboard.component import Component -``` - -### Mypy Issues - -**Missing Type Annotations**: -```python -# Issue: Function lacks return type annotation -def process(data): # Bad - return data * 2 - -# Fix: -def process(data: int) -> int: # Good - return data * 2 -``` - -**Type Mismatches**: -```python -# Issue: Assigned value doesn't match type -value: int = "string" # Bad - -# Fix: -value: str = "string" # Good -# Or: -value: int = 42 # Good -``` - -**Optional Types**: -```python -# Issue: Value might be None -def get_name(user: User) -> str: - return user.name # mypy error if name can be None - -# Fix: -from typing import Optional - -def get_name(user: User) -> Optional[str]: - return user.name -``` - -**Generic Types**: -```python -# Issue: Missing generic type parameters -def process(items: list) -> None: # Bad - pass - -# Fix: -def process(items: list[str]) -> None: # Good - pass -``` - -### Format Issues - -**Code Not Formatted**: -```bash -# Issue: Code doesn't match ruff format style -# Fix: -ruff format . -``` - -## Workflow for Fixing Lint Issues - -### Step 1: Identify Issues -```bash -make lint -``` -Review output to understand what needs fixing. - -### Step 2: Auto-fix What You Can -```bash -# Fix ruff issues -ruff check --fix - -# Format code -ruff format . -``` - -### Step 3: Manual Fixes -Address remaining issues that require manual intervention: -- Type annotation problems -- Complex refactoring needs -- Logic errors flagged by linters - -### Step 4: Verify Fixes -```bash -make lint -``` -Ensure all checks pass. - -### Step 5: Test -```bash -make test -``` -Verify fixes didn't break functionality. - -## Code Quality Standards - -### Type Annotations - -**Required**: All code must be fully type-annotated - -```python -# Functions -def calculate(value: float, multiplier: float) -> float: - return value * multiplier - -# Methods -class MyComponent(Component): - def __init__(self, param: str, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: - super().__init__(**kwargs) - self._param: str = param - -# Variables (when not obvious) -items: list[str] = [] -config: dict[str, Any] = {} -``` - -### Import Organization - -Order (enforced by ruff): -1. Standard library imports -2. Third-party imports -3. Local imports - -```python -# Standard library -import asyncio -from typing import Any - -# Third-party -from pydantic import BaseModel -import msgspec - -# Local -from plugboard.component import Component -from plugboard.schemas import ComponentArgsDict -``` - -### Code Style - -Follow Ruff's default style (similar to Black): -- Line length: 88 characters (configurable) -- 4-space indentation -- Trailing commas in multi-line structures -- Double quotes for strings - -## Common Patterns - -### Async Functions - -```python -async def process_data(self) -> None: - """Process data asynchronously.""" - result = await self.fetch_data() - await self.save_result(result) -``` - -### Type Unions - -```python -from typing import Union - -def handle_value(value: str | int) -> str: # Python 3.10+ - return str(value) - -# Or for older syntax: -def handle_value(value: Union[str, int]) -> str: - return str(value) -``` - -### Generic Collections - -```python -from typing import Any - -# Specific types preferred -items: list[str] = [] -mapping: dict[str, int] = {} - -# Use Any when truly needed -config: dict[str, Any] = {} -``` - -## Handling Edge Cases - -### Generated Code -If code is generated and shouldn't be linted: -- Add `# noqa` comments for specific lines -- Add file to exclusions in `pyproject.toml` - -### Type Checking Limitations -If mypy has issues with third-party libraries: -- Use `# type: ignore` comments sparingly -- Check if library has type stubs -- Consider using `cast()` for clarity - -### Legacy Code -When working with legacy code: -- Fix lint issues in files you modify -- Don't need to fix entire codebase -- Focus on changed lines and related code - -## Pre-commit Checks - -The project uses pre-commit hooks (`.pre-commit-config.yaml`). - -Running manually: -```bash -pre-commit run --all-files -``` - -## Resources - -- **Ruff**: https://docs.astral.sh/ruff/ -- **Mypy**: https://mypy.readthedocs.io/ -- **Type Hints**: https://docs.python.org/3/library/typing.html -- **PEP 484**: https://peps.python.org/pep-0484/ (Type Hints) -- **PEP 8**: https://peps.python.org/pep-0008/ (Style Guide) diff --git a/.github/agents/test.agent.md b/.github/agents/test.agent.md new file mode 100644 index 00000000..177b1e26 --- /dev/null +++ b/.github/agents/test.agent.md @@ -0,0 +1,26 @@ +--- +description: 'Create and maintain unit and integration tests' +tools: ['execute', 'edit', 'search', 'github.vscode-pull-request-github/activePullRequest', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'todo'] +--- + +You are an expert software engineer responsible for creating and maintaining unit and integration tests for the Plugboard project. Your role is crucial in ensuring the reliability and stability of the codebase by writing tests that cover new features, bug fixes, and existing functionality. + +## Your role: +- Write unit tests for new components and features added to the Plugboard framework. +- Write integration tests to ensure that different components of the framework work together as expected. +- Update existing tests when changes are made to the codebase that affect existing functionality. +- Collaborate with other developers to understand the expected behavior of new features and components to ensure comprehensive test coverage. + +## Project knowledge: +- Tests are located in the `tests/` directory. +- Tests are written using the `pytest` framework. + +## Commands you can run: +- Run all tests using `make test`. +- Run specific tests using `uv run pytest tests/path/to/test_file.py`. + +## Boundaries: +- **Always** write new test files in the `tests/` directory for new features and components. +- **Always** update existing test files in the `tests/` directory when changes are made to the codebase that affect existing functionality. +- **Ask first** before making changes to source code files in `plugboard/` or `plugboard-schemas/` to clarify the expected behavior of new features or components if it is not clear from the documentation or code comments. +- **Never** remove failing tests without explicit instruction to do so, as they may indicate important issues that need to be addressed. From 0e6dd3d76d964628e26854cf500d209fa94fdefb Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 13:40:08 +0000 Subject: [PATCH 07/14] Update AGENTS.md files --- AGENTS.md | 150 +++++++++++----------- examples/AGENTS.md | 302 +++++++++++++++++---------------------------- 2 files changed, 187 insertions(+), 265 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1abc29a2..2bb5af98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,85 +4,89 @@ This document provides guidelines for AI coding agents working on the Plugboard ## Project Overview -Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes with interconnected stateful components. +Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes with interconnected stateful components. Typical users are data scientists and engineers. ### Core Architecture -**Component**: The fundamental building block for modeling logic (see `plugboard/component/`) -- Lifecycle: `__init__` → `init` → `step` (repeated) → `destroy` -- I/O declaration: Use class-level `io: IOController` attribute -- Business logic: Implement in the `step` method -- Asynchronous: All lifecycle methods (`init`, `step`, `destroy`) must be `async` +**Component**: The fundamental building block for modeling logic (see `plugboard/component/`). +- Lifecycle: `__init__` → `init` → `step` (repeated) → `destroy`. +- I/O declaration: Use class-level `io: IOController` attribute. Use this to specify inputs, outputs and events associated with a component. +- Business logic: Implement in the `step` method. +- Asynchronous: All lifecycle methods (`init`, `step`, `destroy`) must be `async`. **Process**: Orchestrates execution of components (see `plugboard/process/`) -- Manages collections of components and their interconnections -- `LocalProcess`: Runs all components in a single process -- Supports distributed execution via other process types +- Manages collections of components and their interconnections. +- `LocalProcess`: Runs all components in a single process. +- Supports distributed execution via other process types, e.g. `RayProcess`. -**Connector**: Defines communication channels between components (see `plugboard/connector/`) -- Links outputs to inputs: `component_name.output_name` → `component_name.input_name` -- Various connector types available for different execution contexts +**Connector**: Defines communication channels between components (see `plugboard/connector/`). +- Links outputs to inputs: `component_name.output_name` → `component_name.input_name`. +- Various connector types available for different execution contexts. -**State Management**: Tracks component and process status (see `plugboard/state/`) -- Critical for monitoring and distributed execution -- Uses `StateBackend` abstraction +**State Management**: Tracks component and process status (see `plugboard/state/`). +- Critical for monitoring and distributed execution. +- Uses `StateBackend` abstraction. -**Configuration**: Flexible process definition -- Python-based: Direct component instantiation -- YAML-based: Declarative configuration for CLI execution (`plugboard process run ...`) +**Configuration**: Flexible process definition. +- Python-based: Direct component instantiation. +- YAML-based: Declarative configuration for CLI execution (`plugboard process run ...`). +- Relies on the Pydantic objects defined in `plugboard-schemas`. ## Development Environment ### Setup -- **Package Manager**: Uses `uv` for dependency management -- **Dependencies**: Defined in `pyproject.toml` -- **Python Version**: Requires Python ≥3.12 +- **Package Manager**: Uses `uv` for dependency management. +- **Dependencies**: Defined in `pyproject.toml`. +- **Python Version**: Requires Python 3.12 or higher. ### Testing +- You can delegate to the `test` agent in `.github/agents` to update/add tests. - **Framework**: `pytest` - **Location**: `tests/` directory - **Commands**: - - `make test` - Run all tests - - `make test-integration` - Run integration tests -- **Best Practice**: Always include tests with new features + - `uv run pytest tests/path/to/tests` to run a specific test file or folder. + - `make test` - Run all tests. + - `make test-integration` - Run integration tests. +- **Best Practice**: Always include tests with new features. ### Linting & Formatting +- You can delegate to the `lint` agent in `.github/agents` to resolve linting issues. - **Tools**: - - `ruff` - Formatting and linting - - `mypy` - Static type checking + - `ruff` - Formatting and linting. + - `mypy` - Static type checking. - **Commands**: - - `make lint` - Check for issues - - `make format` - Auto-format code -- **Requirement**: All code must be fully type-annotated + - `make lint` - Check for issues. + - `make format` - Auto-format code. +- **Requirement**: All code must be fully type-annotated. ### CLI -- **Framework**: Built with `typer` -- **Location**: `plugboard/cli/` -- **Usage**: `plugboard --help` +- **Framework**: Built with `typer`. +- **Location**: `plugboard/cli/`. +- **Usage**: `plugboard --help`. ## Code Standards ### Async Pattern -- Entire framework built on `asyncio` -- All I/O operations must be async -- All component lifecycle methods must be async +- Entire framework built on `asyncio`. +- All I/O operations must be async. +- All component lifecycle methods must be async. ### Dependency Injection -- Uses `that-depends` for DI -- Container setup: `plugboard/utils/DI.py` -- Access logger: `self._logger = DI.logger.resolve_sync().bind(...)` +- Uses `that-depends` for DI. +- Container setup: `plugboard/utils/DI.py`. +- Access logger: `self._logger = DI.logger.resolve_sync().bind(...)`. ### Data Structures -- Prefer immutable structures: `msgspec.Struct(frozen=True)` -- Use Pydantic models for validation where needed +- Prefer immutable structures: `msgspec.Struct(frozen=True)`. +- Use Pydantic models for validation where needed. ### Components When creating components: -1. Inherit from `plugboard.component.Component` -2. Always call `super().__init__()` in `__init__` -3. Declare I/O via class-level `io` attribute -4. Implement required async methods -5. Use type hints throughout +1. Inherit from `plugboard.component.Component`. +2. Always call `super().__init__()` in `__init__`. +3. Declare I/O via class-level `io` attribute. +4. Implement required async methods. +5. Use type hints throughout. Example: ```python @@ -107,46 +111,46 @@ class MyComponent(Component): ``` ### Events -- Event system for component communication -- Define events by inheriting from `plugboard.events.Event` -- Add handlers with `@Event.handler` decorator -- Emit events via `self.io.queue_event()` or return from handlers +- Event system for component communication. +- Define events by inheriting from `plugboard.events.Event`. +- Add handlers with `@Event.handler` decorator. +- Emit events via `self.io.queue_event()` or return from handlers. ## Best Practices -1. **Minimal Changes**: Make surgical, focused changes -2. **Type Safety**: Maintain full type annotations -3. **Testing**: Add tests for new functionality -4. **Documentation**: Update docstrings and docs for public APIs -5. **Async Discipline**: Never use blocking I/O operations -6. **Immutability**: Prefer immutable data structures -7. **Logging**: Use structured logging via `structlog` -8. **Error Handling**: Use appropriate exception types from `plugboard.exceptions` +1. **Minimal Changes**: Make surgical, focused changes. +2. **Type Safety**: Maintain full type annotations. +3. **Testing**: Add tests for new functionality. +4. **Documentation**: Update docstrings and docs for public APIs. You can delegate to the `docs` agent in `.github/agents` to maintain the project documentation. +5. **Async Discipline**: Never use blocking I/O operations. +6. **Immutability**: Prefer immutable data structures. +7. **Logging**: Use structured logging via `structlog`. +8. **Error Handling**: Use appropriate exception types from `plugboard.exceptions`. ## Common Tasks ### Adding a New Component -1. Create class inheriting from `Component` -2. Define `io` with inputs/outputs -3. Implement `__init__` with proper signature -4. Implement async `step` method -5. Add tests in `tests/` -6. Update documentation if public API +1. Create class inheriting from `Component`. +2. Define `io` with inputs/outputs. +3. Implement `__init__` with proper signature. +4. Implement async `step` method. +5. Add tests in `tests/`. +6. Update documentation if public API. ### Modifying Core Framework -1. Understand impact on existing components -2. Ensure backward compatibility where possible -3. Update type stubs if needed -4. Run full test suite -5. Update relevant documentation +1. Understand impact on existing components. +2. Ensure backward compatibility where possible. +3. Update type stubs if needed. +4. Run full test suite. +5. Update relevant documentation. ### Working with Events -1. Define event class with data model -2. Declare in component's `io` (input_events/output_events) -3. Implement handlers with decorators -4. Use `EventConnectorBuilder` for wiring +1. Define event class with data model. +2. Declare in component's `io` (input_events/output_events). +3. Implement handlers with decorators. +4. Use `EventConnectorBuilder` for wiring. -## Resources +## Online Resources - **Repository**: https://github.com/plugboard-dev/plugboard - **Documentation**: https://docs.plugboard.dev diff --git a/examples/AGENTS.md b/examples/AGENTS.md index e7e69fce..b8fa2abe 100644 --- a/examples/AGENTS.md +++ b/examples/AGENTS.md @@ -9,61 +9,48 @@ These examples demonstrate how to use Plugboard to model and simulate complex pr ## Example Categories ### Tutorials (`tutorials/`) + Step-by-step learning materials for new users. Focus on: -- Clear explanations of concepts -- Progressive complexity -- Runnable code with expected outputs -- Markdown documentation alongside code +- Clear explanations of concepts. +- Progressive complexity. +- Runnable code with expected outputs. +- Markdown documentation alongside code. You can delegate to the `docs` agent to make these updates. ### Demos (`demos/`) -Practical applications organized by domain: -- `fundamentals/`: Core Plugboard concepts -- `llm/`: LLM and AI integrations -- `physics-models/`: Physics-based simulations -- `finance/`: Financial modeling examples -## Creating Examples +Practical applications are organized by domain into folders. -### Planning a Model +## Creating a Plugboard model -Help users structure their model from concept to implementation: +Always using the following sequence of steps to help users plan, implement and run their models. -1. **Define Scope** - - Clear problem statement - - Expected inputs and outputs - - Success criteria +### Planning a Model -2. **Component Design** - - Break problem into logical components - - Define each component's inputs, outputs, and parameters - - Identify data dependencies and flow +Help users to plan their models from a high-level overview to a detailed design. This should include: -3. **Data Flow** - - Map connections between components - - Identify feedback loops - - Plan event-driven interactions if needed +* The inputs and outputs of the model; +* The components that will be needed to implement each part of the model, and any inputs, outputs and parameters they will need; +* The data flow between components, either via connectors or events. Identify any feedback loops and resolve if necessary. -Example breakdown for a hot-water tank model: -- **Components**: WaterTank, Heater, Thermostat, DataLoader, ResultSaver -- **Flow**: Temperature sensor → Thermostat → Heater → Tank → Temperature sensor -- **Data**: Load initial conditions, save temperature over time +Ask questions if anything is not clear about the business logic or you require additional domain expertise from the user. ### Implementing Components -Use appropriate components from the library or create custom ones: +Always check whether the functionality you need is already available in the library components in `plugboard.library`. For example, try to use: +- `FileReader` and `FileWriter` for reading/writing data from CSV or parquet files. +- `SQLReader` and `SQLReader` for reading/writing data from SQL databases. +- `LLMChat` for interacting with standard LLMs, e.g. OpenAI, Gemini, etc. **Using Built-in Components** -```python -from plugboard.library import Load, Save, Random -# Load data from file -data_loader = Load(name="load_data", path="input.csv") +```python +from plugboard.library import FileReader -# Save results -results_saver = Save(name="save_results", path="output.csv") +data_loader = FileReader(name="input_data", path="input.csv", field_names=["x", "y", "value"]) ``` **Creating Custom Components** + ```python import typing as _t from plugboard.component import Component, IOController as IO @@ -82,15 +69,37 @@ class Offset(Component): self._offset = offset async def step(self) -> None: + # Implement business logic here self.x = self.a + self._offset ``` +If a component is intended to be a source of new data into the model, then it should await `self.io.close()` when it has finished all the iterations it needs to do. This sends a signal into the `Process` so that other components know when the model has completed. For example, this component runs for a fixed number of iterations: + +```python +class Iterator(Component): + io = IO(outputs=["x"]) + + def __init__(self, iters: int, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: + super().__init__(**kwargs) + self._iters = iters + + async def init(self) -> None: + self._seq = iter(range(self._iters)) + + async def step(self) -> None: + try: + self.x = next(self._seq) + except StopIteration: + await self.io.close() +``` + ### Assembling a Process -Connect components and create a runnable process: +Connect components and create a runnable process. Unless asked otherwise, use a `LocalProcess`. ```python from plugboard.connector import AsyncioConnector +from plugboard.library import FileWriter from plugboard.process import LocalProcess from plugboard.schemas import ConnectorSpec @@ -106,23 +115,28 @@ process = LocalProcess( Offset(name="offset", offset=10), Scale(name="scale", scale=2), Sum(name="sum"), - Save(name="save-input", path="input.txt"), - Save(name="save-output", path="output.txt"), + FileWriter(name="save-output", path="results.csv", field_names=["input_value", "output_value"]), ], connectors=[ - connect("random.x", "save-input.value_to_save"), + connect("random.x", "save-output.input_value"), connect("random.x", "offset.a"), connect("random.x", "scale.a"), connect("offset.x", "sum.a"), connect("scale.x", "sum.b"), - connect("sum.x", "save-output.value_to_save"), + connect("sum.x", "save-output.output_value"), ], ) ``` +Check for circular loops when defining connectors in the `Process`. These will need to be resolved using the `initial_values` argument to a component somewhere within the loop, e.g. + +```python +my_component = MyComponent(name="test", initial_values={"x": [False], "y": [False]}) +``` + ### Visualizing Process Flow -Generate diagrams to help understand the model: +You can create a mermaid diagram to help users understand their models visually. ```python from plugboard.diagram import markdown_diagram @@ -133,139 +147,101 @@ print(diagram) ### Running the Model -Execute the process asynchronously: +You can help users to run their model. For example, to run the model defined above: ```python -import asyncio - -async def main(): - async with process: - await process.run() - -if __name__ == "__main__": - asyncio.run(main()) +async with process: + await process.run() ``` ## Event-Driven Models -For models requiring dynamic interactions and responses: +You can help users to implement event-driven models using Plugboard's event system. Components can emit and handle events to communicate with each other. -### When to Use Events -- Monitoring thresholds and triggering alerts -- Conditional workflows (if X happens, do Y) -- Multi-agent systems with communication -- Real-time data processing with decision points +Examples of where you might want to use events include: +* A component that monitors a data stream and emits an event when a threshold is crossed. +* A component that listens for events and triggers actions in response, e.g. sending an alert. +* A trading algorithm that uses events to signal buy/sell decisions. +* Where a model has conditional workflows, e.g. process data differently in the model depending on a specific condition. -### Defining Events +Events must be defined by inheriting from the `plugboard.events.Event` class. Each event class should define the data it carries using a Pydantic `BaseModel`. For example: ```python from pydantic import BaseModel from plugboard.events import Event -class ThresholdCrossedData(BaseModel): - component: str - value: float - threshold: float - timestamp: float +class MyEventData(BaseModel): + some_value: int + another_value: str -class ThresholdCrossed(Event): - data: ThresholdCrossedData +class MyEvent(Event): + data: MyEventData ``` -### Publishing Events +Components can emit events using the `self.io.queue_event()` method or by returning them from an event handler. Event handlers are defined using methods decorated with `@EventClass.handler`. For example: ```python from plugboard.component import Component, IOController as IO -class Monitor(Component): - io = IO( - inputs=["value"], - outputs=[], - output_events=[ThresholdCrossed] - ) - - def __init__(self, threshold: float, **kwargs) -> None: - super().__init__(**kwargs) - self._threshold = threshold - - async def step(self) -> None: - if self.value > self._threshold: - event_data = ThresholdCrossedData( - component=self.name, - value=self.value, - threshold=self._threshold, - timestamp=time.time() - ) - self.io.queue_event( - ThresholdCrossed(source=self.name, data=event_data) - ) -``` - -### Subscribing to Events +class MyEventPublisher(Component): + io = IO(inputs=["some_input"], output_events=[MyEvent]) -```python -class AlertHandler(Component): - io = IO( - inputs=[], - outputs=[], - input_events=[ThresholdCrossed], - output_events=[AlertSent] - ) - - @ThresholdCrossed.handler - async def handle_threshold(self, event: ThresholdCrossed) -> AlertSent: - print(f"ALERT: {event.data.component} exceeded threshold!") - - # Return new event - alert_data = AlertSentData( - original_event=event.data, - alert_method="console" - ) - return AlertSent(source=self.name, data=alert_data) + async def step(self) -> None: + # Emit an event + event_data = MyEventData(some_value=42, another_value=f"received {self.some_input}") + self.io.queue_event(MyEvent(source=self.name, data=event_data)) + +class MyEventSubscriber(Component): + io = IO(input_events=[MyEvent], output_events=[MyEvent]) + + @MyEvent.handler + async def handle_my_event(self, event: MyEvent) -> MyEvent: + # Handle the event + print(f"Received event: {event.data}") + output_event_data = MyEventData(some_value=event.data.some_value + 1, another_value="handled") + return MyEvent(source=self.name, data=output_event_data) ``` -### Wiring Event-Driven Components +To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using `plugboard.events.event_connector_builder.EventConnectorBuilder`. For example: ```python -from plugboard.connector import ConnectorBuilder +from plugboard.connector import AsyncioConnector, ConnectorBuilder from plugboard.events.event_connector_builder import EventConnectorBuilder +from plugboard.process import LocalProcess -# Regular connectors -connect = lambda src, tgt: AsyncioConnector( - spec=ConnectorSpec(source=src, target=tgt) -) +# Define components.... +component_1 = ... +component_2 = ... + +# Define connectors for non-event components as before +connect = lambda in_, out_: AsyncioConnector(spec=ConnectorSpec(source=in_, target=out_)) connectors = [ - connect("data_source.value", "monitor.value"), - # ... other data connectors + connect("component_1.output", "component_2.input"), + ... ] -# Event connectors (automatically wired) connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) -event_connector_builder = EventConnectorBuilder( - connector_builder=connector_builder -) -components = [monitor, alert_handler, ...] -event_connectors = list( - event_connector_builder.build(components).values() -) +event_connector_builder = EventConnectorBuilder(connector_builder=connector_builder) +event_connectors = list(event_connector_builder.build(components).values()) -# Create process with both types process = LocalProcess( - components=components, + components=[ + component_1, component_2, ... + ], connectors=connectors + event_connectors, ) ``` ## Exporting Models -Save process configuration for reuse: +Save a process configuration for reuse: ```python -# Export to YAML -process.dump("my_model.yaml") +process.dump("my-model.yaml") -# Later, load and run via CLI -# $ plugboard process run my_model.yaml +Later, load and run via CLI +```sh +plugboard process run my-model.yaml ``` ## Jupyter Notebooks @@ -273,6 +249,7 @@ process.dump("my_model.yaml") For interactive demonstrations: 1. **Structure** + - Title markdown cell in the same format as the other notebooks, including badges to run on Github/Colab - Clear markdown sections - Code cells with explanations - Visualizations of results @@ -286,69 +263,10 @@ For interactive demonstrations: - Clean up resources properly 3. **Output** - - Include sample outputs in notebook + - Clear cell output before committing - Generate plots where helpful - Provide interpretation of results -## Documentation Standards - -### Code Comments -- Explain *why*, not *what* -- Document assumptions -- Note limitations or edge cases - -### Docstrings -- Use Google-style docstrings -- Document all parameters and return values -- Include usage examples for complex functions - -### Markdown Files -- Clear headings and structure -- Code blocks with syntax highlighting -- Link to relevant API documentation -- Include expected outputs - -## Testing Examples - -While examples are primarily educational: -- Ensure all code actually runs -- Verify outputs are as expected -- Check for common error cases -- Test with different parameter values - -## Common Patterns - -### Data Processing Pipeline -```python -components = [ - Load(name="load", path="data.csv"), - Transform(name="transform", ...), - Filter(name="filter", ...), - Save(name="save", path="output.csv"), -] -``` - -### Simulation Loop -```python -components = [ - Clock(name="clock", steps=100), - Model(name="model", ...), - Observer(name="observer", ...), - Save(name="save", ...), -] -``` - -### Multi-Agent System -```python -components = [ - Agent(name="agent_1", ...), - Agent(name="agent_2", ...), - Environment(name="env", ...), - Coordinator(name="coord", ...), -] -# Use events for agent communication -``` - ## Resources - **Library Components**: `plugboard.library` From b0df8b00199129bc47f7cd6b5c419926fef8fc6d Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 13:43:37 +0000 Subject: [PATCH 08/14] Enable nested md files in devcontainer --- .devcontainer/devcontainer.json | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 250e14cb..c6d5c965 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -46,7 +46,9 @@ "python.testing.unittestEnabled": false, "python.testing.pytestEnabled": true, "ruff.fixAll": true, - "ruff.organizeImports": true + "ruff.organizeImports": true, + "chat.useAgentsMdFile": true, + "chat.useNestedAgentsMdFiles": true } } } From ffcc94ec8aeb67e5be1f249e89e6fa40263906dd Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 13:47:45 +0000 Subject: [PATCH 09/14] Update contributing guidelines --- CONTRIBUTING.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d38121ae..c18c82d9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,9 +44,8 @@ uv run mkdocs serve This repo includes custom AI agent prompts to assist with development: -- [AGENTS.md](AGENTS.md) - General guidelines for working with the Plugboard codebase -- [examples/AGENTS.md](examples/AGENTS.md) - Specific guidance for building example models and demos -- [.github/agents/docs/AGENTS.md](.github/agents/docs/AGENTS.md) - Documentation maintenance agent -- [.github/agents/linting/AGENTS.md](.github/agents/linting/AGENTS.md) - Linting and code quality agent +- [AGENTS.md](AGENTS.md) - General guidelines for working with the Plugboard codebase. +- [examples/AGENTS.md](examples/AGENTS.md) - Specific guidance for building example models and demos. +- Copilot-specific agents `docs`, `lint` and `test` which you can @-mention in a pull request. If you use GitHub Copilot or other AI coding assistants that support the AGENTS.md convention, these prompts can help you build Plugboard models from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. From 1a658129935367be673e67584505ef5d3b983fc6 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 13:53:20 +0000 Subject: [PATCH 10/14] Add more linting tools --- .github/agents/lint.agent.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/agents/lint.agent.md b/.github/agents/lint.agent.md index 855a8e5a..e6552f55 100644 --- a/.github/agents/lint.agent.md +++ b/.github/agents/lint.agent.md @@ -10,6 +10,7 @@ You are responsible for maintaining code quality in the Plugboard project by run - Review the output from these tools and identify any issues that need to be resolved. - Edit the code to fix any linting issues or type errors that are identified. - Ensure that all code is fully type-annotated and adheres to the project's coding standards. +- Review the code complexity using `xenon` and carry out refactoring if required. - Ensure that public methods and functions have docstrings that follow the project's documentation standards. ## Project knowledge: @@ -21,8 +22,12 @@ You are responsible for maintaining code quality in the Plugboard project by run - Run `uv run ruff format` to reformat the code. - Run `uv run ruff check` to check for linting issues. - Run `uv run mypy .` to check for type errors. +- Run `uv lock --check` to check that the uv lockfile is up to date. +- Run `uv run xenon --max-absolute B --max-modules A --max-average A plugboard/` to check for code complexity. +- Run `find . -name '*.ipynb' -not -path "./.venv/*" -exec uv run nbstripout --verify {} +` to check that Jupyter notebooks are stripped of output. ## Boundaries: - **Always** fix any linting issues or type errors that are identified by the tools. - **Always** ensure that all code is fully type-annotated and adheres to the project's coding standards. -- **Never** change the logic of the code - only make changes necessary to resolve linting issues or type errors. +- **Never** change the fundamental logic of the code - only make changes necessary to resolve code quality issues. + \ No newline at end of file From 31a5301b75631113e7d9c7573bc6ed1a474bd013 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 13:57:14 +0000 Subject: [PATCH 11/14] Typo --- .github/agents/lint.agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/agents/lint.agent.md b/.github/agents/lint.agent.md index e6552f55..1998b07f 100644 --- a/.github/agents/lint.agent.md +++ b/.github/agents/lint.agent.md @@ -6,7 +6,7 @@ tools: ['execute', 'read', 'edit', 'search', 'ms-python.python/getPythonEnvironm You are responsible for maintaining code quality in the Plugboard project by running linting tools and resolving any issues that arise. ## Your role: -- Run `ruff` to check for formatting andlinting issues and `mypy` to check for type errors. +- Run `ruff` to check for formatting and linting issues and `mypy` to check for type errors. - Review the output from these tools and identify any issues that need to be resolved. - Edit the code to fix any linting issues or type errors that are identified. - Ensure that all code is fully type-annotated and adheres to the project's coding standards. From 1aa6f5e3685aa109072de28ddf2319e1589e0cb5 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 14:33:22 +0000 Subject: [PATCH 12/14] Update example for event-driven components --- examples/AGENTS.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/examples/AGENTS.md b/examples/AGENTS.md index b8fa2abe..0d3f2fb2 100644 --- a/examples/AGENTS.md +++ b/examples/AGENTS.md @@ -202,11 +202,10 @@ class MyEventSubscriber(Component): return MyEvent(source=self.name, data=output_event_data) ``` -To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using `plugboard.events.event_connector_builder.EventConnectorBuilder`. For example: +To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using a `ConnectorBuilder`. For example: ```python from plugboard.connector import AsyncioConnector, ConnectorBuilder -from plugboard.events.event_connector_builder import EventConnectorBuilder from plugboard.process import LocalProcess # Define components.... @@ -219,10 +218,8 @@ connectors = [ connect("component_1.output", "component_2.input"), ... ] - -connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) -event_connector_builder = EventConnectorBuilder(connector_builder=connector_builder) -event_connectors = list(event_connector_builder.build(components).values()) +# Define connectors for events +event_connectors = AsyncioConnector.builder().build_event_connectors(components) process = LocalProcess( components=[ From 02797e8bf0b381008aaec0ae744436163861e3b8 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 14:36:33 +0000 Subject: [PATCH 13/14] Update event-driven prompt example --- examples/AGENTS.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/examples/AGENTS.md b/examples/AGENTS.md index 0d3f2fb2..9fc4bca8 100644 --- a/examples/AGENTS.md +++ b/examples/AGENTS.md @@ -243,7 +243,7 @@ plugboard process run my-model.yaml ## Jupyter Notebooks -For interactive demonstrations: +Use the following guidelines when creating demo notebooks: 1. **Structure** - Title markdown cell in the same format as the other notebooks, including badges to run on Github/Colab @@ -257,7 +257,6 @@ For interactive demonstrations: - Add docstrings to helper functions - Show intermediate results - Include error handling - - Clean up resources properly 3. **Output** - Clear cell output before committing From 3d4e68eb811b36c4c7912e7d767f84035fc887f7 Mon Sep 17 00:00:00 2001 From: Toby Coleman Date: Sun, 15 Feb 2026 14:47:31 +0000 Subject: [PATCH 14/14] Add logging example --- examples/AGENTS.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/AGENTS.md b/examples/AGENTS.md index 9fc4bca8..8917cfdd 100644 --- a/examples/AGENTS.md +++ b/examples/AGENTS.md @@ -51,6 +51,8 @@ data_loader = FileReader(name="input_data", path="input.csv", field_names=["x", **Creating Custom Components** +New components should inherit from `plugboard.componen.Component`. Add logging messages where it would be helpful by using the bound logger `self._logger`. + ```python import typing as _t from plugboard.component import Component, IOController as IO @@ -90,6 +92,7 @@ class Iterator(Component): try: self.x = next(self._seq) except StopIteration: + self._logger.info("Iterator exhausted", total_iterations=self._iters) await self.io.close() ```