diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 250e14cb..c6d5c965 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -46,7 +46,9 @@ "python.testing.unittestEnabled": false, "python.testing.pytestEnabled": true, "ruff.fixAll": true, - "ruff.organizeImports": true + "ruff.organizeImports": true, + "chat.useAgentsMdFile": true, + "chat.useNestedAgentsMdFiles": true } } } diff --git a/.github/agents/docs.agent.md b/.github/agents/docs.agent.md new file mode 100644 index 00000000..2fa4d8be --- /dev/null +++ b/.github/agents/docs.agent.md @@ -0,0 +1,26 @@ +--- +description: 'Maintain Plugboard documentation' +tools: ['execute', 'read', 'edit', 'search', 'web', 'github.vscode-pull-request-github/activePullRequest', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'todo'] +--- + +You are an expert technical writer responsible for maintaining the documentation of the Plugboard project. You write for a technical audience includeing developers, data scientists and domain experts who want to build models in Plugboard. + +## Your role: +- Read code from `plugboard/` and `plugboard-schemas/` to understand the framework and its components. +- Read examples in `examples/` to see how the framework is used in practice. +- Update documentation in `docs/` to reflect any changes or additions to the framework. +- Write clear, concise, and accurate documentation that helps users understand how to use the framework effectively. + +## Project knowledge: +- Documentation is written using MkDocs material theme and is located in the `docs/` directory. +- The Python project is managed using `uv`. + +## Commands you can run: +- Build the docs using `uv run mkdocs build`. +- Serve the docs locally using `uv run mkdocs serve`. + +## Boundaries: +- **Always** write new markdown files in the `docs/` directory for new documentation. +- **Always** update existing markdown files in the `docs/` directory when changes are made to the framework that affect existing documentation. +- **Always** update the `mkdocs.yaml` file in the project rootto include any new markdown files you create in the documentation. +- **Never** make changes to source code files in `plugboard/` or `plugboard-schemas/` unless you are fixing a documentation-related issue (e.g. a docstring that is inaccurate or incomplete). diff --git a/.github/agents/lint.agent.md b/.github/agents/lint.agent.md new file mode 100644 index 00000000..1998b07f --- /dev/null +++ b/.github/agents/lint.agent.md @@ -0,0 +1,33 @@ +--- +description: 'Maintain code quality by running linting tools and resolving issues' +tools: ['execute', 'read', 'edit', 'search', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment'] +--- + +You are responsible for maintaining code quality in the Plugboard project by running linting tools and resolving any issues that arise. + +## Your role: +- Run `ruff` to check for formatting and linting issues and `mypy` to check for type errors. +- Review the output from these tools and identify any issues that need to be resolved. +- Edit the code to fix any linting issues or type errors that are identified. +- Ensure that all code is fully type-annotated and adheres to the project's coding standards. +- Review the code complexity using `xenon` and carry out refactoring if required. +- Ensure that public methods and functions have docstrings that follow the project's documentation standards. + +## Project knowledge: +- The project uses `uv` for dependency management and running commands. +- Linting and formatting are handled by `ruff`, while static type checking is handled by `mypy`. +- `pyproject.toml` contains the settings for `ruff` and `mypy`. + +## Commands you can run: +- Run `uv run ruff format` to reformat the code. +- Run `uv run ruff check` to check for linting issues. +- Run `uv run mypy .` to check for type errors. +- Run `uv lock --check` to check that the uv lockfile is up to date. +- Run `uv run xenon --max-absolute B --max-modules A --max-average A plugboard/` to check for code complexity. +- Run `find . -name '*.ipynb' -not -path "./.venv/*" -exec uv run nbstripout --verify {} +` to check that Jupyter notebooks are stripped of output. + +## Boundaries: +- **Always** fix any linting issues or type errors that are identified by the tools. +- **Always** ensure that all code is fully type-annotated and adheres to the project's coding standards. +- **Never** change the fundamental logic of the code - only make changes necessary to resolve code quality issues. + \ No newline at end of file diff --git a/.github/agents/test.agent.md b/.github/agents/test.agent.md new file mode 100644 index 00000000..177b1e26 --- /dev/null +++ b/.github/agents/test.agent.md @@ -0,0 +1,26 @@ +--- +description: 'Create and maintain unit and integration tests' +tools: ['execute', 'edit', 'search', 'github.vscode-pull-request-github/activePullRequest', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'todo'] +--- + +You are an expert software engineer responsible for creating and maintaining unit and integration tests for the Plugboard project. Your role is crucial in ensuring the reliability and stability of the codebase by writing tests that cover new features, bug fixes, and existing functionality. + +## Your role: +- Write unit tests for new components and features added to the Plugboard framework. +- Write integration tests to ensure that different components of the framework work together as expected. +- Update existing tests when changes are made to the codebase that affect existing functionality. +- Collaborate with other developers to understand the expected behavior of new features and components to ensure comprehensive test coverage. + +## Project knowledge: +- Tests are located in the `tests/` directory. +- Tests are written using the `pytest` framework. + +## Commands you can run: +- Run all tests using `make test`. +- Run specific tests using `uv run pytest tests/path/to/test_file.py`. + +## Boundaries: +- **Always** write new test files in the `tests/` directory for new features and components. +- **Always** update existing test files in the `tests/` directory when changes are made to the codebase that affect existing functionality. +- **Ask first** before making changes to source code files in `plugboard/` or `plugboard-schemas/` to clarify the expected behavior of new features or components if it is not clear from the documentation or code comments. +- **Never** remove failing tests without explicit instruction to do so, as they may indicate important issues that need to be addressed. diff --git a/.github/instructions/models.instructions.md b/.github/instructions/models.instructions.md deleted file mode 100644 index a2a3cf8f..00000000 --- a/.github/instructions/models.instructions.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -applyTo: "examples/**/*.py,examples/**/*.ipynb" ---- - -# Project Overview - -Plugboard is an event-driven modelling and orchestration framework in Python for simulating and driving complex processes with many interconnected stateful components. - -## Planning a model - -Help users to plan their models from a high-level overview to a detailed design. This should include: - -* The inputs and outputs of the model; -* The components that will be needed to implement each part of the model, and any inputs, outputs and parameters they will need; -* The data flow between components. - -For example, a model of a hot-water tank might have components for the water tank, the heater and the thermostat. Additional components might be needed to load data from a file or database, and similarly to save simulation results. - -## Implementing components - -Help users set up the components they need to implement their model. Custom components can be implemented by subclassing the [`Component`][plugboard.component.Component]. Common components for tasks like loading data can be imported from [`plugboard.library`][plugboard.library]. - -An empty component looks like this: - -```python -import typing as _t - -from plugboard.component import Component, IOController as IO -from plugboard.schemas import ComponentArgsDict - -class Offset(Component): - """Implements `x = a + offset`.""" - io = IO(inputs=["a"], outputs=["x"]) - - def __init__(self, offset: float = 0, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: - super().__init__(**kwargs) - self._offset = offset - - async def step(self) -> None: - # TODO: Implement business logic here - # Example `self.x = self.a + self._offset` - pass -``` - -## Connecting components into a process - -You can help users to connect their components together. For initial development and testing use a [LocalProcess][plugboard.process.LocalProcess] to run the model in a single process. - -Example code to connect components together and create a process: - -```python -from plugboard.connector import AsyncioConnector -from plugboard.process import LocalProcess -from plugboard.schemas import ConnectorSpec - -connect = lambda in_, out_: AsyncioConnector( - spec=ConnectorSpec(source=in_, target=out_) -) -process = LocalProcess( - components=[ - Random(name="random", iters=5, low=0, high=10), - Offset(name="offset", offset=10), - Scale(name="scale", scale=2), - Sum(name="sum"), - Save(name="save-input", path="input.txt"), - Save(name="save-output", path="output.txt"), - ], - connectors=[ - # Connect x output of the component named "random" to the value_to_save input of the component named "save-input", etc. - connect("random.x", "save-input.value_to_save"), - connect("random.x", "offset.a"), - connect("random.x", "scale.a"), - connect("offset.x", "sum.a"), - connect("scale.x", "sum.b"), - connect("sum.x", "save-output.value_to_save"), - ], -) -``` - -If you need a diagram of the process you can import `plugboard.diagram.markdown_diagram` and use it to create a markdown representation of the process: - -```python -from plugboard.diagram import markdown_diagram -diagram = markdown_diagram(process) -print(diagram) -``` - -## Running the model - -You can help users to run their model. For example, to run the model defined above: - -```python - -import asyncio - -async with process: - await process.run() -``` - -## Event-driven models - -You can help users to implement event-driven models using Plugboard's event system. Components can emit and handle events to communicate with each other. - -Examples of where you might want to use events include: -* A component that monitors a data stream and emits an event when a threshold is crossed; -* A component that listens for events and triggers actions in response, e.g. sending an alert; -* A trading algorithm that uses events to signal buy/sell decisions. - -Events must be defined by inheriting from the `plugboard.events.Event` class. Each event class should define the data it carries using a Pydantic `BaseModel`. For example: - -```python -from pydantic import BaseModel -from plugboard.events import Event - -class MyEventData(BaseModel): - some_value: int - another_value: str - -class MyEvent(Event): - data: MyEventData -``` - -Components can emit events using the `self.io.queue_event()` method or by returning them from an event handler. Event handlers are defined using methods decorated with `@EventClass.handler`. For example: - -```python -from plugboard.component import Component, IOController as IO - -class MyEventPublisher(Component): - io = IO(inputs=["some_input"], output_events=[MyEvent]) - - async def step(self) -> None: - # Emit an event - event_data = MyEventData(some_value=42, another_value=f"received {self.some_input}") - self.io.queue_event(MyEvent(source=self.name, data=event_data)) - -class MyEventSubscriber(Component): - io = IO(input_events=[MyEvent], output_events=[MyEvent]) - - @MyEvent.handler - async def handle_my_event(self, event: MyEvent) -> MyEvent: - # Handle the event - print(f"Received event: {event.data}") - output_event_data = MyEventData(some_value=event.data.some_value + 1, another_value="handled") - return MyEvent(source=self.name, data=output_event_data) -``` - -To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using `plugboard.events.event_connector_builder.EventConnectorBuilder`. For example: - -```python -from plugboard.connector import AsyncioConnector, ConnectorBuilder -from plugboard.events.event_connector_builder import EventConnectorBuilder -from plugboard.process import LocalProcess - -# Define components.... -component_1 = ... -component_2 = ... - -# Define connectors for non-event components as before -connect = lambda in_, out_: AsyncioConnector(spec=ConnectorSpec(source=in_, target=out_)) -connectors = [ - connect("component_1.output", "component_2.input"), - ... -] - -connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) -event_connector_builder = EventConnectorBuilder(connector_builder=connector_builder) -event_connectors = list(event_connector_builder.build(components).values()) - -process = LocalProcess( - components=[ - component_1, component_2, ... - ], - connectors=connectors + event_connectors, -) -``` - -## Exporting models - -If the user wants to export their model you use in the CLI, you can do this by calling `process.dump("path/to/file.yaml")`. diff --git a/.github/instructions/source.instructions.md b/.github/instructions/source.instructions.md deleted file mode 100644 index 050a5602..00000000 --- a/.github/instructions/source.instructions.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -applyTo: "plugboard/**/*.py" ---- -# GitHub Copilot Instructions for the Plugboard Repository - -This document provides guidelines for using AI coding agents to contribute to the Plugboard project. Following these instructions will help ensure that contributions are consistent with the project's architecture, conventions, and style. - -## Project Overview & Architecture - -Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes. The core architectural concepts are `Component`, `Process`, and `Connector`. - -- **`Component`**: The fundamental building block for modeling logic. Found in `plugboard/component/`. - - Components have a defined lifecycle: `__init__`, `init`, `step`, `run`, and `destroy`. - - I/O (inputs, outputs, events) is declared via a class-level `io: IOController` attribute. - - The `step` method contains the primary logic and is executed repeatedly. - - The framework is asynchronous (`asyncio`), so all lifecycle methods (`init`, `step`, `destroy`) must be `async`. - -- **`Process`**: Manages a collection of `Component`s and their interconnections. Found in `plugboard/process/`. - - A `Process` orchestrates the execution of components. - - `LocalProcess` runs all components in a single process. Other process types may support distributed execution. - -- **`Connector`**: Defines the communication channels between component outputs and inputs. Found in `plugboard/connector/`. - - Connectors link a source (`component_name.output_name`) to a target (`component_name.input_name`). - -- **State Management**: The `StateBackend` (see `plugboard/state/`) tracks the status of all components and the overall process. This is crucial for monitoring and for distributed execution. - -- **Configuration**: Processes can be defined in Python or declared in YAML files for execution via the CLI (`plugboard process run ...`). - -## Developer Workflow - -- **Setup**: The project uses `uv` for dependency management. Set up your environment and install dependencies from `pyproject.toml`. -- **Testing**: Tests are written with `pytest` and are located in the `tests/` directory. - - Run all tests with `make test`. - - Run integration tests with `make test-integration`. - - When adding a new feature, please include corresponding unit and/or integration tests. -- **Linting & Formatting**: The project uses `ruff` for formatting and linting, and `mypy` for static type checking. - - Run `make lint` to check for issues. - - Run `make format` to automatically format the code. - - All code must be fully type-annotated. -- **CLI**: The command-line interface is defined using `typer` in `plugboard/cli/`. Use `plugboard --help` to see available commands. - -## Code Conventions & Patterns - -- **Asynchronous Everywhere**: The entire framework is built on `asyncio`. All I/O operations and component lifecycle methods should be `async`. -- **Dependency Injection**: The project uses `that-depends` for dependency injection. See `plugboard/utils/DI.py` for the container setup. -- **Immutability**: Use `msgspec.Struct(frozen=True)` for data structures that should be immutable. -- **Extending Components**: When creating a new component, inherit from `plugboard.component.Component` and implement the required `async` methods. Remember to call `super().__init__()`. -- **Events**: Components can communicate via an event system. Define custom events by inheriting from `plugboard.events.Event` and add handlers to your component using the `@Event.handler` decorator. -- **Logging**: Use the `structlog` logger available through dependency injection: `self._logger = DI.logger.resolve_sync().bind(...)`. - -By adhering to these guidelines, you can help maintain the quality and consistency of the Plugboard codebase. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..2bb5af98 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,158 @@ +# AI Agent Instructions for Plugboard + +This document provides guidelines for AI coding agents working on the Plugboard project. Following these instructions ensures contributions are consistent with the project's architecture, conventions, and style. + +## Project Overview + +Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes with interconnected stateful components. Typical users are data scientists and engineers. + +### Core Architecture + +**Component**: The fundamental building block for modeling logic (see `plugboard/component/`). +- Lifecycle: `__init__` → `init` → `step` (repeated) → `destroy`. +- I/O declaration: Use class-level `io: IOController` attribute. Use this to specify inputs, outputs and events associated with a component. +- Business logic: Implement in the `step` method. +- Asynchronous: All lifecycle methods (`init`, `step`, `destroy`) must be `async`. + +**Process**: Orchestrates execution of components (see `plugboard/process/`) +- Manages collections of components and their interconnections. +- `LocalProcess`: Runs all components in a single process. +- Supports distributed execution via other process types, e.g. `RayProcess`. + +**Connector**: Defines communication channels between components (see `plugboard/connector/`). +- Links outputs to inputs: `component_name.output_name` → `component_name.input_name`. +- Various connector types available for different execution contexts. + +**State Management**: Tracks component and process status (see `plugboard/state/`). +- Critical for monitoring and distributed execution. +- Uses `StateBackend` abstraction. + +**Configuration**: Flexible process definition. +- Python-based: Direct component instantiation. +- YAML-based: Declarative configuration for CLI execution (`plugboard process run ...`). +- Relies on the Pydantic objects defined in `plugboard-schemas`. + +## Development Environment + +### Setup +- **Package Manager**: Uses `uv` for dependency management. +- **Dependencies**: Defined in `pyproject.toml`. +- **Python Version**: Requires Python 3.12 or higher. + +### Testing +- You can delegate to the `test` agent in `.github/agents` to update/add tests. +- **Framework**: `pytest` +- **Location**: `tests/` directory +- **Commands**: + - `uv run pytest tests/path/to/tests` to run a specific test file or folder. + - `make test` - Run all tests. + - `make test-integration` - Run integration tests. +- **Best Practice**: Always include tests with new features. + +### Linting & Formatting +- You can delegate to the `lint` agent in `.github/agents` to resolve linting issues. +- **Tools**: + - `ruff` - Formatting and linting. + - `mypy` - Static type checking. +- **Commands**: + - `make lint` - Check for issues. + - `make format` - Auto-format code. +- **Requirement**: All code must be fully type-annotated. + +### CLI +- **Framework**: Built with `typer`. +- **Location**: `plugboard/cli/`. +- **Usage**: `plugboard --help`. + +## Code Standards + +### Async Pattern +- Entire framework built on `asyncio`. +- All I/O operations must be async. +- All component lifecycle methods must be async. + +### Dependency Injection +- Uses `that-depends` for DI. +- Container setup: `plugboard/utils/DI.py`. +- Access logger: `self._logger = DI.logger.resolve_sync().bind(...)`. + +### Data Structures +- Prefer immutable structures: `msgspec.Struct(frozen=True)`. +- Use Pydantic models for validation where needed. + +### Components +When creating components: +1. Inherit from `plugboard.component.Component`. +2. Always call `super().__init__()` in `__init__`. +3. Declare I/O via class-level `io` attribute. +4. Implement required async methods. +5. Use type hints throughout. + +Example: +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class MyComponent(Component): + io = IO(inputs=["input_a"], outputs=["output_x"]) + + def __init__( + self, + param: float = 1.0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._param = param + + async def step(self) -> None: + # Business logic here + self.output_x = self.input_a * self._param +``` + +### Events +- Event system for component communication. +- Define events by inheriting from `plugboard.events.Event`. +- Add handlers with `@Event.handler` decorator. +- Emit events via `self.io.queue_event()` or return from handlers. + +## Best Practices + +1. **Minimal Changes**: Make surgical, focused changes. +2. **Type Safety**: Maintain full type annotations. +3. **Testing**: Add tests for new functionality. +4. **Documentation**: Update docstrings and docs for public APIs. You can delegate to the `docs` agent in `.github/agents` to maintain the project documentation. +5. **Async Discipline**: Never use blocking I/O operations. +6. **Immutability**: Prefer immutable data structures. +7. **Logging**: Use structured logging via `structlog`. +8. **Error Handling**: Use appropriate exception types from `plugboard.exceptions`. + +## Common Tasks + +### Adding a New Component +1. Create class inheriting from `Component`. +2. Define `io` with inputs/outputs. +3. Implement `__init__` with proper signature. +4. Implement async `step` method. +5. Add tests in `tests/`. +6. Update documentation if public API. + +### Modifying Core Framework +1. Understand impact on existing components. +2. Ensure backward compatibility where possible. +3. Update type stubs if needed. +4. Run full test suite. +5. Update relevant documentation. + +### Working with Events +1. Define event class with data model. +2. Declare in component's `io` (input_events/output_events). +3. Implement handlers with decorators. +4. Use `EventConnectorBuilder` for wiring. + +## Online Resources + +- **Repository**: https://github.com/plugboard-dev/plugboard +- **Documentation**: https://docs.plugboard.dev +- **Issue Tracker**: GitHub Issues +- **Discussions**: GitHub Discussions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2c60c9ad..c18c82d9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,6 +40,12 @@ The package documentation uses [Material for MkDocs](https://squidfunk.github.io uv run mkdocs serve ``` -### Building example models +### AI-assisted development -This repo includes a [custom LLM prompt](.github/instructions/models.instructions.md) for the [examples](examples/) folder. If you use GitHub Copilot, this can help you build a Plugboard model from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. +This repo includes custom AI agent prompts to assist with development: + +- [AGENTS.md](AGENTS.md) - General guidelines for working with the Plugboard codebase. +- [examples/AGENTS.md](examples/AGENTS.md) - Specific guidance for building example models and demos. +- Copilot-specific agents `docs`, `lint` and `test` which you can @-mention in a pull request. + +If you use GitHub Copilot or other AI coding assistants that support the AGENTS.md convention, these prompts can help you build Plugboard models from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. diff --git a/examples/AGENTS.md b/examples/AGENTS.md new file mode 100644 index 00000000..8917cfdd --- /dev/null +++ b/examples/AGENTS.md @@ -0,0 +1,275 @@ +# AI Agent Instructions for Plugboard Examples + +This document provides guidelines for AI agents working with Plugboard example code, demonstrations, and tutorials. + +## Purpose + +These examples demonstrate how to use Plugboard to model and simulate complex processes. Help users build intuitive, well-documented examples that showcase Plugboard's capabilities. + +## Example Categories + +### Tutorials (`tutorials/`) + +Step-by-step learning materials for new users. Focus on: +- Clear explanations of concepts. +- Progressive complexity. +- Runnable code with expected outputs. +- Markdown documentation alongside code. You can delegate to the `docs` agent to make these updates. + +### Demos (`demos/`) + +Practical applications are organized by domain into folders. + +## Creating a Plugboard model + +Always using the following sequence of steps to help users plan, implement and run their models. + +### Planning a Model + +Help users to plan their models from a high-level overview to a detailed design. This should include: + +* The inputs and outputs of the model; +* The components that will be needed to implement each part of the model, and any inputs, outputs and parameters they will need; +* The data flow between components, either via connectors or events. Identify any feedback loops and resolve if necessary. + +Ask questions if anything is not clear about the business logic or you require additional domain expertise from the user. + +### Implementing Components + +Always check whether the functionality you need is already available in the library components in `plugboard.library`. For example, try to use: +- `FileReader` and `FileWriter` for reading/writing data from CSV or parquet files. +- `SQLReader` and `SQLReader` for reading/writing data from SQL databases. +- `LLMChat` for interacting with standard LLMs, e.g. OpenAI, Gemini, etc. + +**Using Built-in Components** + +```python +from plugboard.library import FileReader + +data_loader = FileReader(name="input_data", path="input.csv", field_names=["x", "y", "value"]) +``` + +**Creating Custom Components** + +New components should inherit from `plugboard.componen.Component`. Add logging messages where it would be helpful by using the bound logger `self._logger`. + +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class Offset(Component): + """Adds a constant offset to input value.""" + io = IO(inputs=["a"], outputs=["x"]) + + def __init__( + self, + offset: float = 0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._offset = offset + + async def step(self) -> None: + # Implement business logic here + self.x = self.a + self._offset +``` + +If a component is intended to be a source of new data into the model, then it should await `self.io.close()` when it has finished all the iterations it needs to do. This sends a signal into the `Process` so that other components know when the model has completed. For example, this component runs for a fixed number of iterations: + +```python +class Iterator(Component): + io = IO(outputs=["x"]) + + def __init__(self, iters: int, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: + super().__init__(**kwargs) + self._iters = iters + + async def init(self) -> None: + self._seq = iter(range(self._iters)) + + async def step(self) -> None: + try: + self.x = next(self._seq) + except StopIteration: + self._logger.info("Iterator exhausted", total_iterations=self._iters) + await self.io.close() +``` + +### Assembling a Process + +Connect components and create a runnable process. Unless asked otherwise, use a `LocalProcess`. + +```python +from plugboard.connector import AsyncioConnector +from plugboard.library import FileWriter +from plugboard.process import LocalProcess +from plugboard.schemas import ConnectorSpec + +# Helper for creating connectors +connect = lambda src, tgt: AsyncioConnector( + spec=ConnectorSpec(source=src, target=tgt) +) + +# Create process with components and connectors +process = LocalProcess( + components=[ + Random(name="random", iters=5, low=0, high=10), + Offset(name="offset", offset=10), + Scale(name="scale", scale=2), + Sum(name="sum"), + FileWriter(name="save-output", path="results.csv", field_names=["input_value", "output_value"]), + ], + connectors=[ + connect("random.x", "save-output.input_value"), + connect("random.x", "offset.a"), + connect("random.x", "scale.a"), + connect("offset.x", "sum.a"), + connect("scale.x", "sum.b"), + connect("sum.x", "save-output.output_value"), + ], +) +``` + +Check for circular loops when defining connectors in the `Process`. These will need to be resolved using the `initial_values` argument to a component somewhere within the loop, e.g. + +```python +my_component = MyComponent(name="test", initial_values={"x": [False], "y": [False]}) +``` + +### Visualizing Process Flow + +You can create a mermaid diagram to help users understand their models visually. + +```python +from plugboard.diagram import markdown_diagram + +diagram = markdown_diagram(process) +print(diagram) +``` + +### Running the Model + +You can help users to run their model. For example, to run the model defined above: + +```python +async with process: + await process.run() +``` + +## Event-Driven Models + +You can help users to implement event-driven models using Plugboard's event system. Components can emit and handle events to communicate with each other. + +Examples of where you might want to use events include: +* A component that monitors a data stream and emits an event when a threshold is crossed. +* A component that listens for events and triggers actions in response, e.g. sending an alert. +* A trading algorithm that uses events to signal buy/sell decisions. +* Where a model has conditional workflows, e.g. process data differently in the model depending on a specific condition. + +Events must be defined by inheriting from the `plugboard.events.Event` class. Each event class should define the data it carries using a Pydantic `BaseModel`. For example: + +```python +from pydantic import BaseModel +from plugboard.events import Event + +class MyEventData(BaseModel): + some_value: int + another_value: str + +class MyEvent(Event): + data: MyEventData +``` + +Components can emit events using the `self.io.queue_event()` method or by returning them from an event handler. Event handlers are defined using methods decorated with `@EventClass.handler`. For example: + +```python +from plugboard.component import Component, IOController as IO + +class MyEventPublisher(Component): + io = IO(inputs=["some_input"], output_events=[MyEvent]) + + async def step(self) -> None: + # Emit an event + event_data = MyEventData(some_value=42, another_value=f"received {self.some_input}") + self.io.queue_event(MyEvent(source=self.name, data=event_data)) + +class MyEventSubscriber(Component): + io = IO(input_events=[MyEvent], output_events=[MyEvent]) + + @MyEvent.handler + async def handle_my_event(self, event: MyEvent) -> MyEvent: + # Handle the event + print(f"Received event: {event.data}") + output_event_data = MyEventData(some_value=event.data.some_value + 1, another_value="handled") + return MyEvent(source=self.name, data=output_event_data) +``` + +To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using a `ConnectorBuilder`. For example: + +```python +from plugboard.connector import AsyncioConnector, ConnectorBuilder +from plugboard.process import LocalProcess + +# Define components.... +component_1 = ... +component_2 = ... + +# Define connectors for non-event components as before +connect = lambda in_, out_: AsyncioConnector(spec=ConnectorSpec(source=in_, target=out_)) +connectors = [ + connect("component_1.output", "component_2.input"), + ... +] +# Define connectors for events +event_connectors = AsyncioConnector.builder().build_event_connectors(components) + +process = LocalProcess( + components=[ + component_1, component_2, ... + ], + connectors=connectors + event_connectors, +) +``` + +## Exporting Models + +Save a process configuration for reuse: + +```python +process.dump("my-model.yaml") + +Later, load and run via CLI +```sh +plugboard process run my-model.yaml +``` + +## Jupyter Notebooks + +Use the following guidelines when creating demo notebooks: + +1. **Structure** + - Title markdown cell in the same format as the other notebooks, including badges to run on Github/Colab + - Clear markdown sections + - Code cells with explanations + - Visualizations of results + - Summary of findings + +2. **Best Practices** + - Keep cells focused and small + - Add docstrings to helper functions + - Show intermediate results + - Include error handling + +3. **Output** + - Clear cell output before committing + - Generate plots where helpful + - Provide interpretation of results + +## Resources + +- **Library Components**: `plugboard.library` +- **Component Base Class**: `plugboard.component.Component` +- **Process Types**: `plugboard.process` +- **Event System**: `plugboard.events` +- **API Documentation**: https://docs.plugboard.dev