Feature/element data classes#602
Draft
FBumann wants to merge 403 commits intofeature/batched-modelingfrom
Draft
Conversation
FlowsData (batched.py):
1. Added categorizations: with_flow_hours, with_load_factor
2. Renamed: size_minimum → effective_size_lower, size_maximum → effective_size_upper
3. Properties now only include relevant flows (no NaN padding):
- flow_hours_minimum/maximum → only with_flow_hours
- flow_hours_minimum/maximum_over_periods → only with_flow_hours_over_periods
- load_factor_minimum/maximum → only with_load_factor
4. Added absolute_lower_bounds, absolute_upper_bounds for all flows
5. Added _stack_values_for_subset() helper
FlowsModel (elements.py):
1. Removed hours and hours_over_periods variables - not needed
2. Simplified constraints to compute inline:
- constraint_flow_hours() - directly constrains sum_temporal(rate)
- constraint_flow_hours_over_periods() - directly constrains weighted sum
- constraint_load_factor_min/max() - compute hours inline
3. rate variable uses self.data.absolute_lower_bounds/upper_bounds directly
4. Removed obsolete bound collection methods
Benefits:
- Cleaner separation: data in FlowsData, constraints in FlowsModel
- No NaN handling needed - properties only include relevant flows
- Fewer variables in the model
- More explicit about which flows have which constraints
1. Added Status Data Properties to FlowsData (batched.py) Added new cached properties for status-related bounds: - min_uptime, max_uptime - uptime bounds for flows with uptime tracking - min_downtime, max_downtime - downtime bounds for flows with downtime tracking - startup_limit_values - startup limits for flows with startup limit - previous_uptime, previous_downtime - computed previous durations using StatusHelpers.compute_previous_duration() 2. Simplified FlowsModel Variable Creation (elements.py) Refactored uptime, downtime, and startup_count methods to use the new FlowsData properties instead of inline computation: - uptime: Now uses self.data.min_uptime, self.data.max_uptime, self.data.previous_uptime - downtime: Now uses self.data.min_downtime, self.data.max_downtime, self.data.previous_downtime - startup_count: Now uses self.data.startup_limit_values 3. Kept active_hours (Plan Adjustment) The original plan called for removing active_hours, but functional tests (test_on_total_max, test_on_total_bounds) demonstrated that active_hours is required to enforce active_hours_min and active_hours_max parameters. Without it, the optimizer would ignore those constraints. Verification All tests pass: - pytest tests/test_functional.py -v - 26 tests passed - pytest tests/test_flow.py -v -k "time_only" - 22 tests passed - pytest tests/test_component.py -v -k "time_only" - 9 tests passed
1. Combined min/max pairs - Created _uptime_bounds and _downtime_bounds cached properties that compute both min and max in a single iteration - Individual properties (min_uptime, max_uptime, etc.) now delegate to these cached tuples 2. Added helper methods - _build_status_bounds(flow_ids, min_attr, max_attr) - builds both bounds in one pass - _build_previous_durations(flow_ids, target_state, min_attr) - consolidates previous duration logic 3. Used more efficient patterns - Pre-allocated numpy arrays (np.empty, np.full) instead of Python list appends - Cached dict lookups - params = self.status_params at loop start instead of repeated self.status_params[fid] - Reduced redundant iterations - accessing min/max uptime now only iterates once instead of twice
- status_effects_per_startup → effects_per_startup
1. compute_previous_duration - Simple helper for computing previous duration (used by FlowsData)
2. add_batched_duration_tracking - Creates duration tracking constraints (used by FlowsModel)
3. create_status_features - Used by ComponentsModel (separate code path, not part of FlowsModel refactoring)
Removed:
- collect_status_effects - replaced with simpler _build_status_effects helper directly in FlowsData
The effect building is now consistent - effects_per_active_hour and effects_per_startup use the same pattern as effects_per_flow_hour:
# Simple, direct approach - no intermediate dict
def _build_status_effects(self, attr: str) -> xr.DataArray | None:
flow_factors = [
xr.concat(
[xr.DataArray(getattr(params[fid], attr).get(eff, np.nan)) for eff in effect_ids],
dim='effect',
coords='minimal',
).assign_coords(effect=effect_ids)
for fid in flow_ids
]
return concat_with_coords(flow_factors, 'flow', flow_ids)
…oup of elements:
StatusData provides:
Categorizations:
- with_startup_tracking - IDs needing startup/shutdown tracking
- with_downtime_tracking - IDs needing downtime tracking
- with_uptime_tracking - IDs needing uptime duration tracking
- with_startup_limit - IDs with startup limit
- with_effects_per_active_hour - IDs with effects_per_active_hour
- with_effects_per_startup - IDs with effects_per_startup
Bounds (computed in single pass):
- min_uptime, max_uptime - uptime bounds
- min_downtime, max_downtime - downtime bounds
- startup_limit - startup limit values
Previous Durations:
- previous_uptime, previous_downtime - computed from previous states
Effects:
- effects_per_active_hour, effects_per_startup - effect factor arrays
FlowsData now delegates to StatusData:
@cached_property
def _status_data(self) -> StatusData | None:
if not self.with_status:
return None
return StatusData(
params=self.status_params,
dim_name='flow',
effect_ids=list(self._fs.effects.keys()),
timestep_duration=self._fs.timestep_duration,
previous_states=self.previous_states,
)
@Property
def min_uptime(self) -> xr.DataArray | None:
return self._status_data.min_uptime if self._status_data else None
This class can now be reused by ComponentsModel for component-level status as well.
1. Removed InvestmentEffectsMixin from FlowsModel - FlowsModel now inherits only from TypeModel
2. Replaced mixin interface properties with direct delegation - The effect properties now delegate to self.data._investment_data:
- effects_per_size
- effects_of_investment
- effects_of_retirement
- effects_of_investment_mandatory
- effects_of_retirement_constant
3. Updated imports - Removed unused InvestmentEffectsMixin import from elements.py
Note: StoragesModel in components.py still uses InvestmentEffectsMixin. If you want consistency, we can update it similarly to use an InvestmentData instance. That would allow removing InvestmentEffectsMixin from features.py entirely.
1. Updated FlowsModel (elements.py):
- Removed InvestmentEffectsMixin inheritance
- Added direct property delegation to self.data._investment_data
2. Updated StoragesModel (components.py):
- Removed InvestmentEffectsMixin inheritance
- Added _investment_data cached property that creates an InvestmentData instance
- Added direct property delegation for all effect properties
3. Removed InvestmentEffectsMixin (features.py):
- Deleted the entire mixin class (~105 lines) since it's no longer used
Architecture after changes:
- InvestmentData (in batched.py) is the single source for batched investment data
- Both FlowsModel and StoragesModel delegate to InvestmentData for effect properties
- No more mixin inheritance - simpler, more explicit code
…ame matches "Bus1" inside "Bus10". Changed to element_id in con_name.split('|') for delimiter-aware exact matching.
2. Lines 487-493 — Boolean mask becomes float. mask.reindex() with NaN fill turns booleans to float. Added fill_value=False to the reindex call and mask = mask.astype(bool) after expand_dims to keep the dtype boolean.
…red element names but didn't reset the model's _is_built flag, so get_status() would still report MODEL_BUILT. Added fs.model._is_built = False when fs.model is not None.
…taArray() (a scalar with no dims) when empty, breaking downstream .dims checks. Changed to xr.DataArray(dims=['case'], coords={'case': []}) so the 'case' dimension is always present.
- PiecewiseBuilder.create_piecewise_constraints — removed the zero_point parameter entirely. It now always creates sum(inside_piece) <= 1. - Callers (FlowsModel and StoragesModel) — add the tighter <= invested constraint separately, only for optional IDs that exist in invested_var. No coord mismatch possible. - ConvertersModel — was already passing None, just cleaned up the dead code.
…erseded/math/ directory. Here's a summary of the changes made:
Summary of Updated Tests
test_flow.py (88 tests)
- Updated variable names: flow|rate, flow|size, flow|invested, flow|status, flow|active_hours
- Updated constraint names: share|temporal(costs), share|periodic(costs) instead of 'ComponentName->effect(temporal)'
- Updated uptime/downtime constraints: flow|uptime|forward, flow|uptime|backward, flow|uptime|min instead of flow|uptime|fwd/bwd/lb
- Updated switch constraints: flow|switch_transition instead of flow|switch
- Removed non-existent flow|fixed constraint check (fixed profile uses flow|invest_lb/ub)
test_storage.py (48 tests)
- Updated variable names: storage|charge, storage|netto, storage|size, storage|invested
- Updated constraint names: storage|balance, storage|netto_eq, storage|initial_equals_final
- Updated status variable from status|status to flow|status
- Updated prevent simultaneous constraint: storage|prevent_simultaneous
- Fixed effects_of_investment syntax to use dict: {'costs': 100}
test_component.py (40 tests)
- Updated status variables: component|status, flow|status instead of status|status
- Updated active_hours variables: component|active_hours
- Updated uptime variables: component|uptime
- Updated constraints: component|status|lb/ub/eq, component|uptime|initial
- Removed non-existent flow|total_flow_hours checks
test_linear_converter.py (36 tests)
- Updated constraint names: converter|conversion (no index suffix)
- Updated status variables: component|status, component|active_hours
- Updated share constraints: share|temporal(costs)
- Made piecewise tests more flexible with pattern matching
test_effect.py (26 tests)
- No changes needed - tests already working
1. Storage charge state scalar bounds (batched.py): Added .astype(float) after expand_dims().copy() to prevent silent int→float truncation when assigning final charge state overrides (0.5 was being truncated to 0 on an int64 array). 2. SourceAndSink deserialization (components.py): Convert inputs/outputs from dict to list before + concatenation in __init__, fixing TypeError: unsupported operand type(s) for +: 'dict' and 'dict' during NetCDF save/reload. 3. Legacy config leaking between test modules (test_math/conftest.py, superseded/math/conftest.py, test_legacy_solution_access.py): Converted module-level fx.CONFIG.Legacy.solution_access = True to autouse fixtures that restore the original value after each test, preventing the plotting isinstance(solution, xr.Dataset) test from failing.
Member
Author
Test Comparison:
|
| main | feature | |
|---|---|---|
| Passed | 1638 | 1383 |
| Skipped | 3 | 99 |
| Deselected | 6 | 0 |
| Failed | 0 | 0 |
| Warnings | 216 | 698 |
| Total collected | 1647 | 1482 |
| Time | 19m 08s | 5m 05s |
Test Count Difference (-165 total)
The difference is entirely due to test file restructuring — no coverage was lost:
- 22 files removed from
tests/deprecated/(596 tests on main) - 11 files added as
tests/superseded/math/+ new test files (431 tests) - Net: -165 tests from reorganization
Skipped (3 → 99)
The +96 skipped tests are the tsam-dependent superseded math tests (importorskip('tsam')).
Warnings (216 → 698)
- ResourceWarning: 39 → 7 (fewer because deprecated
test_results_plots.pywith plotly socket warnings was removed) - DeprecationWarning: increased because
test_math/tests use theoptimizefixture parametrized 3 ways, tripling legacy solution access warnings. All expected.
Performance
Feature branch is ~4x faster (5m vs 19m) — the removed deprecated tests were slow functional tests.
CI Failures (not related to this branch's changes)
- Tests (3.11/3.12/3.13):
test_full_scenario_optimizationfails with gurobi license error ("Model too large for size-limited license") — infrastructure issue, not a code bug - Docs build:
tsam==3.0.0was yanked upstream; dependency constrainttsam>=3.0.0,<3.1.0needs updating to include3.1.0
* Here's a summary of everything that was done: Summary Phase 1: TransmissionsData - Added flow_ids parameter to TransmissionsData.__init__ - Moved 12 cached properties from TransmissionsModel to TransmissionsData: bidirectional_ids, balanced_ids, _build_flow_mask(), in1_mask, out1_mask, in2_mask, out2_mask, relative_losses, absolute_losses, has_absolute_losses_mask, transmissions_with_abs_losses - Updated TransmissionsModel.create_constraints() to use self.data.* - Updated BatchedAccessor.transmissions to pass flow_ids Phase 2: BusesData - Added balance_coefficients cached property to BusesData - Updated BusesModel.create_constraints() to use self.data.balance_coefficients Phase 3: ConvertersData - Added flow_ids and timesteps parameters to ConvertersData.__init__ - Moved 13 cached properties from ConvertersModel to ConvertersData: factor_element_ids, max_equations, equation_mask, signed_coefficients, n_equations_per_converter, piecewise_element_ids, piecewise_segment_counts_dict, piecewise_max_segments, piecewise_segment_mask, piecewise_flow_breakpoints, piecewise_segment_counts_array, piecewise_breakpoints - Updated ConvertersModel methods to use self.data.* - Removed unused defaultdict and stack_along_dim imports from elements.py Phase 4: ComponentsData - Added flows_data, effect_ids, timestep_duration parameters to ComponentsData.__init__ - Moved 6 cached properties from ComponentsModel to ComponentsData: with_prevent_simultaneous, status_params, previous_status_dict, status_data, flow_mask, flow_count - Moved _get_previous_status_for_component() helper to ComponentsData - Updated ComponentsModel to use self.data.* throughout Bug Fix - Discovered a stale cache issue: FlowsData.previous_states could be cached before all previous_flow_rate values were set (e.g., in from_old_results). Fixed by having ComponentsData._get_previous_status_for_component() compute previous status directly from flow attributes instead of going through the potentially-stale FlowsData.previous_states cache. * 1. _build_flow_mask exposure: Added balanced_in1_mask and balanced_in2_mask cached properties to TransmissionsData. TransmissionsModel now uses these instead of calling the private d._build_flow_mask(). 2. EffectsModel/elements: Confirmed not a bug — EffectsModel does not inherit from TypeModel and never accesses .elements on its data, so EffectsData not having elements is fine. 3. FlowsData.dim_name: Changed from @cached_property to @Property to match all other Data classes.
* Capture the solver log in an extra sub logger, which propagates by default to the regular logger.
* What's implemented
- stream_solver_log() in io.py: tails a log file in a background thread, forwards lines to logging.getLogger('flixopt.solver') at INFO level
- _disable_solver_console(): overrides solver console options (log_to_console/LogToConsole)
- capture_solver_log config setting (default False, enabled by presets)
- progress=False when capturing — eliminates tqdm progress bar noise
Known limitation (documented in docstring)
Gurobi may print a few lines (license banner, LP reading, parameter setting) directly to stdout before LogToConsole=0 takes effect. This is a Gurobi/linopy limitation. HiGHS
handles it cleanly.
* Improve setting log to console False
* Dont mutate when capturing, instead do depending on config.py
* Update CHANGELOG.md
* Revert test changes
* Updated the docstring for capture_solver_log in config.py to:
- Explain the setting is independent of log_to_console
- Document the double-logging scenario clearly (capture + native console + console logger = double output)
- Provide three example configurations showing how to avoid it:
- File capture only: capture=True, log_to_console=False, enable_file()
- Logger to console: capture=True, log_to_console=False, enable_console()
- Native console only: capture=False, log_to_console=True
The user is in full control — if they enable both, they get both. The docs make it obvious how to avoid double logging.
* Added a truncation at line 1553 — when the user provides a log_fn that already exists, it's emptied before the tail thread starts. The cleanup=True (temp file) path is
unchanged since mkstemp already creates a fresh empty file.
* Imrpove defaults in configs
* - New log_fn parameter (pathlib.Path | str | None, default None) — lets users persist the solver log to a specific file
* Fix test
* Improve docs
* Add progress parameter to solve/optimize for linopy pass through
* Add progress and make CONFIG.Solving.log_to_console purely about the solver log itself
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
….1 (#604) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…gin to v1.5.1 (#603) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…eractions (#608) * Add more tests for combinations * 10 Test Classes, 20 Tests (x3 modes = 60 runs) ┌──────────────────────────────────────┬───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Class │ Tests │ What it covers │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestPiecewiseWithInvestment │ 2 │ Piecewise conversion + invest sizing; piecewise invest cost + optional skip │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestPiecewiseWithStatus │ 4 │ Non-1:1 piecewise + startup cost; gap-based min load + status; no zero point + status off; no zero point + startup cost │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestPiecewiseThreeSegments │ 3 │ 3-segment piecewise at high/low/mid load │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestStatusWithEffects │ 2 │ Startup cost on CO2 effect; active-hour cost on multiple effects │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestInvestWithRelativeMinimum │ 1 │ Invest sizing with relative_minimum forcing off-state (asserts status[0]<0.5) │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestConversionWithTimeVaryingEffects │ 2 │ Time-varying effects_per_flow_hour; dual-output CHP with time-varying costs │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestPiecewiseInvestWithStatus │ 1 │ Piecewise invest (economy of scale in seg2) + startup costs │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestStatusWithMultipleConstraints │ 2 │ startup_limit + max_downtime; min_uptime + min_downtime │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestEffectsWithConversion │ 2 │ Effect share_from_temporal + investment; effect maximum_total + status contribution │ ├──────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ TestInvestWithEffects │ 1 │ invest_per_size on non-cost (CO2) effect with maximum_total constraint │ └──────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Sensitivity highlights Every test has been validated to produce a different result if the feature it tests were removed: - Piecewise tests use non-1:1 ratios (e.g., fuel [30→60] / heat [30→50]) so removing piecewise changes the answer - Status tests verify that startup costs / off-states contribute uniquely to the objective - The test_invest_sizing_respects_relative_minimum now directly asserts status[0] < 0.5 (boiler OFF at t=0) and produces cost=125 vs 80 without relative_minimum - The "no zero point" tests cover the critical interaction where status must allow OFF despite no [0,0] piece in the piecewise definition * minor fixes
…classes # Conflicts: # CHANGELOG.md # tests/deprecated/test_config.py
- StatusData.with_effects_per_active_hour and StatusData.with_effects_per_startup — categorization lists that were never accessed. The actual effect values (effects_per_active_hour, effects_per_startup) are applied correctly via a different path. Removed passthrough properties (components.py — StoragesModel) 9 properties eliminated, replaced with direct self.data.X at all call sites: - with_investment, with_optional_investment, with_mandatory_investment - storages_with_investment, storages_with_optional_investment - optional_investment_ids, mandatory_investment_ids - invest_params, _investment_data Kept investment_ids — it has 4 external callers in optimization.py. Removed passthrough property (elements.py — FlowsModel) - _previous_status → replaced 3 call sites with self.data.previous_states
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Major refactoring of the model building pipeline to use batched/vectorized operations instead of per-element loops. This brings significant performance improvements, especially for large models.
Key Changes
Batched Type-Level Models: New
FlowsModel,StoragesModel,BusesModelclasses that handle ALL elements of a type in single batched operations instead of individualFlowModel,StorageModelinstances.FlowsData/StoragesData Classes: Pre-compute and cache element data as xarray DataArrays with element dimensions, enabling vectorized constraint creation.
Mask-based Variable Creation: Variables use linopy's
mask=parameter to handle heterogeneous elements (e.g., only some flows have status variables) while keeping consistent coordinates.Fast NumPy Helpers: Replace slow xarray methods with numpy equivalents:
fast_notnull()/fast_isnull()- ~55x faster than xarray's.notnull()/.isnull()Unified Coordinate Handling: All variables use consistent coordinate order via
.reindex()to prevent alignment errors.Performance Results
Scaling Summary
The batched approach provides 7-32x build speedup depending on model size, with the benefit growing as models get larger.
Scaling by Number of Converters
Base config: 720 timesteps, 1 period, 2 effects, 5 storages
Main scales O(n) with converters (168→1,688 vars), while the feature branch stays constant at 15 vars. Build time on main grows ~11x for 20x more converters; the feature branch grows only ~1.7x.
Scaling by Number of Effects
Base config: 720 timesteps, 1 period, 50 converters (102 flows), each flow contributes to ALL effects
The batched approach handles effect share constraints in O(1) instead of O(n_effects × n_flows). Main grows 7.5x for 20x effects; the feature branch grows only 1.7x.
Scaling by Number of Storages
Base config: 720 timesteps, 1 period, 2 effects, 50 converters
Same pattern: main scales O(n) with storages while the feature branch stays constant.
Scaling by Timesteps and Periods
Speedup remains consistent (~8-12x) regardless of time horizon or period count.
XL System End-to-End (2000h, 300 converters, 50 storages)
Model Size Reduction
The batched approach creates fewer, larger variables instead of many small ones:
Why This Matters
The old approach creates one linopy Variable per flow/storage element. Each creation has ~1ms overhead, so 200 converters × 2 flows = 400 variables = 400ms just for variable creation. Constraints are created per-element in loops.
The new approach creates one batched Variable with an element dimension. A single
flow|ratevariable contains ALL flows in one DataArray, and constraints use vectorized xarray operations with masks. Variable count stays constant regardless of model size.Type of Change
Testing
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Breaking Changes