Storing simulation setup

[JoelLucaAdams]

I maintain a package called sdf-xarray which parses .sdf files into xarray. These files are generated via a simulation and I would like the ability to add the simulation input parameters (which can be loaded from an input.deck file as a dict) from epydeck into the xarray.Dataset and xarray.DataArray objects. I'm considering adding them via attributes but was wondering if there was a better way.

I was also wondering about the ability to display the dict within a jupyter notebook as a series of dropdowns/accordions below the attributes. Alternatively is there another way to render them so that it's integrated inside of the xarray objects a bit like the way the pint_xarray accessor works (see how it renders the data here with the "magnitude" and "units")

Any help or advice on how best to approach this would be extremely greatly appreciated :)

import epydeck
import xarray as xr

ds = xr.open_dataset("0000.sdf")

with open("input.deck") as f:
    deck = epydeck.load(f)

ds.attrs["deck"] = deck

The simulation input files are called "decks" are technically Fortran files that contain multiple sections and are read in as Python dictionaries. Below is an example of one of these:

{
    "control": {
        "nx": 256,
        "ny": 256,
        "t_end": 1e-08,
        "x_min": 0,
        "x_end": 1,
        "y_min": 0,
        "y_max": 1,
        "stdout_frequency": 10,
    },
    "boundaries": {
        "bc_x_min": "simple_outflow",
        "bc_x_max": "simple_outflow",
        "bc_y_min": "periodic",
        "bc_y_max": "periodic",
    },
    "window": {
        "move_window": True,
        "window_start_time": 0,
        "bc_x_min_after_move": "simple_outflow",
        "bc_x_max_after_move": "simple_outflow",
        "window_v_x": 200000000.0,
    },
    "constant": {
      "x0": 1.31,
      "y0": 0.5,
      "r0": "0.4^2",
      "r2": "(x - x0)^2 + (y - y0)^2"
    },
    "species": {
        "electron": {
            "name": "electron",
            "charge": -1.0,
            "mass": 1.0,
            "nparticles_per_cell": 5,
            "number_density": [
                "if(abs(x-x0) lt 0.3, 2, 1)",
                "if(abs(y-y0) lt 0.2, number_density(electron), 1)",
            ],
            "temperature_ev": 0,
        }
    },
    "output": {
        "normal": {
            "name": "normal",
            "dt_snapshot": 1e-10,
            "particles": "always",
            "px": "always",
            "py": "always",
            "pz": "always",
            "vx": "always",
            "vy": "always",
            "vz": "always",
            "particle_weight": "always",
            "grid": "always",
            "number_density": "always + species",
            "temperature": "always + species",
        }
    },
}

[keewis]

Attributes should be fine (I'd create a single key like "model_parameters" on the group / dataset level), just be aware that you'll need to encode the nested dicts if you want to write to files like netcdf4 (zarr should be able to store those just fine, though).

pint_xarray is a bit special in that it has array objects that can customize how they represent themselves, while the group / dataset level attributes don't.

xarray currently does not support it, but what if we'd allow objects in the attributes to customize their HTML repr? That way, you could decode the model parameters into an object that allows querying parameters easier and has a nicer representation, you'd just have to decode after opening and encode before writing.

[JoelLucaAdams]

Thank you very much for the advice!

Adding support for objects in the attributes to customise their HTML repr would be exactly what I'm looking for. That way you could expand the different top level keys to see what each attribute is associated with a bit like the way you can open and close the attributes. Ideally if xarray could support containing collapsible dictionaries within the attributes that would be best.

I wonder additionally if there's a way to customise the main HTML repr so that a new section could be added below the attributes containing the model_parameters. I think that would be the cleanest and most accessible approach to end users

[benbovy]

In xarray-simlab, model parameters are usually represented as data variables. This has several advantages like:

• store parameter metadata (description, units, etc.) as attributes, decoupled from the parameter value
• reuse Xarray dimensions for parameters accepting array values
• reuse Xarray dimensions to represents the inputs of simulation batches with arbitrary dimensions for, e.g., sensitivity analysis, optimization, ML, etc.

Currently parameter sections are handled using a namespace prefix for the variable name, e.g., "control__nx", "control__ny", etc. but maybe using a DataTree would be a convenient alternative approach, also for having simulation inputs and outputs in separate groups/branches (xarray-simlab was developed before DataTree existed in Xarray).

[JoelLucaAdams]

Interesting... could you point me to some examples of this please? I'd be curious to see what it looks like :)

[benbovy]

There is a couple of examples in the documentation (https://xarray-simlab.readthedocs.io/en/latest/run_model.html) and in this blog post: https://medium.com/pangeo/pangeo-data-and-models-280b251ff0cd

[JoelLucaAdams]

Very interesting! Thanks for sharing and might I add a very cool project too

[benbovy]

Thanks! That repo hasn't been maintained for a while and needs some revamp, though.