Mirantis OSS Tooling Library

A library which contains various functions and classes which help to build unified OSS Tools services.

Configuration and Logging

oss_lib.config a module for finding configuration files, parsing them and validating.

Location of Configuration

oss_lib.config provides two functions, such as process_args and process_env, to find a configuration file and use it as a source of settings.

The process_args function accepts arguments from the command line at first priority. If some of them are not specified, then suitable environment variables are used, otherwise the default values are used.

The process_env gets environment variables, otherwise default values are used.

Two of these functions accept the first position argument which is used as a prefix for environment variables, e.g. if "CEAGLE" was specified, then CEAGLE_CONF will be expected as an environment variable.

The full list of supported command line arguments and environment variables:

Argument	Environment variable	Default	Description
--debug	<SERVICE>_DEBUG	false	Use DEBUG instead of INFO for logging, possible values true/false.
--config-file	<SERVICE>_CONF		Path to a YAML-configuration file.
--log-config-file	<SERVICE>_LOG_CONF		Path to a file with configuration for Python logging module

Both functions support the default location of a configuration file in case if it was not specified through --config-file or <SERVICE>_CONF. The default location can be set using the default_config_path parameter and it will be used only if this file exists.

The list of examples to understand priorities how a configuration file is choosen:

--config-file	<SERVICE>_CONF	default_config_path	Result
/etc/ceagle.yaml	/etc/config.yaml	/etc/default.yaml	/etc/ceagle.yaml
<not set>	/etc/config.yaml	/etc/default.yaml	/etc/config.yaml
<not set>	<not set>	/etc/default.yaml (exists)	/etc/default.yaml
<not set>	<not set>	/etc/default.yaml (does not exist)	<not set>

Validation and Defaults

By default oss_lib.config expects that all configuration settings pass through validation in the JSON Schema-like format. The validation_schema parameter expects a dict which populates only the properties parameter in the schema. For example, if your application expects two top defined parameters in :

SCHEMA = {
    "elasticsearch": {
        "type": "object",
        "properties": {
            "hosts": {
                "type": "array",
                "minItems": 1,
                "uniqueItems": True,
                "items": {
                   "type": "object",
                   "properties": {
                       "host": {"type": "string"},
                       "port": {"type": "integer"},
                   },
                   "required": ["host"],
                   "additionalProperties": False,
                },
            },
        },
        "required": ["hosts"],
        "additionalProperties": False,
    },
    "config": {
        "type": "object",
        "properties": {
            "run_every_minutes": {
                "type": "integer",
                "minimum": 1,
            },
        },
        "required": ["run_every_minutes"],
        "additionalProperties": False,
    },
}

config.process_env(...,
                   validation_schema=SCHEMA,
                   ....)

The default values for settings can be also specified through the defaults parameter, e.g.:

DEFAULTS = {
    "elasticsearch": {"hosts": [
        {"host": "127.0.0.1", "port": 9200},
    ]},
    "config": {"run_every_minutes": 2},
}

config.process_env(...,
                   validation_schema=SCHEMA,
                   defaults=DEFAULTS,
                   ....)

If defaults are specified, then they will be used as settings and loaded settings from specified configuration files will be merged into them. For example, if the configuration file contains:

elasticsearch:
  hosts:
    - host: 172.16.169.4
      port: 9200

The resulting config will look like that:

{
    "elasticsearch": {
        "hosts": [
            {"host": "172.16.169.4", "port": 9200},
        ],
    },
    "config": {"run_every_minutes": 2},
}

It means that only dictionary values are merged but primitives are just replaced.

Usage Examples

After initialization of configuration oss_lib.config module provides a single tone object to interect with configuration settings. This object can be accessed through the oss_lib.config.CONF variables in a dict-like way.

Let's take a look on the example how to initialize configuration accepting command line arguments and environment variables example.py:

from oss_lib import config

SCHEMA = {
    "driver": {"enum": ["noop", "openstack"]},
}

DEFAULTS = {
    "driver": "noop",
}

config.process_args("CEAGLE",
                    default_config_path="/etc/default.yaml",
                    validation_schema=SCHEMA,
                    defaults=DEFAULTS)
print(config.CONF["driver"])

So, after that you can run your application in various ways using:

1. The command line argument --config-file:

echo "driver: openstack" > /etc/ceagle.yaml
python example.py --config-file /etc/ceagle.yaml  #-> openstack

1. The environment variable CEAGLE_CONF:

CEAGLE_CONF=/etc/ceagle.yaml
echo "driver: openstack" > $CEAGLE_CONF
python example.py #-> openstack

1. Or without any variables because the default_config_path parameter was specified:

echo "driver: openstack" > /etc/default.yaml
python example.py #-> openstack

1. Or even you can specify nothing because the defaults parameter was set:

python example.py #-> noop

Useful Stuff for Flask

routing.py

Routing stuff like auto-generated HTML and JSON pages with map of routes. This is useful for development process and for exposing APIs.

Example:

from oss_lib import routing
...

app = Flask(...)
...
app.add_url_rule(...)  # add routes
...

# Now add routing map pages
app = routing.add_routing_map(app,
                              html_uri="/api.html",
                              json_uri="/api.json")

Now run the application and find auto-generated pages on given URIs /api.html and /api.json