Release v1.3: new DAG solver, better plotting & "sideffect"

.gitignore

@@ -52,3 +52,8 @@ docs/_build/
 
 # PyBuilder
 target/
+
+# Plots genersated when running sample code
+/*.png
+/*.svg
+/*.pdf

.travis.yml

@@ -2,19 +2,35 @@ language: python
 
 python:
   - "2.7"
-  - "3.4"
   - "3.5"
+  - "3.6"
+  - "3.7"
+
+addons:
+  apt:
+    packages:
+      - graphviz
+
 
 install:
-  - pip install Sphinx sphinx_rtd_theme codecov packaging
+  - pip install Sphinx codecov packaging
   - "python -c $'import os, packaging.version as version\\nv = version.parse(os.environ.get(\"TRAVIS_TAG\", \"1.0\")).public\\nwith open(\"VERSION\", \"w\") as f: f.write(v)'"
-  - python setup.py install
+  - pip install -e .[test]
   - cd docs
   - make clean html
   - cd ..
 
 script:
-  - python setup.py nosetests --with-coverage --cover-package=graphkit
+  # OVERRIDE pytest-defaults adopted in `setup.cfg`:
+  #
+  # Run doctests in latest Python; certainly not < PY3.6 due to unstable dicts.
+  # Also give `-m 'slow or not slow'` since `not slow` adopted in `setup.cfg`.
+  - |
+    if [[ "$TRAVIS_PYTHON_VERSION" = '3.7' ]]; then
+      pytest --cov=graphkit -m 'slow or not slow'
+    else
+      pytest --cov=graphkit test/ 
+    fi
 
 deploy:
   provider: pypi

CHANGES.rst

@@ -0,0 +1,208 @@
+#########
+Changelog
+#########
+
+v1.3.0 (Oct 2019): New DAG solver, better plotting & "sideffect"
+================================================================
+
+Kept external API (hopefully) the same, but revamped pruning algorithm and
+refactored network compute/compile structure, so results may change; significantly
+enhanced plotting.  The only new feature actually is the :class:`sideffect`` modifier.
+
+Network:
+--------
+
++ FIX(:gh:`18`, :gh:`26`, :gh:`29`, :gh:`17`, :gh:`20`): Revamped DAG SOLVER
+  to fix bad pruning described in :gh:`24` & :gh:`25`
+
+  Pruning now works by breaking incoming provide-links to any given
+  intermedediate inputs dropping operations with partial inputs or without outputs.
+
+  The end result is that operations in the graph that do not have all inputs satisfied,
+  they are skipped (in v1.2.4 they crashed).
+
+  Also started annotating edges with optional/sideffects, to make proper use of
+  the underlying ``networkx`` graph.
+
+  |v130-flowchart|
+
++ REFACT(:gh:`21`, :gh:`29`): Refactored Network and introduced :class:`ExecutionPlan` to keep
+  compilation results (the old ``steps`` list, plus input/output names).
+
+  Moved also the check for when to evict a value, from running the execution-plan,
+  to whenbuilding it; thus, execute methods don't need outputs anymore.
+
++ ENH(:gh:`26`): "Pin* input values that may be overriten by calculated ones.
+
+  This required the introduction of the new :class:`PinInstruction` in
+  the execution plan.
+
++ FIX(:gh:`23`, :gh:`22`-2.4.3): Keep consistent order of ``networkx.DiGraph``
+  and *sets*, to generate deterministic solutions.
+
+  *Unfortunately*, it non-determinism has not been fixed in < PY3.5, just
+  reduced the frequency of `spurious failures
+  <https://travis-ci.org/yahoo/graphkit/builds/594729787>`_, caused by
+  unstable dicts, and the use of subgraphs.
+
++ enh: Mark outputs produced by :class:`NetworkOperation`'s needs as ``optional``.
+  TODO: subgraph network-operations would not be fully functional until
+  *"optional outpus"* are dealt with (see :gh:`22`-2.5).
+
++ enh: Annotate operation exceptions with ``ExecutionPlan`` to aid debug sessions,
+
++ drop: methods ``list_layers()``/``show layers()`` not needed, ``repr()`` is
+  a better replacement.
+
+
+Plotting:
+---------
+
++ ENH(:gh:`13`, :gh:`26`, :gh:`29`): Now network remembers last plan and uses that
+  to overlay graphs with the internals of the planing and execution: |sample-plot|
+
+
+    - execution-steps & order
+    - delete & pin instructions
+    - given inputs & asked outputs
+    - solution values (just if they are present)
+    - "optional" needs & broken links during pruning
+
++ REFACT: Move all API doc on plotting in a single module, splitted in 2 phases,
+  build DOT & render DOT
+
++ FIX(:gh:`13`): bring plot writing into files up-to-date from PY2; do not create plot-file
+  if given file-extension is not supported.
+
++ FEAT: path `pydot library <https://pypi.org/project/pydot/>`_ to support rendering
+  in *Jupyter notebooks*.
+
+
+
+Testing & other code:
+---------------------
+
+ - Increased coverage from 77% --> 90%.
+
++ ENH(:gh:`28`): use ``pytest``, to facilitate TCs parametrization.
+
++ ENH(:gh:`30`): Doctest all code; enabled many assertions that were just print-outs
+  in v1.2.4.
+
++ FIX: ``operation.__repr__()`` was crashing when not all arguments
+  had been set - a condition frequtnly met during debugging session or failed
+  TCs (inspired by @syamajala's 309338340).
+
++ enh: Sped up parallel/multihtread TCs by reducing delays & repetitions.
+
+  .. tip::
+    You need ``pytest -m slow`` to run those slow tests.
+
+
+
+Chore & Docs:
+-------------
+
++ FEAT: add changelog in ``CHANGES.rst`` file, containing  flowcharts
+  to compare versions ``v1.2.4 <--> v1.3..0``.
++ enh: updated site & documentation for all new features, comparing with v1.2.4.
++ enh(:gh:`30`): added "API reference' chapter.
++ drop(build): ``sphinx_rtd_theme`` library is the default theme for Sphinx now.
++ enh(build): Add ``test`` *pip extras*.
++ sound: https://www.youtube.com/watch?v=-527VazA4IQ,
+  https://www.youtube.com/watch?v=8J182LRi8sU&t=43s
+
+
+
+v1.2.4 (Mar 7, 2018)
+====================
+
++ Issues in pruning algorithm: :gh:`24`, :gh:`25`
++ Blocking bug in plotting code for Python-3.x.
++ Test-cases without assertions (just prints).
+
+|v124-flowchart|
+
+
+
+1.2.2 (Mar 7, 2018, @huyng): Fixed versioning
+=============================================
+
+Versioning now is manually specified to avoid bug where the version
+was not being correctly reflected on pip install deployments
+
+
+
+1.2.1 (Feb 23, 2018, @huyng): Fixed multi-threading bug and faster compute through caching of `find_necessary_steps`
+====================================================================================================================
+
+We've introduced a cache to avoid computing find_necessary_steps multiple times
+during each inference call.
+
+This has 2 benefits:
+
++ It reduces computation time of the compute call
++ It avoids a subtle multi-threading bug in networkx when accessing the graph
+  from a high number of threads.
+
+
+
+1.2.0 (Feb 13, 2018, @huyng)
+============================
+
+Added `set_execution_method('parallel')` for execution of graphs in parallel.
+
+
+1.1.0 (Nov 9, 2017, @huyng)
+===========================
+
+Update setup.py
+
+
+1.0.4 (Nov 3, 2017, @huyng): Networkx 2.0 compatibility
+=======================================================
+
+Minor Bug Fixes:
+
++ Compatibility fix for networkx 2.0
++ `net.times` now only stores timing info from the most recent run
+
+
+1.0.3 (Jan 31, 2017, @huyng): Make plotting dependencies optional
+=================================================================
+
++ Merge pull request :gh:`6` from yahoo/plot-optional
++ make plotting dependencies optional
+
+
+1.0.2 (Sep 29, 2016, @pumpikano): Merge pull request :gh:`5` from yahoo/remove-packaging-dep
+============================================================================================
+
++ Remove 'packaging' as dependency
+
+
+1.0.1 (Aug 24, 2016)
+====================
+
+1.0 (Aug 2, 2016, @robwhess)
+============================
+
+First public release in PyPi & GitHub.
+
++ Merge pull request :gh:`3` from robwhess/travis-build
++ Travis build
+
+
+.. _substitutions:
+
+
+.. |sample-plot| image:: docs/source/images/sample_plot.svg
+    :alt: sample graphkit plot
+    :width: 120px
+    :align: bottom
+.. |v130-flowchart| image:: docs/source/images/GraphkitFlowchart-v1.3.0.svg
+    :alt: graphkit-v1.3.0 flowchart
+    :scale: 75%
+.. |v124-flowchart| image:: docs/source/images/GraphkitFlowchart-v1.2.4.svg
+    :alt: graphkit-v1.2.4 flowchart
+    :scale: 75%

README.md

@@ -1,56 +1,91 @@
 # GraphKit
 
-[![PyPI version](https://badge.fury.io/py/graphkit.svg)](https://badge.fury.io/py/graphkit) [![Build Status](https://travis-ci.org/yahoo/graphkit.svg?branch=master)](https://travis-ci.org/yahoo/graphkit) [![codecov](https://codecov.io/gh/yahoo/graphkit/branch/master/graph/badge.svg)](https://codecov.io/gh/yahoo/graphkit)
+[![Latest version in PyPI](https://img.shields.io/pypi/v/graphkit.svg?label=PyPi%20version)](https://img.shields.io/pypi/v/graphkit.svg?label=PyPi%20version)
+[![Latest version in GitHub](https://img.shields.io/github/v/release/yahoo/graphkit.svg?label=GitHub%20release&include_prereleases)](https://img.shields.io/github/v/release/yahoo/graphkit.svg?label=GitHub%20release&include_prereleases)
+[![Supported Python versions of latest release in PyPi](https://img.shields.io/pypi/pyversions/graphkit.svg?label=Python)](https://img.shields.io/pypi/pyversions/graphkit.svg?label=Python)
+[![Build Status](https://travis-ci.org/yahoo/graphkit.svg?branch=master)](https://travis-ci.org/yahoo/graphkit)
+[![codecov](https://codecov.io/gh/yahoo/graphkit/branch/master/graph/badge.svg)](https://codecov.io/gh/yahoo/graphkit)
+[![License](https://img.shields.io/pypi/l/graphkit.svg)](https://img.shields.io/pypi/l/graphkit.svg)
+
+[![Github watchers](https://img.shields.io/github/watchers/yahoo/graphkit.svg?style=social)](https://img.shields.io/github/watchers/yahoo/graphkit.svg?style=social)
+[![Github stargazers](https://img.shields.io/github/stars/yahoo/graphkit.svg?style=social)](https://img.shields.io/github/stars/yahoo/graphkit.svg?style=social)
+[![Github forks](https://img.shields.io/github/forks/yahoo/graphkit.svg?style=social)](https://img.shields.io/github/forks/yahoo/graphkit.svg?style=social)
+[![Issues count](http://img.shields.io/github/issues/yahoo/graphkit.svg?style=social)](http://img.shields.io/github/issues/yahoo/graphkit.svg?style=social)
 
 [Full Documentation](https://pythonhosted.org/graphkit/)
 
 > It's a DAG all the way down
 
+![Sample graph](docs/source/images/test_pruning_not_overrides_given_intermediate-asked.png "Sample graph")
+
 ## Lightweight computation graphs for Python
 
-GraphKit is a lightweight Python module for creating and running ordered graphs of computations, where the nodes of the graph correspond to computational operations, and the edges correspond to output --> input dependencies between those operations.  Such graphs are useful in computer vision, machine learning, and many other domains.
+GraphKit is an an understandable and lightweight Python module for building and running
+ordered graphs of computations.
+The API posits a fair compromise between features and complexity without precluding any.
+It might be of use in computer vision, machine learning and other data science domains,
+or become the core of a custom ETL pipelne.
 
 ## Quick start
 
 Here's how to install:
 
-```
-pip install graphkit
-```    
+    pip install graphkit
 
-Here's a Python script with an example GraphKit computation graph that produces multiple outputs (`a * b`, `a - a * b`, and `abs(a - a * b) ** 3`):
+OR with dependencies for plotting support (and you need to install [`Graphviz`](https://graphviz.org)
+program separately with your OS tools):
 
-```
-from operator import mul, sub
-from graphkit import compose, operation
+    pip install graphkit[plot]
 
-# Computes |a|^p.
-def abspow(a, p):
-    c = abs(a) ** p
-    return c
+Here's a Python script with an example GraphKit computation graph that produces
+multiple outputs (`a * b`, `a - a * b`, and `abs(a - a * b) ** 3`):
 
-# Compose the mul, sub, and abspow operations into a computation graph.
-graph = compose(name="graph")(
-    operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
-    operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
-    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
-)
+    >>> from operator import mul, sub
+    >>> from graphkit import compose, operation
 
-# Run the graph and request all of the outputs.
-out = graph({'a': 2, 'b': 5})
+    >>> # Computes |a|^p.
+    >>> def abspow(a, p):
+    ...     c = abs(a) ** p
+    ...     return c
 
-# Prints "{'a': 2, 'a_minus_ab': -8, 'b': 5, 'ab': 10, 'abs_a_minus_ab_cubed': 512}".
-print(out)
+    >>> # Compose the mul, sub, and abspow operations into a computation graph.
+    >>> graphop = compose(name="graphop")(
+    ...     operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
+    ...     operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
+    ...     operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
+    ... )
 
-# Run the graph and request a subset of the outputs.
-out = graph({'a': 2, 'b': 5}, outputs=["a_minus_ab"])
+    >>> # Run the graph and request all of the outputs.
+    >>> out = graphop({'a': 2, 'b': 5})
+    >>> print(out)
+    {'a': 2, 'b': 5, 'ab': 10, 'a_minus_ab': -8, 'abs_a_minus_ab_cubed': 512}
 
-# Prints "{'a_minus_ab': -8}".
-print(out)
-```
+    >>> # Run the graph and request a subset of the outputs.
+    >>> out = graphop({'a': 2, 'b': 5}, outputs=["a_minus_ab"])
+    >>> print(out)
+    {'a_minus_ab': -8}
 
+
 As you can see, any function can be used as an operation in GraphKit, even ones imported from system modules!
 
+
+## Plotting
+
+For debugging the above graph-operation you may plot the *execution plan*
+of the last computation it using these methods:
+
+```python
+graphop.plot(show=True)                # open a matplotlib window
+graphop.plot("intro.svg")              # other supported formats: png, jpg, pdf, ...
+graphop.plot()                         # without arguments return a pydot.DOT object
+graphop.plot(solution=out)             # annotate graph with solution values
+```
+
+![Intro graph](docs/source/images/intro.svg "Intro graph")
+![Graphkit Legend](docs/source/images/GraphkitLegend.svg "Graphkit Legend")
+
+> **TIP:** The `pydot.Dot` instances returned by `plot()` are rendered as SVG in *Jupyter/IPython*.
+
 # License
 
 Code licensed under the Apache License, Version 2.0 license. See LICENSE file for terms.

bin/build.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+# clean, or invalid files in packages
+rm -vrf ./build/* ./dist/* ./*.pyc ./*.tgz ./*.egg-info
+python setup.py sdist bdist_wheel
+