Experiments with Small Devices

[antoinecarme]

ml2cpp, as it is now, allows converting any ML model to a C++ code for inference purposes.

This code is however not yet optimized to run on small devices : these devices may have low CPU speed, low memory, low-power, no FPU, no real OS. ml2cpp Jupyter notebooks were run and inference was tested on a sparc64 or x86-64 server with gigabytes of memory and running full-featured g++ compiler.

We need:

1. get some hardware/micro-controller : STM32, ESP32 and K210 (riscv-64) at least.
2. Check that the generated code can be compiled with the respective gcc versions (C++-17 support, exceptions, RTTI are not always available).
3. check the capacities of these devices (CPU + memory)
4. Check floating point issues : is there an FPU? half-precision floats (float16), bfloat16, ...
5. get an emulator for each device (qemu-???). use it for automated tests.

Priority 1 : Getting at least one of these devices running all these steps (feasibility). choose the one with less constraints (K210 can run a small linux)

The goal is to be able to run the model on the bare metal. No arduino. No MicroPython.

Using STM32, ESP32 or K210 is here only for tuning purposes. Their only added value is to provide a test envoironment so that C++ code will be incrementally adapted in a real-world case.

[antoinecarme]

STM32 + ESP32 + K210 (from left to right)

[antoinecarme]

https://jaycarlson.net/embedded-linux/

[antoinecarme]

SiFive, a pioneer of the RISC-V free and open source instruction set architecture (ISA), has announced the fruit of its work with ArchiTek Corporation: AiOnIc, an edge AI processor that combines SiFive's E3-series RISC-V cores with ArchiTek's Intelligence Pixel Engine (aIPE).

To be tested (when available) :

https://www.hackster.io/news/sifive-architek-announce-high-efficiency-sub-1w-edge-ai-processor-the-aionic-1d8aa537f77f

[antoinecarme]

Kendryte K510 Edge AI Chip as a Triple-Core RISC-V Part with 3 TOPS NPU. Has FP16 support

To be tested (when available) :

https://canaan.io/product/kendryte-k510

[antoinecarme]

Alibaba On The Bleeding Edge Of RISC-V With XT910. Has FP16 support.

To be tested (when available) :

https://www.nextplatform.com/2020/08/21/alibaba-on-the-bleeding-edge-of-risc-v-with-xt910/

[antoinecarme]

https://www.mdpi.com/1999-5903/14/12/363

[antoinecarme]

https://stackoverflow.com/questions/5710942/c-on-small-footprint-microcontrollers

https://www.open-std.org/jtc1/sc22/wg21/docs/TR18015.pdf