LeMon is a framework based on Linux's eBPF subsystem, and is designed for monitoring and collecting statistics across the network stack of Linux-based Access Points.
It's major features include:
- a novel architecture designed to efficiently share data between user and kernel space with minimal processor overhead
- safe concurrent access of data from both kernel and user space without the use of any built-in concurrency mechanisms (such as spinlocks or per-CPU maps)
- a design that can accomodate devices with extremely small amounts of RAM and even fit all recorded data inside the L1 or L2 cache of CPUs within Commercial-Off-The-Shelf Wi-Fi routers
- accurate recording of desired data even when the frequency of generated data is extremely high (see Performance below).
- able to be retrofitted to record almost any data from the Linux network stack without kernel modification
The current design is built around the use of the XDP hook within the Linux kernel to record packet metadata early in the RX path of the kernel. Currently, it analyzes each packet and stores aggregated records for the network flow that each packet belongs to. The storage of each record, as well as the notification of data availability by the kernel, is done using eBPF maps. These are standardized data storage mechanisms that are available as long as the kernel version is new enough, which helps make the system both portable and reusable.
Some of LeMon's performance results so far (when used for network flow monitoring) are displayed in the following table:
Measured Throughput for a Linux-based AP running LeMon1
| Number of Concurrent Flows2 | Transport Protocol Used | Average End-to-End Throughput3 | Percentage of Throughput Drop4 |
|---|---|---|---|
| 10 | TCP | 1.90 Gbps | 2.26% |
| 50 | UDP | 2.07 Gbps | 12.16% |
| 100 | TCP | 1.78 Gbps | 3.19% |
| 400 | UDP | 1.94 Gbps | 13.24% |
Coming soon!
Thanks so much for your interest in trying out LeMon!
As a basic requirement, your system's Linux kernel version must be at least v6.1 or higher. If it is not, LeMon will fail to load properly due to the absence of neccesary eBPF map types in the kernel.
To get started, make sure that the following dependencies are installed on your system:
- Linux kernel headers for your specific kernel version (installation method varies by Linux distro, but generally the following rules should apply):
- For Debian-based systems, you should be able to install your headers by running
sudo apt install linux-headers-$(uname -r) - For distributions based on Fedora, RHEL or CentOS, you'll likely need to install both the development headers and the user-space headers by running either:
sudo dnf install kernel-devel && sudo dnf install kernel-headersORsudo yum install kernel-devel && sudo yum install kernel-headers
- For Arch-based distros:
sudo pacman -S linux-headers && pacman -Q linux linux-headers
- For other distributions, you'll need to look up the specific instructions on how to install your kernel headers. However, this is (usually) a well-documented process and hopefully you should be able to install the headers without too much trouble.
- For Debian-based systems, you should be able to install your headers by running
- libbpf >= v1.5.0. We've include an installation script that should hopefully help if your distribition doesn't offer a libbpf version this recent.
- make
- additionally, you should be familiar enough with Makefiles to change the compilation target to that of your specific system's archicture. We plan to make it less architecture-dependent in the future, but currently the Makefile statically builds for AArch64.
- clang/LLVM
- doxygen (optional; used to build the accompanying documentation for the code)
You can try using our experimental install script to install the neccessary dependencies by running chmod +x install && sudo ./install if you like!
Please build the docs (make docs) and make sure you thoroughly understand the code and change the macro values within the headers to point to the proper network interfaces and allocate the correct sizes for LeMon's eBPF maps.
Afterwards, to build and run LeMon, simply enter the following command: make LeMon && sudo ./LeMon
The current version of LeMon simply retrieves a hash from the kernel-to-user-ring-buffer, submits the hash to the user-to-kernel-ring-buffer, accesses the index of the entry and resubmits the hash to the kernel. It prints out when a collision has occured that could not be resolved by user-space, as well as when a hash is delivered. However, we plan to expand it's capabilities and attempt to offer multiple examples/applications to better demonstrate it's versitility and usefulness.
Additionally, please be aware that this framework was built and tested using the following equipment: AP: Banana Pi R4 Wi-Fi chipset: BPI-R4-NIC-BE14 (MediaTek MT7995AV, Wi-Fi 7) CPU: Mediatek Filogic 880, Quad-Core ARM Cortex-A73 (1.8GHz) RAM: 4GB (DDR4)
The above hardware was running a custom version of the Ubuntu Linux distribution (Linux kernel version 6.12.32) built for the Banana Pi R4 which can be found here. You'll need to build a custom kernel with XDP enabled, along with every type of queuing discipline (qdisc) and TCP congestion control (CC) algorithm to thoroughly test the system.
We use a 10G SFP+ (DAC) via the LAN port to connect with an "uplink server", and the beforementioned Wi-Fi 7 card to connect with the 802.11 station, both of which were one hop away from the AP. The AP was hosted using hostapd v2.12-devel on the 6GHz band in an interferance-free environment, with the receiver using an Intel BE200 Wi-Fi 7 card to connect to the AP.
Also, you'll need to ensure power-save mode is off on both wireless interfaces (AP and station) to ensure optimal performance. All of our results so far were gathered with a downlink configuration (wired-to-wireless), with the wired server using FQ-CoDel as it's qdisc.
The AP also uses FQ-CoDel for it's wired qdisc.
We choose to use TCP Cubic on both sender and receiver, given that it is the current default TCP CC algorithm for the Linux kernel and is often utilized for many popular servers.
eBPF Foundation's eBee logo is a registered trademark of eBPF Foundation. You can learn more details about the eBPF Foundation here.
1: Measured by taking the average end-to-end throughput for each 1-second interval recorded over a continous, 120 second iPerf2 test. Specific testing methodology will be given at some point in the next couple commits.
2: Concurrent flows using the parallel streams option within iPerf2. Note that each stream was explictly given it's own source and destination port to ensure fair results for both Linux and LeMon, as LeMon uses the network five-tuple as part of it's recording process.
3: As measured by iPerf2.
4: Compared to Linux doing packet switching with no monitoring processes active.