Netmap does not start on RHEL7 after boot

[DirkHoffmann]

In the context of #528 I observed that the netmap drivers (netmap.ko and the nm-adapted NIC device drivers) are not loaded after boot. A simple
rmmod ixgbe modprobe ixgbe
loads the modified ixgbe driver (for example) and the netmap driver as a consequence.

I conclude from my investigations that the (standard) drivers are loaded "too early" at boottime, when the physical disks are not yet mounted, but the kernel is still running on the ram image (which contains only the standard system drivers). I failed to twiddle the SystemD startup sequence in order to delay the network driver loading to a later point. (Actually the may be needed to boot diskless systems, which rely on the system disk to be accessible via NFS for example.)

My solution was to create a SystemD service *netmap.service", which simply reloads the ixgbe module and restarts the network.service then, in order to bring all interfaces, which were using these drivers, up again.

1. Did anybody find a better solution?
2. Are you interested in the recipe/sources via git?
3. Is there a command like lsnmdrivers which would properly list those NIC drivers, which can use netmap for a given system/distribution? (Maybe with an option to list only drivers presently in use or "all" (-a)?

[vmaffione]

We have a package for Arch, here https://aur.archlinux.org/packages/netmap/
You can configure netmap to build and install the modified modules with a custom suffix, so that is clear which modules are the original ones, and which ones are the patched ones (e.g., https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=netmap#n78).
At that point you can blacklist the original ones and load the patched ones, e.g., as explained here https://aur.archlinux.org/cgit/aur.git/tree/netmap.install?h=netmap

[DirkHoffmann]

Thank you for your answer. But I think you missed the point.

The problem is not to tell Linux to load the updated modules instead of the standard ones. For RHEL this aspect is perfectly managed with the different directories under /lib/modules/$(uname -r), namely kernel/drivers/net/ethernet/ for the system drivers and updates/drivers/net/ethernet for those installed by netmap (updates supercedes kernel.), as long as I load them (manually) after the boot process and rmmod.

I think Arch also uses an initial ramdisk during boot. This cannot be (easily) modified and contains the standard network drivers, which is my problem. So during boottime, the standard drivers are loaded, because the root filesystem is not yet visible to the kernel (or any other process).

If I understand correctly the explanation for netmap on ArchLinux that you referenced, the problem on Arch is exactly the same. The netmap versions must be loaded manually or be defined/blacklisted in /etc/modprobe.d/netmap.conf — which is again on the root filesystem and not visible during boot. I suppose that you have to rmmod/modprobe once the NIC driver in order to get the netmap version after reboot. You seem to state the opposite. Can you please crosscheck and confirm that you see the netmap drivers right after boot on Arch? (Sorry to insist, but this is what I observe.) I am afraid the blacklisting with /etc/modules.d only works for drivers, which are not yet loaded from the initramfs. Otherwise they must be unloaded and reloaded (rmmod/modprobe), and I did not find a built-in automatic way for the Linux kernel to do that.

[DirkHoffmann]

For demonstration / cross check on my side: I just rebooted my system after blacklisting ixgbe with /etc/modprobe.d.

$ lsmod | grep ixgb
ixgbe                 301640  0 
mdio                   13807  1 ixgbe
ptp                    19231  1 ixgbe
dca                    15130  1 ixgbe
$ cat /etc/modprobe.d/netmap.conf 
#Testing blacklisting of ixgbe at boot. =DH20180816
install ixgbe /bin/true
$ uptime
 23:36:05 up 1 min,  1 user,  load average: 1.67, 0.66, 0.24

It is blacklisted against reload though:

$ sudo modprobe ixgbe
$ lsmod | grep ixgb

(Yes, empty.)

Is it the same for you, or can you blacklist it (or any other netmap-modified NIC module) right away from boot?

[vmaffione]

Yes, Arch uses initramfs.
However, lsinitcpio on the default initramfs reports that no network driver kernel module is included. I guess this is the difference w.r.t your machines, that may include ixgbe.ko in the initramfs.

This is the blacklist file that I use:

[vmaffione@archlinux ~]$ cat /etc/modprobe.d/blacklist.conf 
blacklist e1000
blacklist virtio_net

And this is what I get after boot, with no manual operation (as I was stating)

[vmaffione@archlinux ~]$ lsmod | grep e1000
e1000_netmap          159744  0
netmap                184320  1 e1000_netmap

which means that I'm using the patched drivers.

[vmaffione]

Closing, as it is not a netmap issue.

[DirkHoffmann]

You were absolutely right. So I give the solution for RHEL7 users, in case someone else needs it.

To prevent the system from loading the ixgbe driver on ramdisk, add to boot cmdline (via /etc/default/grub for example, and remaking /boot/grub2/grub.cfg):

modprobe.blacklist=ixgbe

NB: The parameter name changed, for example it was rdblacklist in RHEL6!