How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?

Matt asked:

I have a Supermicro 2027TR-HTFRF server that I’m using as a xen server.

What I’m wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines.

Following this recipe I managed to get the I350 Ethernet to expose the VF’s.

Adding /etc/modprobe.d/mlx4_core.conf containing

options max_vfs=4

Didn’t create the virtual functions for the mellanox device despite the fact that this is a valid parameter to the mlx4_core module. i.e.

The error I get is

mlx4_core: `4' invalid for parameter `max_vfs'

I also tried a value of 1 to which I got a similar error

mlx4_core: `1' invalid for parameter `max_vfs'

Output from lspci -v

06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    Subsystem: Mellanox Technologies Device 0051
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at df900000 (64-bit, non-prefetchable) [size=1M]
    Memory at de000000 (64-bit, prefetchable) [size=8M]
    Capabilities: [40] Power Management version 3
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [148] Device Serial Number 00-25-90-ff-ff-17-57-24
    Capabilities: [18c] #19
    Kernel modules: mlx4_core

So it would seem that SR-IOV is not available.

I then discovered this thread:
http://community.mellanox.com/thread/1073

But I’m using ubuntu 12.04.

Next I installed mstflint using apt-get.

Then I identified what type of card it is.

mstflint -d 06:00.0 q

Unfortunately mstflint crashes.

Warning: memory access to device 06:00.0 failed: Input/output error.
Warning: Fallback on IO: much slower, and unsafe if device in use.
*** buffer overflow detected ***: mstflint terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f66d91ac817]

Some of the strace output:

strace mstflint -d 06:00.0 q
...
open("/proc/bus/pci/06/00.0", O_RDWR|O_DSYNC) = 3
ioctl(3, IIOCNETDIF, 0xdf900000)        = 0
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xdf900000) = 0x7f0e0a200000
munmap(0x7f0e0a200000, 65536)           = 0
close(3)                                = 0
open("/dev/mem", O_RDWR|O_DSYNC)        = 3
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xdf900000) = 0x7f0e092f2000
munmap(0x7f0e092f2000, 65536)           = 0
close(3)                                = 0
write(2, "Warning: memory access to device"..., 136Warning: memory access to device 06:00.0 failed: Input/output error.
Warning: Fallback on IO: much slower, and unsafe if device in use.
) = 136
open("/sys/bus/pci/devices/0000:06:00.0/config", O_RDWR|O_DSYNC) = 3
pwrite(3, "375374373372", 4, 88)    = 4
pread(3, "375374373372", 4, 88)     = 4
close(3)                                = 0
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 3
writev(3, [{"*** ", 4}, {"buffer overflow detected", 24}, {" ***: ", 6}, {"mstflint", 8}, {" terminatedn", 12}], 5*** buffer overflow detected ***: mstflint terminated
) = 54
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e0a3f9000
write(3, "======= Backtrace: =========n", 29======= Backtrace: =========
) = 29
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__fortify_fail", 14}, {"+0x", 3}, {"37", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fc817", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f0e094fc817]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"+0x", 3}, {"109710", 6}, {")", 1}, {"[0x", 3}, {"7f0e094fb710", 12}, {"]n", 2}], 8/lib/x86_64-linux-gnu/libc.so.6(+0x109710)[0x7f0e094fb710]
) = 59
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"+0x", 3}, {"108b79", 6}, {")", 1}, {"[0x", 3}, {"7f0e094fab79", 12}, {"]n", 2}], 8/lib/x86_64-linux-gnu/libc.so.6(+0x108b79)[0x7f0e094fab79]
) = 59
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"_IO_default_xsputn", 18}, {"+0x", 3}, {"dd", 2}, {")", 1}, {"[0x", 3}, {"7f0e0946e13d", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(_IO_default_xsputn+0xdd)[0x7f0e0946e13d]
) = 73
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"_IO_vfprintf", 12}, {"+0x", 3}, {"1ae7", 4}, {")", 1}, {"[0x", 3}, {"7f0e0943c4a7", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1ae7)[0x7f0e0943c4a7]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__vsprintf_chk", 14}, {"+0x", 3}, {"94", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fac14", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x94)[0x7f0e094fac14]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__sprintf_chk", 13}, {"+0x", 3}, {"7d", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fab5d", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__sprintf_chk+0x7d)[0x7f0e094fab5d]
) = 68
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"40de46", 6}, {"]n", 2}], 4mstflint[0x40de46]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"40ee56", 6}, {"]n", 2}], 4mstflint[0x40ee56]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"406475", 6}, {"]n", 2}], 4mstflint[0x406475]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"408d00", 6}, {"]n", 2}], 4mstflint[0x408d00]
) = 19
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__libc_start_main", 17}, {"+0x", 3}, {"ed", 2}, {")", 1}, {"[0x", 3}, {"7f0e0941376d", 12}, {"]n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f0e0941376d]
) = 72
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"401af9", 6}, {"]n", 2}], 4mstflint[0x401af9]
) = 19
write(3, "======= Memory map: ========n", 29======= Memory map: ========
) = 29
open("/proc/self/maps", O_RDONLY)       = 4
read(4, "00400000-0041a000 r-xp 00000000 "..., 1024) = 1024
write(3, "00400000-0041a000 r-xp 00000000 "..., 102400400000-0041a000 r-xp 00000000 08:01 220                                /usr/bin/mstflint
00619000-0061a000 r--p 00019000 08:01 220                                /usr/bin/mstflint
0061a000-0061b000 rw-p 0001a000 08:01 220                                /usr/bin/mstflint
024a3000-024c4000 rw-p 00000000 00:00 0                                  [heap]
7f0e09302000-7f0e093f2000 rw-s df910000 00:05 1028                       /dev/mem
7f0e093f2000-7f0e095a7000 r-xp 00000000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e095a7000-7f0e097a6000 ---p 001b5000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097a6000-7f0e097aa000 r--p 001b4000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097aa000-7f0e097ac000 rw-p 001b8000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097ac000-7f0e097b1000 rw-p 00000000 00:00 0 
7f0e097b1000-7f0e097c6000 r-xp 00000000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1) = 1024
read(4, "n7f0e097c6000-7f0e099c5000 ---p "..., 1024) = 1024
write(3, "n7f0e097c6000-7f0e099c5000 ---p "..., 1024
7f0e097c6000-7f0e099c5000 ---p 00015000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c5000-7f0e099c6000 r--p 00014000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c6000-7f0e099c7000 rw-p 00015000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c7000-7f0e09ac2000 r-xp 00000000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09ac2000-7f0e09cc1000 ---p 000fb000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc1000-7f0e09cc2000 r--p 000fa000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc2000-7f0e09cc3000 rw-p 000fb000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc3000-7f0e09da5000 r-xp 00000000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09da5000-7f0e09fa4000 ---p 000e2000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fa4000-7f0e09fac000 ) = 1024
read(4, "r--p 000e1000 08:01 30579       "..., 1024) = 1024
write(3, "r--p 000e1000 08:01 30579       "..., 1024r--p 000e1000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fac000-7f0e09fae000 rw-p 000e9000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fae000-7f0e09fc3000 rw-p 00000000 00:00 0 
7f0e09fc3000-7f0e09fd9000 r-xp 00000000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e09fd9000-7f0e0a1d8000 ---p 00016000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1d8000-7f0e0a1d9000 r--p 00015000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1d9000-7f0e0a1da000 rw-p 00016000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1da000-7f0e0a1fc000 r-xp 00000000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7f0e0a210000-7f0e0a300000 rw-s df910000 00:03 4026533088                 /proc/bus/pci/06/00.0
7f0e0a300000-7f0e0a3f0000 rw-s df910000 00:0e 19851                      /sys/devices/pci0000:00/0000:00:03.2/0000:06:) = 1024
read(4, "00.0/resource0n7f0e0a3f0000-7f0e"..., 1024) = 570
write(3, "00.0/resource0n7f0e0a3f0000-7f0e"..., 57000.0/resource0
7f0e0a3f0000-7f0e0a3f5000 rw-p 00000000 00:00 0 
7f0e0a3f9000-7f0e0a3fc000 rw-p 00000000 00:00 0 
7f0e0a3fc000-7f0e0a3fd000 r--p 00022000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7f0e0a3fd000-7f0e0a3ff000 rw-p 00023000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7fffdd53f000-7fffdd560000 rw-p 00000000 00:00 0                          [stack]
7fffdd5ff000-7fffdd600000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
) = 570
read(4, "", 1024)                       = 0
close(4)                                = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
gettid()                                = 4122
tgkill(4122, 4122, SIGABRT)             = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)

At this point I’m somewhat stuck.


Update:
I managed to obtain a firmware update from SuperMicro which has SR-IOV enabled.

Also I needed to set options max_vfs= to options num_vfs=. It seems the named parameter is slightly different. This was enough to get it working under Ubuntu.

Installing the firmware I used this method: http://www.panticz.de/mellanox
But with the addition that I had to call insmod on each module before calling mst start. Modprobe didn’t work.

My answer:


Check the obvious stuff first:

  • In your BIOS settings, ensure that Intel VT-d and Intel I/OAT are both Enabled.
  • Try the card in a different PCI slot.
  • Contact the package maintainer to report a bug in the tool.

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.