docs: Update pvrdma device documentation
Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com> Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com> Signed-off-by: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
This commit is contained in:
Yuval Shaia
Marcel Apfelbaum
parent
305fd2ba06
commit
46b69a8824
126
docs/pvrdma.txt
126
docs/pvrdma.txt
|
|
@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for any special guest
|
|||
modifications.
|
||||
|
||||
While it complies with the VMware device, it can also communicate with bare
|
||||
metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
|
||||
can work with Soft-RoCE (rxe).
|
||||
metal RDMA-enabled machines as peers.
|
||||
|
||||
It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
|
||||
|
||||
It does not require the whole guest RAM to be pinned allowing memory
|
||||
over-commit and, even if not implemented yet, migration support will be
|
||||
|
|
@ -78,29 +79,116 @@ the required RDMA libraries.
|
|||
|
||||
3. Usage
|
||||
========
|
||||
|
||||
|
||||
3.1 VM Memory settings
|
||||
======================
|
||||
Currently the device is working only with memory backed RAM
|
||||
and it must be mark as "shared":
|
||||
-m 1G \
|
||||
-object memory-backend-ram,id=mb1,size=1G,share \
|
||||
-numa node,memdev=mb1 \
|
||||
|
||||
The pvrdma device is composed of two functions:
|
||||
- Function 0 is a vmxnet Ethernet Device which is redundant in Guest
|
||||
but is required to pass the ibdevice GID using its MAC.
|
||||
Examples:
|
||||
For an rxe backend using eth0 interface it will use its mac:
|
||||
-device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
|
||||
For an SRIOV VF, we take the Ethernet Interface exposed by it:
|
||||
-device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
|
||||
- Function 1 is the actual device:
|
||||
-device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>
|
||||
where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
|
||||
Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.
|
||||
The rules of conversion are part of the RoCE spec, but since manual conversion
|
||||
is not required, spotting problems is not hard:
|
||||
Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
|
||||
MAC: 7c:fe:90:cb:74:3a
|
||||
Note the difference between the first byte of the MAC and the GID.
|
||||
|
||||
3.2 MAD Multiplexer
|
||||
===================
|
||||
MAD Multiplexer is a service that exposes MAD-like interface for VMs in
|
||||
order to overcome the limitation where only single entity can register with
|
||||
MAD layer to send and receive RDMA-CM MAD packets.
|
||||
|
||||
To build rdmacm-mux run
|
||||
# make rdmacm-mux
|
||||
|
||||
The application accepts 3 command line arguments and exposes a UNIX socket
|
||||
to pass control and data to it.
|
||||
-d rdma-device-name Name of RDMA device to register with
|
||||
-s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux)
|
||||
-p rdma-device-port Port number of RDMA device to register with (default 1)
|
||||
The final UNIX socket file name is a concatenation of the 3 arguments so
|
||||
for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2
|
||||
will be created.
|
||||
|
||||
pvrdma requires this service.
|
||||
|
||||
Please refer to contrib/rdmacm-mux for more details.
|
||||
|
||||
|
||||
3.3 Service exposed by libvirt daemon
|
||||
=====================================
|
||||
The control over the RDMA device's GID table is done by updating the
|
||||
device's Ethernet function addresses.
|
||||
Usually the first GID entry is determined by the MAC address, the second by
|
||||
the first IPv6 address and the third by the IPv4 address. Other entries can
|
||||
be added by adding more IP addresses. The opposite is the same, i.e.
|
||||
whenever an address is removed, the corresponding GID entry is removed.
|
||||
The process is done by the network and RDMA stacks. Whenever an address is
|
||||
added the ib_core driver is notified and calls the device driver add_gid
|
||||
function which in turn update the device.
|
||||
To support this in pvrdma device the device hooks into the create_bind and
|
||||
destroy_bind HW commands triggered by pvrdma driver in guest.
|
||||
|
||||
Whenever changed is made to the pvrdma port's GID table a special QMP
|
||||
messages is sent to be processed by libvirt to update the address of the
|
||||
backend Ethernet device.
|
||||
|
||||
pvrdma requires that libvirt service will be up.
|
||||
|
||||
|
||||
3.4 PCI devices settings
|
||||
========================
|
||||
RoCE device exposes two functions - an Ethernet and RDMA.
|
||||
To support it, pvrdma device is composed of two PCI functions, an Ethernet
|
||||
device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The
|
||||
Ethernet function can be used for other Ethernet purposes such as IP.
|
||||
|
||||
|
||||
3.5 Device parameters
|
||||
=====================
|
||||
- netdev: Specifies the Ethernet device function name on the host for
|
||||
example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet
|
||||
device used to create it.
|
||||
- ibdev: The IB device name on host for example rxe0, mlx5_0 etc.
|
||||
- mad-chardev: The name of the MAD multiplexer char device.
|
||||
- ibport: In case of multi-port device (such as Mellanox's HCA) this
|
||||
specify the port to use. If not set 1 will be used.
|
||||
- dev-caps-max-mr-size: The maximum size of MR.
|
||||
- dev-caps-max-qp: Maximum number of QPs.
|
||||
- dev-caps-max-sge: Maximum number of SGE elements in WR.
|
||||
- dev-caps-max-cq: Maximum number of CQs.
|
||||
- dev-caps-max-mr: Maximum number of MRs.
|
||||
- dev-caps-max-pd: Maximum number of PDs.
|
||||
- dev-caps-max-ah: Maximum number of AHs.
|
||||
|
||||
Notes:
|
||||
- The first 3 parameters are mandatory settings, the rest have their
|
||||
defaults.
|
||||
- The last 8 parameters (the ones that prefixed by dev-caps) defines the top
|
||||
limits but the final values is adjusted by the backend device limitations.
|
||||
- netdev can be extracted from ibdev's sysfs
|
||||
(/sys/class/infiniband/<ibdev>/device/net/)
|
||||
|
||||
|
||||
3.6 Example
|
||||
===========
|
||||
Define bridge device with vmxnet3 network backend:
|
||||
<interface type='bridge'>
|
||||
<mac address='56:b4:44:e9:62:dc'/>
|
||||
<source bridge='bridge1'/>
|
||||
<model type='vmxnet3'/>
|
||||
<address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0' multifunction='on'/>
|
||||
</interface>
|
||||
|
||||
Define pvrdma device:
|
||||
<qemu:commandline>
|
||||
<qemu:arg value='-object'/>
|
||||
<qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/>
|
||||
<qemu:arg value='-numa'/>
|
||||
<qemu:arg value='node,memdev=mb1'/>
|
||||
<qemu:arg value='-chardev'/>
|
||||
<qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/>
|
||||
<qemu:arg value='-device'/>
|
||||
<qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/>
|
||||
</qemu:commandline>
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue