From 4e38eef817813c1bbd8a9cf972e4cf0134d24308 Mon Sep 17 00:00:00 2001 From: Alex Williams Date: Sun, 1 Dec 2019 21:58:13 -0800 Subject: dpdk: Add new DPDK stack to integrate with I/O services docs: Update DPDK docs with new parameters: Parameter names have had their hyphens changed to underscores, and the I/O CPU argument is now named after the lcores and reflects the naming used by DPDK. transport: Add new udp_dpdk_link, based atop the new APIs: This link is tightly coupled with the DPDK I/O service. The link class carries all the address information to communicate with the other host, and it can send packets directly through the DPDK NIC ports. However, for receiving packets, the I/O service must pull the packets from the DMA queue and attach them to the appropriate link object. The link object merely formats the frame_buff object underneath, which is embedded in the rte_mbuf container. For get_recv_buff, the link will pull buffers only from its internal queue (the one filled by the I/O service). transport: Add DPDK-specific I/O service: The I/O service is split into two parts, the user threads and the I/O worker threads. The user threads submit requests through various appropriate queues, and the I/O threads perform all the I/O on their behalf. This includes routing UDP packets to the correct receiver and getting the MAC address of a destination (by performing the ARP request and handling the ARP replies). The DPDK context stores I/O services. The context spawns all I/O services on init(), and I/O services can be fetched from the dpdk_ctx object by using a port ID. I/O service clients: The clients have two lockless ring buffers. One is to get a buffer from the I/O service; the other is to release a buffer back to the I/O service. Threads sleeping on buffer I/O are kept in a separate list from the service queue and are processed in the course of doing RX or TX. The list nodes are embedded in the dpdk_io_if, and the head of the list is on the dpdk_io_service. The I/O service will transfer the embedded wait_req to the list if it cannot acquire the mutex to complete the condition for waking. Co-authored-by: Martin Braun Co-authored-by: Ciro Nishiguchi Co-authored-by: Brent Stapleton --- host/docs/dpdk.dox | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) (limited to 'host/docs') diff --git a/host/docs/dpdk.dox b/host/docs/dpdk.dox index e840fc235..3f71231a5 100644 --- a/host/docs/dpdk.dox +++ b/host/docs/dpdk.dox @@ -75,7 +75,23 @@ load that driver with the following command: modprobe vfio-pci For NICs that require vfio-pci (like Intel's X520), you'll want to use the -`dpdk-devbind.py` script to the vfio-pci driver. +`dpdk-devbind.py` script to the vfio-pci driver. This script is shipped with +DPDK and installed to `$prefix/share/dpdk/usertools`. If the NIC uses the +vfio-pci driver, and the package was installed apt-get, then a typical invocation +might be + + /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci ens6f0 + +If successful, the script might provide an updated status like this: + + /usr/share/dpdk/usertools/dpdk-devbind.py -s + + Network devices using DPDK-compatible driver + ============================================ + 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe + [...] + + See https://doc.dpdk.org/guides-18.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules for more details. @@ -102,7 +118,7 @@ options: ;instead and swap between them [use_dpdk=1] ;dpdk_mtu is the NIC's MTU setting - ;This is separate from MPM's maximum packet size--tops out at 4000 + ;This is separate from MPM's maximum packet size dpdk_mtu=9000 ;dpdk_driver is the -d flag for the DPDK EAL. If DPDK doesn't pick up the driver for your NIC ;automatically, you may need this argument to point it to the folder where it can find the drivers @@ -115,8 +131,8 @@ options: ;dpdk_num_mbufs is the total number of packet buffers allocated ;to each direction's packet buffer pool ;This will be multiplied by the number of NICs, but NICs on the same - ;CPU socket share a pool - dpdk_num_mbufs=512 + ;CPU socket share a pool. + dpdk_num_mbufs=4096 ;dpdk_mbuf_cache_size is the number of buffers to cache for a CPU ;The cache reduces the interaction with the global pool dpdk_mbuf_cache_size=64 @@ -127,20 +143,24 @@ address, and it must be in a particular format. Hex digits must all be lower case, and octets must be separated by colons. Here is an example: [dpdk_mac=3c:fd:fe:a2:a9:09] - ;dpdk_io_cpu selects the CPU that this NIC's driver will run on - ;Multiple NICs may occupy one CPU, but the I/O thread will completely - ;consume that CPU. Also, 0 is reserved for the master thread (i.e. + ;dpdk_lcore selects the lcore that this NIC's driver will run on + ;Multiple NICs may occupy one lcore, but the I/O thread will completely + ;consume that lcore's CPU. Also, 0 is reserved for the master thread (i.e. ;the initial UHD thread that calls init() for DPDK). Attempting to ;use it as an I/O thread will only result in hanging. - dpdk_io_cpu = 1 + ;Note also that by default, the lcore ID will be the same as the CPU ID. + dpdk_lcore = 1 ;dpdk_ipv4 specifies the IPv4 address, and both the address and ;subnet mask are required (and in this format!). DPDK uses the ;netmask to create a basic routing table. Routing to other networks ;(i.e. via gateways) is not permitted. dpdk_ipv4 = 192.168.10.1/24 + ;dpdk_num_desc is the number of descriptors in each DMA ring. + ;Must be a power of 2. + dpdk_num_desc=4096 [dpdk_mac=3c:fd:fe:a2:a9:0a] - dpdk_io_cpu = 1 + dpdk_lcore = 1 dpdk_ipv4 = 192.168.20.1/24 \section dpdk_using Using DPDK in UHD -- cgit v1.2.3