DPDK环境搭建
时间：2022-09-15 03:00:00
1、DPDK
2.环境配套信息
VMware Workstation 15 Pro Ubuntu 18.04.1 DPDK 19.08.2 3、VMware配置
加两张网卡：
配置网卡支持多队列：
处理器数量需要大于1
修改虚拟机配置
配置文件修改：
ethernet0.virtualDev = "e1000" 改为： ethernet0.virtualDev = "vmxnet3"确认配置生效：
4、DPDK源码下载
5.准备系统环境
- 依靠工具安装
sudo apt install build-essential sudo apt-get install libnuma-dev- 配置大页内存，打开immon功能
sudo vim /etc/default/grub 在GRUB_CMDLINE_LINUX末尾追加：hugepages=512 intel_iommu=on iommu=pt GRUB_CMDLINE_LINUX="find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US hugepages=512 intel_iommu=on iommu=pt"- 重生引导配置
sudo grub-mkconfig -o /boot/grub/grub.cfg- 重启并验证
cat /proc/meminfo |grep -i HugePages dmesg | grep DMAR6、编译DPDK
- 设置环境变量
cd dpdk-stable-19.08.2 export RTE_SDK=`pwd` export RTE_TARGET=x86_64-default-linuxapp-gcc - 运行dpdk-setup.sh编译
sunny@ubuntu:~/dpdk-stable-19.08.2$ cd usertools/ sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ./dpdk-setup.sh ------------------------------------------------------------------------------ RTE_SDK exported as /home/sunny/dpdk-stable-19.08.2 ------------------------------------------------------------------------------ ---------------------------------------------------------- Step 1: Select the DPDK environment to build ---------------------------------------------------------- [1] arm64-armada-linuxapp-gcc [2] arm64-armada-linux-gcc [3] arm64-armv8a-linuxapp-clang [4] arm64-armv8a-linuxapp-gcc [5] arm64-armv8a-linux-clang [6] arm64-armv8a-linux-gcc [7] arm64-bluefield-linuxapp-gcc [8] arm64-bluefield-linux-gcc [9] arm64-dpaa2-linuxapp-gcc [10] arm64-dpaa2-linux-gcc [11] arm64-dpaa-linuxapp-gcc [12] arm64-dpaa-linux-gcc [13] arm64-octeontx2-linuxapp-gcc [14] arm64-octeontx2-linux-gcc [15] arm64-stingray-linuxapp-gcc [16] arm64-stingray-linux-gcc [17] arm64-thunderx2-linuxapp-gcc [18] arm64-thunderx2-linux-gcc [19] arm64-thunderx-linuxapp-gcc [20] arm64-thunderx-linux-gcc [21] arm64-xgene1-linuxapp-gcc [22] arm64-xgene1-linux-gcc [23] arm-armv7a-linuxapp-gcc [24] arm-armv7a-linux-gcc [25] i686-native-linuxapp-gcc [26] i686-native-linuxapp-icc [27] i686-native-linux-gcc [28] i686-native-linux-icc [29] ppc_64-power8-linuxapp-gcc [30] ppc_64-power8-linux-gcc [31] x86_64-native-bsdapp-clang [32] x86_64-native-bsdapp-gcc [33] x86_64-native-freebsd-clang [34] x86_64-native-freebsd-gcc [35] x86_64-native-linuxapp-clang [36] x86_64-native-linuxapp-gcc [37] x86_64-native-linuxapp-icc [38] x86_64-native-linux-clang [39] x86_64-native-linux-gcc [40] x86_64-native-linux-icc [41] x86_x32-native-linuxapp-gcc [42] x86_x32-native-linux-gcc ---------------------------------------------------------- Step 2: Setup linux environment ---------------------------------------------------------- [43] Insert IGB UIO module [44] Insert VFIO module [45] Insert KNI module [46] Setup hugepage mappings for non-NUMA systems [47] Setup hugepage mappings for NUMA systems [48] Display current Ethernet/Baseband/Crypto device settings [49] Bind Ethernet/Baseband/Crypto device to IGB UIO module [50] Bind Ethernet/Baseband/Crypto device to VFIO module [51] Setup VFIO permissions ---------------------------------------------------------- Step 3: Run test application for linux environment ---------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
----------------------------------------------------------
Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo
----------------------------------------------------------
Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings
[60] Exit Script
Option:
- 版本选择，编译36
[36] x86_64-native-linuxapp-gcc
Option: 36
Configuration done using x86_64-native-linuxapp-gcc
== Build lib
== Build lib/librte_kvargs
== Build lib/librte_eal
/home/sunny/dpdk-stable-19.08.2/mk/toolchain/gcc/rte.toolchain-compat.mk:30: AVX512 support disabled because of binutils 2.30. See Bug 97
== Build lib/librte_eal/common
== Build lib/librte_eal/linux
== Build lib/librte_eal/linux/eal
CC eal_memory.o
CC eal_thread.o
CC eal_log.o
CC eal_vfio.o
CC eal_vfio_mp_sync.o
CC eal_memalloc.o
CC eal_debug.o
CC eal_lcore.o
CC eal_timer.o- 加载驱动43
[43] Insert IGB UIO module
Option: 43
Unloading any existing DPDK UIO module
Loading uio module
Loading DPDK UIO module- 配置大页内存46
[46] Setup hugepage mappings for non-NUMA systems
Option: 46
Removing currently reserved hugepages
Unmounting /mnt/huge and removing directory
Input the number of 2048kB hugepages
Example: to have 128MB of hugepages available in a 2MB huge page system,
enter '64' to reserve 64 * 2MB pages
Number of pages: 512
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs- 查看网卡状态48，0000:03:00.0即网卡PCI地址
[48] Display current Ethernet/Baseband/Crypto device settings
Option: 48
Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3,vfio-pci
Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci *Active*
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================- 绑定网卡49
需要先将网卡down:
sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ifconfig
ens33: flags=4163 mtu 1500
inet 192.168.0.160 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::7074:2a70:8ce9:db20 prefixlen 64 scopeid 0x20
ether 00:0c:29:71:e5:61 txqueuelen 1000 (Ethernet)
RX packets 1122 bytes 148489 (148.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 202 bytes 16795 (16.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens160: flags=4163 mtu 1500
inet 192.168.0.144 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::d784:b54e:4161:626c prefixlen 64 scopeid 0x20
ether 00:0c:29:71:e5:57 txqueuelen 1000 (Ethernet)
RX packets 68354 bytes 100105014 (100.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 25950 bytes 1935742 (1.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 403 bytes 35535 (35.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 403 bytes 35535 (35.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ifconfig ens160 down
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
Option: 49
Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens160 drv=vmxnet3 unused=igb_uio,vfio-pci
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
Enter PCI address of device to bind to IGB UIO driver: 03:00.0
OK- 运行测试程序53
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
Option: 53
Enter hex bitmask of cores to execute testpmd app on
Example: to execute app on cores 0 to 7, enter 0xff
bitmask: 7
Launching app
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:01.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 15ad:7b0 net_vmxnet3
Interactive-mode selected
testpmd: create a new mbuf pool : n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 00:0C:29:71:E5:57
Checking link statuses...
Done
testpmd> start
io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 335 RX-dropped: 0 RX-total: 335
TX-packets: 335 TX-dropped: 0 TX-total: 335
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 335 RX-dropped: 0 RX-total: 335
TX-packets: 335 TX-dropped: 0 TX-total: 335
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done. - 查看大页内存使用情况
sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ cat /proc/meminfo |grep -i HugePages
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 512
HugePages_Free: 254
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
- 环境搭建完成，60退出
[60] Exit Script
Option: 60- 以上过程中可能会出现各种错误，需要逐步分析解决
7、使用DPDK Kit
- 编译完成Kit位置
sunny@ubuntu:~/dpdk-stable-19.08.2$ pwd
/home/sunny/dpdk-stable-19.08.2
sunny@ubuntu:~/dpdk-stable-19.08.2$ ls
app config drivers kernel MAINTAINERS meson_options.txt usertools
arm64-armv8a-linux-clang devtools examples lib Makefile mk VERSION
buildtools doc GNUmakefile license meson.build README x86_64-native-linuxapp-gcc
sunny@ubuntu:~/dpdk-stable-19.08.2$ cd x86_64-native-linuxapp-gcc/
sunny@ubuntu:~/dpdk-stable-19.08.2/x86_64-native-linuxapp-gcc$ ls
app build include kmod lib Makefile
- Kit组件
sunny@ubuntu:~/dpdk-stable-19.08.2/x86_64-native-linuxapp-gcc$ tree -d
.
├── app
├── build
│ ├── app
│ │ ├── pdump
│ │ ├── proc-info
│ │ ├── test
│ │ ├── test-acl
│ │ ├── test-bbdev
│ │ ├── test-cmdline
│ │ ├── test-compress-perf
│ │ ├── test-crypto-perf
│ │ ├── test-eventdev
│ │ ├── test-pipeline
│ │ └── test-pmd
│ ├── buildtools
│ │ └── pmdinfogen
│ ├── drivers
│ │ ├── baseband
│ │ │ ├── fpga_lte_fec
│ │ │ ├── null
│ │ │ └── turbo_sw
│ │ ├── bus
│ │ │ ├── dpaa
│ │ │ │ └── base
│ │ │ │ ├── fman
│ │ │ │ └── qbman
│ │ │ ├── fslmc
│ │ │ │ ├── mc
│ │ │ │ ├── portal
│ │ │ │ └── qbman
│ │ │ ├── ifpga
│ │ │ ├── pci
│ │ │ │ └── linux
│ │ │ ├── vdev
│ │ │ └── vmbus
│ │ │ └── linux
│ │ ├── common
│ │ │ ├── cpt
│ │ │ ├── dpaax
│ │ │ ├── octeontx
│ │ │ ├── octeontx2
│ │ │ └── qat
│ │ ├── compress
│ │ │ └── octeontx
│ │ ├── crypto
│ │ │ ├── caam_jr
│ │ │ ├── dpaa2_sec
│ │ │ │ └── mc
│ │ │ ├── dpaa_sec
│ │ │ ├── null
│ │ │ ├── octeontx
│ │ │ ├── scheduler
│ │ │ └── virtio
│ │ ├── event
│ │ │ ├── dpaa
│ │ │ ├── dpaa2
│ │ │ ├── dsw
│ │ │ ├── octeontx
│ │ │ ├── octeontx2
│ │ │ ├── opdl
│ │ │ ├── skeleton
│ │ │ └── sw
│ │ ├── mempool
│ │ │ ├── bucket
│ │ │ ├── dpaa
│ │ │ ├── dpaa2
│ │ │ ├── octeontx
│ │ │ ├── octeontx2
│ │ │ ├── ring
│ │ │ └── stack
│ │ ├── net
│ │ │ ├── af_packet
│ │ │ ├── ark
│ │ │ ├── atlantic
│ │ │ ├── avp
│ │ │ ├── axgbe
│ │ │ ├── bnxt
│ │ │ ├── bonding
│ │ │ ├── cxgbe
│ │ │ ├── dpaa
│ │ │ ├── dpaa2
│ │ │ │ ├── base
│ │ │ │ └── mc
│ │ │ ├── e1000
│ │ │ ├── ena
│ │ │ ├── enetc
│ │ │ ├── enic
│ │ │ │ └── base
│ │ │ ├── failsafe
│ │ │ ├── fm10k
│ │ │ ├── hinic
│ │ │ ├── i40e
│ │ │ ├── iavf
│ │ │ ├── ice
│ │ │ ├── ifc
│ │ │ ├── ipn3ke
│ │ │ ├── ixgbe
│ │ │ ├── kni
│ │ │ ├── liquidio
│ │ │ ├── memif
│ │ │ ├── netvsc
│ │ │ ├── nfp
│ │ │ ├── null
│ │ │ ├── octeontx
│ │ │ ├── octeontx2
│ │ │ ├── qede
│ │ │ ├── ring
│ │ │ ├── sfc
│ │ │ ├── softnic
│ │ │ ├── tap
│ │ │ ├── thunderx
│ │ │ ├── vdev_netvsc
│ │ │ ├── vhost
│ │ │ ├── virtio
│ │ │ │ └── virtio_user
│ │ │ └── vmxnet3
│ │ └── raw
│ │ ├── dpaa2_cmdif
│ │ ├── dpaa2_qdma
│ │ ├── ifpga
│ │ ├── ioat
│ │ ├── ntb
│ │ ├── octeontx2_dma
│ │ └── skeleton
│ ├── kernel
│ │ └── linux
│ │ ├── igb_uio
│ │ └── kni
│ └── lib
│ ├── librte_acl
│ ├── librte_bbdev
│ ├── librte_bitratestats
│ ├── librte_bpf
│ ├── librte_cfgfile
│ ├── librte_cmdline
│ ├── librte_compressdev
│ ├── librte_cryptodev
│ ├── librte_distributor
│ ├── librte_eal
│ │ ├── common
│ │ └── linux
│ │ └── eal
│ ├── librte_efd
│ ├── librte_ethdev
│ ├── librte_eventdev
│ ├── librte_flow_classify
│ ├── librte_gro
│ ├── librte_gso
│ ├── librte_hash
│ ├── librte_ip_frag
│ ├── librte_ipsec
│ ├── librte_jobstats
│ ├── librte_kni
│ ├── librte_kvargs
│ ├── librte_latencystats
│ ├── librte_lpm
│ ├── librte_mbuf
│ ├── librte_member
│ ├── librte_mempool
│ ├── librte_meter
│ ├── librte_metrics
│ ├── librte_net
│ ├── librte_pci
│ ├── librte_pdump
│ ├── librte_pipeline
│ ├── librte_port
│ ├── librte_power
│ ├── librte_rawdev
│ ├── librte_rcu
│ ├── librte_reorder
│ ├── librte_ring
│ ├── librte_sched
│ ├── librte_security
│ ├── librte_stack
│ ├── librte_table
│ ├── librte_timer
│ └── librte_vhost
├── include
│ └── generic
├── kmod
└── lib
181 directories
