dpdk

dpdk备忘录

by 
1
2
3
4

nic: network interface card
kni: kernel nic interface
pmd: poll mode driver
eal: environment abstraction layer

refer to:
http://vinllen.com/tun-tap/

how to run EuhatOfp

by

create three vmware guest machines for example,
ip of no.1 guest machine is 192.168.204.33
ip of no.2 guest machine is 192.168.204.34
ip of no.3 guest machine is 192.168.204.35

setup dpdk in every machine, all script and binary are in the setup directory, refer to <5. setup dpdk> in

compile ofp over dpdk in vmware

modify test/servers.cfg

1
2
3

192.168.204.33:8061
192.168.204.34:8062
192.168.204.35:8063

for every guest machine modify self ip setting in test/ofp_netwrap.cli

1
2
3

debug 0
loglevel set debug
ifconfig fp0 192.168.204.33/24

in no.1 guest machine

1
2

cd test
./release/test 0

in no.2 guest machine

1
2

cd test
./release/test 1

in no.3 guest machine

1
2

cd test
./release/test 2

then you can check the log files in test/log directory.

how to build EuhatOfp

by

EuhatOfp is a library for peer to peers communication framework through user space tcp stack, now implemented in OFP over DPDK.

1. build ofp over dpdk

compile ofp over dpdk in vmware

2. build EuhatOfp

1

git clone https://github.com/euhat/EuhatOfp.git

modify the paths of dpdk, odp-dpdk and ofp in test/Makefile.

1
2
3

cd test
make clean
make

compile ofp over dpdk in vmware

by

1. get software

1
2

git libtool automake build-essential pkg-config libssl-dev doxygen libconfig-dev libnuma-dev libpcap-dev net-tools
ln -s /usr/bin/python3 /usr/bin/python

2. compile dpdk

1

git clone http://dpdk.org/git/dpdk-stable --branch 19.11 --depth 1 dpdk-19.11

run dpdk-19.11/usertools/dpdk-setup.sh
>>> select and run "x86_64-native-linuxapp-gcc"
>>> Exit Script

3. compile odp-dpdk

1

git clone https://github.com/OpenDataPlane/odp-dpdk.git

odp-dpdk/m4/odp_dpdk.m4

1
2
3
4
5
6
7
8
9
10
11
12
13

# _ODP_DPDK_LEGACY(PATH, ACTION-IF-FOUND, ACTION-IF-NOT-FOUND)
# ------------------------------------------------------------------------
# Locate DPDK installation
AC_DEFUN([_ODP_DPDK_LEGACY], [dnl
    DPDK_CFLAGS="-isystem $1/include"
    DPDK_LIB_PATH="$1/lib"
    DPDK_LDFLAGS="-L$DPDK_LIB_PATH"
    AS_IF([test -r "$DPDK_LIB_PATH"/libdpdk.so], [dnl
        DPDK_RPATH="-Wl,-rpath,$DPDK_LIB_PATH"
        DPDK_RPATH_LT="-R$DPDK_LIB_PATH"
        DPDK_SHARED=yes],
        [test ! -r "$DPDK_LIB_PATH"/libdpdk.a], [dnl
        AC_MSG_FAILURE([Could not find DPDK])])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/bin/sh
 
DPDK_PATH=/home/eu/work/dpdk/dpdk-19.11/x86_64-native-linuxapp-gcc
 
cd odp-dpdk
./bootstrap
#--enable-debug --enable-debug-print
#--without-openssl
./configure --with-dpdk-path=${DPDK_PATH} \
                --prefix=`pwd`/installed \
                --enable-debug=full \
                --enable-helper-debug \
                --enable-debug-print \
                --enable-helper-debug-print \
                --disable-shared
make install

4. compile ofp

1

git clone https://github.com/OpenFastPath/ofp.git
1
2
3
4
5
6
7
8
9
10
11
12
13

#!/bin/sh
 
ODP_DPDK_PATH=`pwd`/odp-dpdk/installed
 
cd ofp
./bootstrap
./configure --with-odp=${ODP_DPDK_PATH} \
        --with-odp-lib=odp-dpdk \
        --disable-shared \
        --enable-debug=yes \
        --enable-sp=no \
        --prefix=`pwd`/installed
make install

5. setup dpdk

1
2
3
4

cd dpdk-19.11/x86_64-native-linuxapp-gcc/kmod
modprobe uio
insmod igb_uio.ko
ifconfig ens33 down

run dpdk-19.11/usertools/dpdk-setup.sh
>>> select and run "Setup hugepage mappings for non-NUMA systems", enter 128
>>> select and run "Bind Ethernet/Baseband/Crypto device to IGB UIO module", will display like
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci
>>>>>> enter 02:01.0
>>> Exit Script

6. test

let the vmware guest machine use NAT.

use ipconfig check ip settings in host machine. for example, my ip for "VMware Network Adapter VMnet8" is 192.168.204.1.

ofp/example/webserver/ofp.cli

1
2
3

debug 0
loglevel set debug
ifconfig fp0 192.168.204.33/24

ofp/example/webserver/www/aa.txt

1

Hello, the world!

ofp/example/webserver/run.sh

1
2

export www_dir=`pwd`/www
./webserver -i 0 -f ofp.cli

using web browser access

1

http://192.168.204.33:2048/aa.txt

ofp中的慢平面

by

默认编译OpenFastPath时慢平面是启用的，慢平面的功能之一是通过netlink方式向内核查询路由表，这是用内核公开的接口，不需要另开发内核驱动。

慢平面的一个局限，比方电脑只有一张网卡，而这块网卡已被dpdk绑定，网络包就不会在内核中被解析，此时慢平面的路由功能也就无效，所以干脆通过

1

./configure ... --enable-sp=no

将慢平面功能去掉。

而此时我们跑在ofp上的应用就没有路由表了么？
不是的。首先应用的本机ip地址是通过

1

myApp -i 0 -f ofp.cli

中的ofp.cli配置的。如

1
2
3

debug 0
loglevel set debug
ifconfig fp0 192.168.56.33/24

中的192.168.56.33就是本机地址。
而通过arp协议获取其它机器ip与mac关系的路由表是由

1

	ofp_start_cli_thread(instance, app_init_params.linux_core_id, params.cli_file);

这个线程在dpdk绑定的网卡上进行解析维护。

一个很好的例子就是ofp/example/webserver。

ofp和dpdk都可完全编译成静态库运行，不要为了LD_PRELOAD的那一点便利一门心思去编译出动态库。在自已的源码中加一层宏翻译，静态库与epoll api兼容性也很好。

github上的odp与dpdk-odp的区别是前者现在只包含了odp-linux，一种对linux系统api的包装，后者是对dpdk的包装，所以高性能选后者。原话见
doc/users-guide/users-guide.adoc

refer to:
https://github.com/OpenFastPath/ofp
https://github.com/OpenDataPlane/odp-dpdk
https://my.oschina.net/u/4361425/blog/3269917

odp禁止ipsec

by 
1
2
3
4
5
6
7
8
9
10
11
12
13

	odp_init_t init_param;
 
	odp_init_param_init(&init_param);
	init_param.not_used.feat.ipsec = 1;
	init_param.not_used.feat.crypto = 1;
	init_param.not_used.feat.compress = 1;
	init_param.not_used.feat.tm = 1;
 
	if (0 != odp_init_global(&instance, &init_param, NULL))
	{
		DBG(("init odp global failed.\n"));
		return 0;
	}