Script for testing HBM (Host Bandwidth Manager) framework.
It creates a cgroup to use for testing and load a BPF program to limit
egress bandwidht. It then uses iperf3 or netperf to create
loads. The output is the goodput in Mbps (unless -D is used).
It can work on a single host using loopback or among two hosts (with netperf).
When using loopback, it is recommended to also introduce a delay of at least
1ms (-d=1), otherwise the assigned bandwidth is likely to be underutilized.
USAGE: $name [out] [-b=<prog>|--bpf=<prog>] [-c=<cc>|--cc=<cc>] [-D]
[-d=<delay>|--delay=<delay>] [--debug] [-E]
[-f=<#flows>|--flows=<#flows>] [-h] [-i=<id>|--id=<id >] [-l]
[-N] [-p=<port>|--port=<port>] [-P] [-q=<qdisc>]
[-R] [-s=<server>|--server=<server] [--stats]
[-t=<time>|--time=<time>] [-w] [cubic|dctcp]
Where:
out Egress (default egress)
-b or --bpf BPF program filename to load and attach.
Default is nrm_out_kern.o for egress,
-c or -cc TCP congestion control (cubic or dctcp)
-d or --delay Add a delay in ms using netem
-D In addition to the goodput in Mbps, it also outputs
other detailed information. This information is
test dependent (i.e. iperf3 or netperf).
--debug Print BPF trace buffer
-E Enable ECN (not required for dctcp)
-f or --flows Number of concurrent flows (default=1)
-i or --id cgroup id (an integer, default is 1)
-l Do not limit flows using loopback
-N Use netperf instead of iperf3
-h Help
-p or --port iperf3 port (default is 5201)
-P Use an iperf3 instance for each flow
-q Use the specified qdisc.
-r or --rate Rate in Mbps (default 1s 1Gbps)
-R Use TCP_RR for netperf. 1st flow has req
size of 10KB, rest of 1MB. Reply in all
cases is 1 byte.
More detailed output for each flow can be found
in the files netperf.<cg>.<flow>, where <cg> is the
cgroup id as specified with the -i flag, and <flow>
is the flow id starting at 1 and increasing by 1 for
flow (as specified by -f).
-s or --server hostname of netperf server. Used to create netperf
test traffic between to hosts (default is within host)
netserver must be running on the host.
--stats Get HBM stats (marked, dropped, etc.)
-t or --time duration of iperf3 in seconds (default=5)
-w Work conserving flag. cgroup can increase its
bandwidth beyond the rate limit specified
while there is available bandwidth. Current
implementation assumes there is only one NIC
(eth0), but can be extended to support multiple
NICs. This is just a proof of concept.
cubic or dctcp specify TCP CC to use
Examples:
./do_hbm_test.sh -l -d=1 -D --stats
Runs a 5 second test, using a single iperf3 flow and with the default
rate limit of 1Gbps and a delay of 1ms (using netem) using the default
TCP congestion control on the loopback device (hence we use "-l" to
enforce bandwidth limit on loopback device). Since no direction is
specified, it defaults to egress. Since no TCP CC algorithm is
specified it uses the system default (Cubic for this test).
With no -D flag, only the value of the AGGREGATE OUTPUT would show.
id refers to the cgroup id and is useful when running multi cgroup
tests (supported by a future patch).
This patchset does not support calling TCP's congesion window
reduction, even when packets are dropped by the BPF program, resulting
in a large number of packets dropped. It is recommended that the current
HBM implemenation only be used with ECN enabled flows. A future patch
will add support for reducing TCP's cwnd and will increase the
performance of non-ECN enabled flows.
Output:
Details for HBM in cgroup 1
id:1
rate_mbps:493
duration:4.8 secs
packets:11355
bytes_MB:590
pkts_dropped:4497
bytes_dropped_MB:292
pkts_marked_percent: 39.60
bytes_marked_percent: 49.49
pkts_dropped_percent: 39.60
bytes_dropped_percent: 49.49
PING AVG DELAY:2.075
AGGREGATE_GOODPUT:505
./do_nrm_test.sh -l -d=1 -D --stats dctcp
Same as above but using dctcp. Note that fewer bytes are dropped
(0.01% vs. 49%).
Output:
Details for HBM in cgroup 1
id:1
rate_mbps:945
duration:4.9 secs
packets:16859
bytes_MB:578
pkts_dropped:1
bytes_dropped_MB:0
pkts_marked_percent: 28.74
bytes_marked_percent: 45.15
pkts_dropped_percent: 0.01
bytes_dropped_percent: 0.01
PING AVG DELAY:2.083
AGGREGATE_GOODPUT:965
./do_nrm_test.sh -d=1 -D --stats
As first example, but without limiting loopback device (i.e. no
"-l" flag). Since there is no bandwidth limiting, no details for
HBM are printed out.
Output:
Details for HBM in cgroup 1
PING AVG DELAY:2.019
AGGREGATE_GOODPUT:42655
./do_hbm.sh -l -d=1 -D --stats -f=2
Uses iper3 and does 2 flows
./do_hbm.sh -l -d=1 -D --stats -f=4 -P
Uses iperf3 and does 4 flows, each flow as a separate process.
./do_hbm.sh -l -d=1 -D --stats -f=4 -N
Uses netperf, 4 flows
./do_hbm.sh -f=1 -r=2000 -t=5 -N -D --stats dctcp -s=<server-name>
Uses netperf between two hosts. The remote host name is specified
with -s= and you need to start the program netserver manually on
the remote host. It will use 1 flow, a rate limit of 2Gbps and dctcp.
./do_hbm.sh -f=1 -r=2000 -t=5 -N -D --stats -w dctcp \
-s=<server-name>
As previous, but allows use of extra bandwidth. For this test the
rate is 8Gbps vs. 1Gbps of the previous test.
Signed-off-by: Lawrence Brakmo <brakmo@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
eBPF sample programs
====================
This directory contains a test stubs, verifier test-suite and examples
for using eBPF. The examples use libbpf from tools/lib/bpf.
Build dependencies
==================
Compiling requires having installed:
* clang >= version 3.4.0
* llvm >= version 3.7.1
Note that LLVM's tool 'llc' must support target 'bpf', list version
and supported targets with command: ``llc --version``
Kernel headers
--------------
There are usually dependencies to header files of the current kernel.
To avoid installing devel kernel headers system wide, as a normal
user, simply call::
make headers_install
This will creates a local "usr/include" directory in the git/build top
level directory, that the make system automatically pickup first.
Compiling
=========
For building the BPF samples, issue the below command from the kernel
top level directory::
make samples/bpf/
Do notice the "/" slash after the directory name.
It is also possible to call make from this directory. This will just
hide the the invocation of make as above with the appended "/".
Manually compiling LLVM with 'bpf' support
------------------------------------------
Since version 3.7.0, LLVM adds a proper LLVM backend target for the
BPF bytecode architecture.
By default llvm will build all non-experimental backends including bpf.
To generate a smaller llc binary one can use::
-DLLVM_TARGETS_TO_BUILD="BPF"
Quick sniplet for manually compiling LLVM and clang
(build dependencies are cmake and gcc-c++)::
$ git clone http://llvm.org/git/llvm.git
$ cd llvm/tools
$ git clone --depth 1 http://llvm.org/git/clang.git
$ cd ..; mkdir build; cd build
$ cmake .. -DLLVM_TARGETS_TO_BUILD="BPF;X86"
$ make -j $(getconf _NPROCESSORS_ONLN)
It is also possible to point make to the newly compiled 'llc' or
'clang' command via redefining LLC or CLANG on the make command line::
make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
Cross compiling samples
-----------------------
In order to cross-compile, say for arm64 targets, export CROSS_COMPILE and ARCH
environment variables before calling make. This will direct make to build
samples for the cross target.
export ARCH=arm64
export CROSS_COMPILE="aarch64-linux-gnu-"
make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
4ffd44cfd1