1015 lines
36 KiB
Plaintext
1015 lines
36 KiB
Plaintext
Rocker Network Switch Register Programming Guide
|
||
Copyright (c) Scott Feldman <sfeldma@gmail.com>
|
||
Copyright (c) Neil Horman <nhorman@tuxdriver.com>
|
||
Version 0.11, 12/29/2014
|
||
|
||
LICENSE
|
||
=======
|
||
|
||
This program is free software; you can redistribute it and/or modify
|
||
it under the terms of the GNU General Public License as published by
|
||
the Free Software Foundation; either version 2 of the License, or
|
||
(at your option) any later version.
|
||
|
||
This program is distributed in the hope that it will be useful,
|
||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||
GNU General Public License for more details.
|
||
|
||
SECTION 1: Introduction
|
||
=======================
|
||
|
||
Overview
|
||
--------
|
||
|
||
This document describes the hardware/software interface for the Rocker switch
|
||
device. The intended audience is authors of OS drivers and device emulation
|
||
software.
|
||
|
||
Notations and Conventions
|
||
-------------------------
|
||
|
||
o In register descriptions, [n:m] indicates a range from bit n to bit m,
|
||
inclusive.
|
||
o Use of leading 0x indicates a hexadecimal number.
|
||
o Use of leading 0b indicates a binary number.
|
||
o The use of RSVD or Reserved indicates that a bit or field is reserved for
|
||
future use.
|
||
o Field width is in bytes, unless otherwise noted.
|
||
o Register are (R) read-only, (R/W) read/write, (W) write-only, or (COR) clear
|
||
on read
|
||
o TLV values in network-byte-order are designated with (N).
|
||
|
||
|
||
SECTION 2: PCI Configuration Registers
|
||
======================================
|
||
|
||
PCI Configuration Space
|
||
-----------------------
|
||
|
||
Each switch instance registers as a PCI device with PCI configuration space:
|
||
|
||
offset width description value
|
||
---------------------------------------------
|
||
0x0 2 Vendor ID 0x1b36
|
||
0x2 2 Device ID 0x0006
|
||
0x4 4 Command/Status
|
||
0x8 1 Revision ID 0x01
|
||
0x9 3 Class code 0x2800
|
||
0xC 1 Cache line size
|
||
0xD 1 Latency timer
|
||
0xE 1 Header type
|
||
0xF 1 Built-in self test
|
||
0x10 4 Base address low
|
||
0x14 4 Base address high
|
||
0x18-28 Reserved
|
||
0x2C 2 Subsystem vendor ID *
|
||
0x2E 2 Subsystem ID *
|
||
0x30-38 Reserved
|
||
0x3C 1 Interrupt line
|
||
0x3D 1 Interrupt pin 0x00
|
||
0x3E 1 Min grant 0x00
|
||
0x3D 1 Max latency 0x00
|
||
0x40 1 TRDY timeout
|
||
0x41 1 Retry count
|
||
0x42 2 Reserved
|
||
|
||
|
||
* Assigned by sub-system implementation
|
||
|
||
SECTION 3: Memory-Mapped Register Space
|
||
=======================================
|
||
|
||
There are two memory-mapped BARs. BAR0 maps device register space and is
|
||
0x2000 in size. BAR1 maps MSI-X vector and PBA tables and is also 0x2000 in
|
||
size, allowing for 256 MSI-X vectors.
|
||
|
||
All registers are 4 or 8 bytes long. It is assumed host software will access 4
|
||
byte registers with one 4-byte access, and 8 byte registers with either two
|
||
4-byte accesses or a single 8-byte access. In the case of two 4-byte accesses,
|
||
access must be lower and then upper 4-bytes, in that order.
|
||
|
||
BAR0 device register space is organized as follows:
|
||
|
||
offset description
|
||
------------------------------------------------------
|
||
0x0000-0x000f Bogus registers to catch misbehaving
|
||
drivers. Writes do nothing. Reads
|
||
back as 0xDEADBABE.
|
||
0x0010-0x00ff Test registers
|
||
0x0300-0x03ff General purpose registers
|
||
0x1000-0x1fff Descriptor control
|
||
|
||
Holes in register space are reserved. Writes to reserved registers do nothing.
|
||
Reads to reserved registers read back as 0.
|
||
|
||
No fancy stuff like write-combining is enabled on any of the registers.
|
||
|
||
BAR1 MSI-X register space is organized as follows:
|
||
|
||
offset description
|
||
------------------------------------------------------
|
||
0x0000-0x0fff MSI-X vector table (256 vectors total)
|
||
0x1000-0x1fff MSI-X PBA table
|
||
|
||
|
||
SECTION 4: Interrupts, DMA, and Endianness
|
||
==========================================
|
||
|
||
PCI Interrupts
|
||
--------------
|
||
|
||
The device supports only MSI-X interrupts. BAR1 memory-mapped region contains
|
||
the MSI-X vector and PBA tables, with support for up to 256 MSI-X vectors.
|
||
|
||
The vector assignment is:
|
||
|
||
vector description
|
||
-----------------------------------------------------
|
||
0 Command descriptor ring completion
|
||
1 Event descriptor ring completion
|
||
2 Test operation completion
|
||
3 RSVD
|
||
4-255 Tx and Rx descriptor ring completion
|
||
Tx vector is even
|
||
Rx vector is odd
|
||
|
||
A MSI-X vector table entry is 16 bytes:
|
||
|
||
field offset width description
|
||
-------------------------------------------------------------
|
||
lower_addr 0x0 4 [31:2] message address[31:2]
|
||
[1:0] Rsvd (4 byte alignment
|
||
required)
|
||
upper_addr 0x4 4 [31:19] Rsvd
|
||
[14:0] message address[46:32]
|
||
data 0x8 4 message data[31:0]
|
||
control 0xc 4 [31:1] Rsvd
|
||
[0] mask (0 = enable,
|
||
1 = masked)
|
||
|
||
Software should install the Interrupt Service Routine (ISR) before any ports
|
||
are enabled or any commands are issued on the command ring.
|
||
|
||
DMA Operations
|
||
--------------
|
||
|
||
DMA operations are used for packet DMA to/from the CPU, command and event
|
||
processing. Command processing includes statistical counters and table dumps,
|
||
table insertion/deletion, and more. Event processing provides an async
|
||
notification method for device-originating events. Each DMA operation has a
|
||
set of control registers to manage a descriptor ring. The descriptor rings are
|
||
allocated from contiguous host DMA-able memory and registers specify the rings
|
||
base address, size and current head and tail indices. Software always writes
|
||
the head, and hardware always writes the tail.
|
||
|
||
The higher-order bit of DMA_DESC_COMP_ERR is used to mark hardware completion
|
||
of a descriptor. Software will clear this bit when posting a descriptor to the
|
||
ring, and hardware will set this bit when the descriptor is complete.
|
||
|
||
Descriptor ring sizes must be a power of 2 and range from 2 to 64K entries.
|
||
Descriptor rings' base address must be 8-byte aligned. Descriptors must be
|
||
packed within ring. Each descriptor in each ring must also be aligned on an 8
|
||
byte boundary. Each descriptor ring will have these registers:
|
||
|
||
DMA_DESC_xxx_BASE_ADDR, offset 0x1000 + (x * 32), 64-bit, (R/W)
|
||
DMA_DESC_xxx_SIZE, offset 0x1008 + (x * 32), 32-bit, (R/W)
|
||
DMA_DESC_xxx_HEAD, offset 0x100c + (x * 32), 32-bit, (R/W)
|
||
DMA_DESC_xxx_TAIL, offset 0x1010 + (x * 32), 32-bit, (R)
|
||
DMA_DESC_xxx_CTRL, offset 0x1014 + (x * 32), 32-bit, (W)
|
||
DMA_DESC_xxx_CREDITS, offset 0x1018 + (x * 32), 32-bit, (R/W)
|
||
DMA_DESC_xxx_RSVD1, offset 0x101c + (x * 32), 32-bit, (R/W)
|
||
|
||
Where x is descriptor ring index:
|
||
|
||
index ring
|
||
--------------------
|
||
0 CMD
|
||
1 EVENT
|
||
2 TX (port 0)
|
||
3 RX (port 0)
|
||
4 TX (port 1)
|
||
5 RX (port 1)
|
||
.
|
||
.
|
||
.
|
||
124 TX (port 61)
|
||
125 RX (port 61)
|
||
126 Resv
|
||
127 Resv
|
||
|
||
Writing BASE_ADDR or SIZE will reset HEAD and TAIL to zero. HEAD cannot be
|
||
written past TAIL. To do so would wrap the ring. An empty ring is when HEAD
|
||
== TAIL. A full ring is when HEAD is one position behind TAIL. Both HEAD and
|
||
TAIL increment and modulo wrap at the ring size.
|
||
|
||
CTRL register bits:
|
||
|
||
bit name description
|
||
------------------------------------------------------------------------
|
||
[0] CTRL_RESET Reset the descriptor ring
|
||
[1:31] Reserved
|
||
|
||
All descriptor types share some common fields:
|
||
|
||
field width description
|
||
-------------------------------------------------------------------
|
||
DMA_DESC_BUF_ADDR 8 Phys addr of desc payload, 8-byte
|
||
aligned
|
||
DMA_DESC_COOKIE 8 Desc cookie for completion matching,
|
||
upper-most bit is reserved
|
||
DMA_DESC_BUF_SIZE 2 Desc payload size in bytes
|
||
DMA_DESC_TLV_SIZE 2 Desc payload total size in bytes
|
||
used for TLVs. Must be <=
|
||
DMA_DESC_BUF_SIZE.
|
||
DMA_DESC_COMP_ERR 2 Completion status of associated
|
||
desc payload. High order bit is
|
||
clear on new descs, toggled by
|
||
hw for completed items.
|
||
|
||
To support forward- and backward-compatibility, descriptor and completion
|
||
payloads are specified in TLV format. Fields are packed with Type=field name,
|
||
Length=field length, and Value=field value. Software will ignore unknown fields
|
||
filled in by the switch. Likewise, the switch will ignore unknown fields
|
||
filled in by software.
|
||
|
||
Descriptor payload buffer is 8-byte aligned and TLVs are 8-byte aligned. The
|
||
value within a TLV is also 8-byte aligned. The (packed, 8 byte) TLV header is:
|
||
|
||
field width description
|
||
-----------------------------
|
||
type 4 TLV type
|
||
len 2 TLV value length
|
||
pad 2 Reserved
|
||
|
||
The alignment requirements for descriptors and TLVs are to avoid unaligned
|
||
access exceptions in software. Note that the payload for each TLV is also
|
||
8 byte aligned.
|
||
|
||
Figure 1 shows an example descriptor buffer with two TLVs.
|
||
|
||
<------- 8 bytes ------->
|
||
|
||
8-byte +––––+ +–––––––––––+–––––+–––––+ +–+
|
||
align | type | len | pad | TLV#1 hdr |
|
||
+–––––––––––+–––––+–––––+ (len=22) |
|
||
| | |
|
||
| value | TVL#1 value |
|
||
| | (padded to 8-byte |
|
||
| +–––––+ alignment) |
|
||
| |/////| |
|
||
8-byte +––––+ +–––––––––––+–––––––––––+ |
|
||
align | type | len | pad | TLV#2 hdr DESC_BUF_SIZE
|
||
+–––––+–––––+–––––+–––––+ (len=2) |
|
||
|value|/////////////////| TLV#2 value |
|
||
+–––––+/////////////////| |
|
||
|///////////////////////| |
|
||
|///////////////////////| |
|
||
|///////////////////////| |
|
||
|////////unused/////////| |
|
||
|////////space//////////| |
|
||
|///////////////////////| |
|
||
|///////////////////////| |
|
||
|///////////////////////| |
|
||
+–––––––––––––––––––––––+ +–+
|
||
|
||
fig. 1
|
||
|
||
TLVs can be nested within the NEST TLV type.
|
||
|
||
Interrupt credits
|
||
^^^^^^^^^^^^^^^^^
|
||
|
||
MSI-X vectors used for descriptor ring completions use a credit mechanism for
|
||
efficient device, PCIe bus, OS and driver operations. Each descriptor ring has
|
||
a credit count which represents the number of outstanding descriptors to be
|
||
processed by the driver. As the device marks descriptors complete, the credit
|
||
count is incremented. As the driver processes those outstanding descriptors,
|
||
it returns credits back to the device. This way, the device knows the driver's
|
||
progress and can make decisions about when to fire the next interrupt or not.
|
||
When the credit count is zero, and the first descriptors are posted for the
|
||
driver, a single interrupt is fired. Once the interrupt is fired, the
|
||
interrupt is disabled (auto-masked*). In response to the interrupt, the driver
|
||
will process descriptors and PIO write a returned credit value for that
|
||
descriptor ring. If the driver returns all credits (the driver caught up with
|
||
the device and there is no outstanding work), then the interrupt is unmasked,
|
||
but not fired. If only partial credits are returned, the interrupt remains
|
||
masked but the device generates an interrupt, signaling the driver that more
|
||
outstanding work is available.
|
||
|
||
(* this masking is unrelated to the MSI-X interrupt mask register)
|
||
|
||
Endianness
|
||
----------
|
||
|
||
Device registers are hard-coded to little-endian (LE). The driver should
|
||
convert to/from host endianness to LE for device register accesses.
|
||
|
||
Descriptors are LE. Descriptor buffer TLVs will have LE type and length
|
||
fields, but the value field can either be LE or network-byte-order, depending
|
||
on context. TLV values containing network packet data will be in network-byte
|
||
order. A TLV value containing a field or mask used to compare against network
|
||
packet data is network-byte order. For example, flow match fields (and masks)
|
||
are network-byte-order since they're matched directly, byte-by-byte, against
|
||
network packet data. All non-network-packet TLV multi-byte values will be LE.
|
||
|
||
TLV values in network-byte-order are designated with (N).
|
||
|
||
|
||
SECTION 5: Test Registers
|
||
=========================
|
||
|
||
Rocker has several test registers to support troubleshooting register access,
|
||
interrupt generation, and DMA operations:
|
||
|
||
TEST_REG, offset 0x0010, 32-bit (R/W)
|
||
TEST_REG64, offset 0x0018, 64-bit (R/W)
|
||
TEST_IRQ, offset 0x0020, 32-bit (R/W)
|
||
TEST_DMA_ADDR, offset 0x0028, 64-bit (R/W)
|
||
TEST_DMA_SIZE, offset 0x0030, 32-bit (R/W)
|
||
TEST_DMA_CTRL, offset 0x0034, 32-bit (R/W)
|
||
|
||
Reads to TEST_REG and TEST_REG64 will read a value equal to twice the last
|
||
value written to the register. The 32-bit and 64-bit versions are for testing
|
||
32-bit and 64-bit host accesses.
|
||
|
||
A vector can be written to TEST_IRQ and the device will generate an interrupt
|
||
for that vector.
|
||
|
||
To test basic DMA operations, allocate a DMA-able host buffer and put the
|
||
buffer address into TEST_DMA_ADDR and size into TEST_DMA_SIZE. Then, write to
|
||
TEST_DMA_CTRL to manipulate the buffer contents. TEST_DMA_CTRL operations are:
|
||
|
||
operation value description
|
||
-----------------------------------------------------------
|
||
TEST_DMA_CTRL_CLEAR 1 clear buffer
|
||
TEST_DMA_CTRL_FILL 2 fill buffer bytes with 0x96
|
||
TEST_DMA_CTRL_INVERT 4 invert bytes in buffer
|
||
|
||
Various buffer address and sizes should be tested to verify no address boundary
|
||
issue exists. In particular, buffers that start on odd-8-byte boundary and/or
|
||
span multiple PAGE sizes should be tested.
|
||
|
||
|
||
SECTION 6: Ports
|
||
================
|
||
|
||
Physical and Logical Ports
|
||
------------------------------------
|
||
|
||
The switch supports up to 62 physical (front-panel) ports. Register
|
||
PORT_PHYS_COUNT returns the actual number of physical ports available:
|
||
|
||
PORT_PHYS_COUNT, offset 0x0304, 32-bit, (R)
|
||
|
||
In addition to front-panel ports, the switch supports logical ports for
|
||
tunnels.
|
||
|
||
Front-panel ports and logical tunnel ports are mapped into a single 32-bit port
|
||
space. A special CPU port is assigned port 0. The front-panel ports are
|
||
mapped to ports 1-62. A special loopback port is assigned port 63. Logical
|
||
tunnel ports are assigned ports 0x0001000-0x0001ffff.
|
||
To summarize the port assignments:
|
||
|
||
port mapping
|
||
-------------------------------------------------------
|
||
0 CPU port (for packets to/from host CPU)
|
||
1-62 front-panel physical ports
|
||
63 loopback port
|
||
64-0x0000ffff RSVD
|
||
0x00010000-0x0001ffff logical tunnel ports
|
||
0x00020000-0xffffffff RSVD
|
||
|
||
Physical Port Mode
|
||
------------------
|
||
|
||
Switch front-panel ports operate in a mode. Currently, the only mode is
|
||
OF-DPA. OF-DPA[1] mode is based on OpenFlow Data Plane Abstraction (OF-DPA)
|
||
Abstract Switch Specification, Version 1.0, from Broadcom Corporation. To
|
||
set/get the mode for front-panel ports, see port settings, below.
|
||
|
||
Port Settings
|
||
-------------
|
||
|
||
Link status for all front-panel ports is available via PORT_PHYS_LINK_STATUS:
|
||
|
||
PORT_PHYS_LINK_STATUS, offset 0x0310, 64-bit, (R)
|
||
|
||
Value is port bitmap. Bits 0 and 63 always read 0. Bits 1-62
|
||
read 1 for link UP and 0 for link DOWN for respective front-panel ports.
|
||
|
||
Other properties for front-panel ports are available via DMA CMD descriptors:
|
||
|
||
Get PORT_SETTINGS descriptor:
|
||
|
||
field width description
|
||
----------------------------------------------
|
||
PORT_SETTINGS 2 CMD_GET
|
||
PPORT 4 Physical port #
|
||
|
||
Get PORT_SETTINGS completion:
|
||
|
||
field width description
|
||
----------------------------------------------
|
||
PPORT 4 Physical port #
|
||
SPEED 4 Current port interface speed, in Mbps
|
||
DUPLEX 1 1 = Full, 0 = Half
|
||
AUTONEG 1 1 = enabled, 0 = disabled
|
||
MACADDR 6 Port MAC address
|
||
MODE 1 0 = OF-DPA
|
||
LEARNING 1 MAC address learning on port
|
||
1 = enabled
|
||
0 = disabled
|
||
PHYS_NAME <var> Physical port name (string)
|
||
|
||
Set PORT_SETTINGS descriptor:
|
||
|
||
field width description
|
||
----------------------------------------------
|
||
PORT_SETTINGS 2 CMD_SET
|
||
PPORT 4 Physical port #
|
||
SPEED 4 Port interface speed, in Mbps
|
||
DUPLEX 1 1 = Full, 0 = Half
|
||
AUTONEG 1 1 = enabled, 0 = disabled
|
||
MACADDR 6 Port MAC address
|
||
MODE 1 0 = OF-DPA
|
||
|
||
Port Enable
|
||
-----------
|
||
|
||
Front-panel ports are initially disabled, which means port ingress and egress
|
||
packets will be dropped. To enable or disable a port, use PORT_PHYS_ENABLE:
|
||
|
||
PORT_PHYS_ENABLE: offset 0x0318, 64-bit, (R/W)
|
||
|
||
Value is bitmap of first 64 ports. Bits 0 and 63 are ignored
|
||
and always read as 0. Write 1 to enable port; write 0 to disable it.
|
||
Default is 0.
|
||
|
||
|
||
SECTION 7: Switch Control
|
||
=========================
|
||
|
||
This section covers switch-wide register settings.
|
||
|
||
Control
|
||
-------
|
||
|
||
This register is used for low level control of the switch.
|
||
|
||
CONTROL: offset 0x0300, 32-bit, (W)
|
||
|
||
bit name description
|
||
------------------------------------------------------------------------
|
||
[0] CONTROL_RESET If set, device will perform reset
|
||
[1:31] Reserved
|
||
|
||
Switch ID
|
||
---------
|
||
|
||
The switch has a SWITCH_ID to be used by software to uniquely identify the
|
||
switch:
|
||
|
||
SWITCH_ID: offset 0x0320, 64-bit, (R)
|
||
|
||
Value is opaque to switch software and no special encoding is implied.
|
||
|
||
|
||
SECTION 8: Events
|
||
=================
|
||
|
||
Non-I/O asynchronous events from the device are notified to the host using the
|
||
event ring. The TLV structure for events is:
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
TYPE 4 Event type, one of:
|
||
1: LINK_CHANGED
|
||
2: MAC_VLAN_SEEN
|
||
INFO <nest> Event info (details below)
|
||
|
||
Link Changed Event
|
||
------------------
|
||
|
||
When link status changes on a physical port, this event is generated.
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
INFO <nest>
|
||
PPORT 4 Physical port
|
||
LINKUP 1 Link status:
|
||
0: down
|
||
1: up
|
||
|
||
MAC VLAN Seen Event
|
||
-------------------
|
||
|
||
When a packet ingresses on a port and the source MAC/VLAN isn't known to the
|
||
device, the device will generate this event. In response to the event, the
|
||
driver should install to the device the MAC/VLAN on the port into the bridge
|
||
table. Once installed, the MAC/VLAN is known on the port and this event will
|
||
no longer be generated.
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
INFO <nest>
|
||
PPORT 4 Physical port
|
||
MAC 6 MAC address
|
||
VLAN 2 VLAN ID
|
||
|
||
|
||
SECTION 9: CPU Packet Processing
|
||
================================
|
||
|
||
Ingress packets directed to the host CPU for further processing are delivered
|
||
in the DMA RX ring. Likewise, host CPU originating packets destined to egress
|
||
on switch ports are scheduled by software using the DMA TX ring.
|
||
|
||
Tx Packet Processing
|
||
--------------------
|
||
|
||
Software schedules packets for egress on switch ports using the DMA TX ring. A
|
||
TX descriptor buffer describes the packet location and size in host DMA-able
|
||
memory, the destination port, and any hardware-offload functions (such as L3
|
||
payload checksum offload). Software then bumps the descriptor head to signal
|
||
hardware of new Tx work. In response, hardware will DMA read Tx descriptors up
|
||
to head, DMA read descriptor buffer and packet data, perform offloading
|
||
functions, and finally frame packet on wire (network). Once packet processing
|
||
is complete, hardware will writeback status to descriptor(s) to signal to
|
||
software that Tx is complete and software resources (e.g. skb) backing packet
|
||
can be released.
|
||
|
||
Figure 2 shows an example 3-fragment packet queued with one Tx descriptor. A
|
||
TLV is used for each packet fragment.
|
||
|
||
pkt frag 1
|
||
+–––––––+ +–+
|
||
+–––+ | |
|
||
desc buf | | | |
|
||
+––––––––+ | | | |
|
||
Tx ring +–––+ +–––––+ | | |
|
||
+–––––––––+ | | TLVs | +–––––––+ |
|
||
| +–––+ +––––––––+ pkt frag 2 |
|
||
| desc 0 | | +–––––+ +–––––––+ |
|
||
+–––––––––+ | TLVs | +–––+ | |
|
||
head+–+ | +––––––––+ | | |
|
||
| desc 1 | | +–––––+ +–––––––+ |pkt
|
||
+–––––––––+ | TLVs | | |
|
||
| | +––––––––+ | pkt frag 3 |
|
||
| | | +–––––––+ |
|
||
+–––––––––+ +–––+ | |
|
||
| | | | |
|
||
| | | | |
|
||
+–––––––––+ | | |
|
||
| | | | |
|
||
| | | | |
|
||
+–––––––––+ | | |
|
||
| | +–––––––+ +–+
|
||
| |
|
||
+–––––––––+
|
||
|
||
fig 2.
|
||
|
||
The TLVs for Tx descriptor buffer are:
|
||
|
||
field width description
|
||
---------------------------------------------------------------------
|
||
PPORT 4 Destination physical port #
|
||
TX_OFFLOAD 1 Hardware offload modes:
|
||
0: no offload
|
||
1: insert IP csum (ipv4 only)
|
||
2: insert TCP/UDP csum
|
||
3: L3 csum calc and insert
|
||
into csum offset (TX_L3_CSUM_OFF)
|
||
16-bit 1's complement csum value.
|
||
IPv4 pseudo-header and IP
|
||
already calculated by OS
|
||
and inserted.
|
||
4: TSO (TCP Segmentation Offload)
|
||
TX_L3_CSUM_OFF 2 For L3 csum offload mode, the offset,
|
||
from the beginning of the packet,
|
||
of the csum field in the L3 header
|
||
TX_TSO_MSS 2 For TSO offload mode, the
|
||
Maximum Segment Size in bytes
|
||
TX_TSO_HDR_LEN 2 For TSO offload mode, the
|
||
length of ethernet, IP, and
|
||
TCP/UDP headers, including IP
|
||
and TCP options.
|
||
TX_FRAGS <array> Packet fragments
|
||
TX_FRAG <nest> Packet fragment
|
||
TX_FRAG_ADDR 8 DMA address of packet fragment
|
||
TX_FRAG_LEN 2 Packet fragment length
|
||
|
||
Possible status return codes in descriptor on completion are:
|
||
|
||
DESC_COMP_ERR reason
|
||
--------------------------------------------------------------------
|
||
0 OK
|
||
-ROCKER_ENXIO address or data read err on desc buf or packet
|
||
fragment
|
||
-ROCKER_EINVAL bad pport or TSO or csum offloading error
|
||
-ROCKER_ENOMEM no memory for internal staging tx fragment
|
||
|
||
Rx Packet Processing
|
||
--------------------
|
||
|
||
For packets ingressing on switch ports that are not forwarded by the switch but
|
||
rather directed to the host CPU for further processing are delivered in the DMA
|
||
RX ring. Rx descriptor buffers are allocated by software and placed on the
|
||
ring. Hardware will fill Rx descriptor buffers with packet data, write the
|
||
completion, and signal to software that a new packet is ready. Since Rx packet
|
||
size is not known a-priori, the Rx descriptor buffer must be allocated for
|
||
worst-case packet size. A single Rx descriptor will contain the entire Rx
|
||
packet data in one RX_FRAG. Other Rx TLVs describe and hardware offloads
|
||
performed on the packet, such as checksum validation.
|
||
|
||
The TLVs for Rx descriptor buffer are:
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
PPORT 4 Source physical port #
|
||
RX_FLAGS 2 Packet parsing flags:
|
||
(1 << 0): IPv4 packet
|
||
(1 << 1): IPv6 packet
|
||
(1 << 2): csum calculated
|
||
(1 << 3): IPv4 csum good
|
||
(1 << 4): IP fragment
|
||
(1 << 5): TCP packet
|
||
(1 << 6): UDP packet
|
||
(1 << 7): TCP/UDP csum good
|
||
(1 << 8): Offload forward
|
||
RX_CSUM 2 IP calculated checksum:
|
||
IPv4: IP payload csum
|
||
IPv6: header and payload csum
|
||
(Only valid is RX_FLAGS:csum calc is set)
|
||
RX_FRAG_ADDR 8 DMA address of packet fragment
|
||
RX_FRAG_MAX_LEN 2 Packet maximum fragment length
|
||
RX_FRAG_LEN 2 Actual packet fragment length after receive
|
||
|
||
Offload forward RX_FLAG indicates the device has already forwarded the packet
|
||
so the host CPU should not also forward the packet.
|
||
|
||
Possible status return codes in descriptor on completion are:
|
||
|
||
DESC_COMP_ERR reason
|
||
--------------------------------------------------------------------
|
||
0 OK
|
||
-ROCKER_ENXIO address or data read err on desc buf
|
||
-ROCKER_ENOMEM no memory for internal staging desc buf
|
||
-ROCKER_EMSGSIZE Rx descriptor buffer wasn't big enough to contain
|
||
packet data TLV and other TLVs.
|
||
|
||
|
||
SECTION 10: OF-DPA Mode
|
||
======================
|
||
|
||
OF-DPA mode allows the switch to offload flow packet processing functions to
|
||
hardware. An OpenFlow controller would communicate with an OpenFlow agent
|
||
installed on the switch. The OpenFlow agent would (directly or indirectly)
|
||
communicate with the Rocker switch driver, which in turn would program switch
|
||
hardware with flow functionality, as defined in OF-DPA. The block diagram is:
|
||
|
||
+–––––––––––––––----–––+
|
||
| OF |
|
||
| Remote Controller |
|
||
+––––––––+––----–––––––+
|
||
|
|
||
|
|
||
+––––––––+–––––––––+
|
||
| OF |
|
||
| Local Agent |
|
||
+––––––––––––––––––+
|
||
| |
|
||
| Rocker Driver |
|
||
+––––––––––––––––––+
|
||
<this spec>
|
||
+––––––––––––––––––+
|
||
| |
|
||
| Rocker Switch |
|
||
+––––––––––––––––––+
|
||
|
||
To participate in flow functions, ports must be configure for OF-DPA mode
|
||
during switch initialization.
|
||
|
||
OF-DPA Flow Table Interface
|
||
---------------------------
|
||
|
||
There are commands to add, modify, delete, and get stats of flow table entries.
|
||
The commands are issued using the DMA CMD descriptor ring. The following
|
||
commands are defined:
|
||
|
||
CMD_ADD: add an entry to flow table
|
||
CMD_MOD: modify an entry in flow table
|
||
CMD_DEL: delete an entry from flow table
|
||
CMD_GET_STATS: get stats for flow entry
|
||
|
||
TLVs for add and modify commands are:
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_CMD 2 CMD_[ADD|MOD]
|
||
OF_DPA_TBL 2 Flow table ID
|
||
0: ingress port
|
||
10: vlan
|
||
20: termination mac
|
||
30: unicast routing
|
||
40: multicast routing
|
||
50: bridging
|
||
60: ACL policy
|
||
OF_DPA_PRIORITY 4 Flow priority
|
||
OF_DPA_HARDTIME 4 Hard timeout for flow
|
||
OF_DPA_IDLETIME 4 Idle timeout for flow
|
||
OF_DPA_COOKIE 8 Cookie
|
||
|
||
Additional TLVs based on flow table ID:
|
||
|
||
Table ID 0: ingress port
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_IN_PPORT 4 ingress physical port number
|
||
OF_DPA_GOTO_TBL 2 goto table ID; zero to drop
|
||
|
||
Table ID 10: vlan
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_IN_PPORT 4 ingress physical port number
|
||
OF_DPA_VLAN_ID 2 (N) vlan ID
|
||
OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask
|
||
OF_DPA_GOTO_TBL 2 goto table ID; zero to drop
|
||
OF_DPA_NEW_VLAN_ID 2 (N) new vlan ID
|
||
|
||
Table ID 20: termination mac
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_IN_PPORT 4 ingress physical port number
|
||
OF_DPA_IN_PPORT_MASK 4 ingress physical port number mask
|
||
OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd
|
||
OF_DPA_DST_MAC 6 (N) destination MAC
|
||
OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask
|
||
OF_DPA_VLAN_ID 2 (N) vlan ID
|
||
OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask
|
||
OF_DPA_GOTO_TBL 2 only acceptable values are
|
||
unicast or multicast routing
|
||
table IDs
|
||
OF_DPA_OUT_PPORT 2 if specified, must be
|
||
controller, set zero otherwise
|
||
|
||
Table ID 30: unicast routing
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd
|
||
OF_DPA_DST_IP 4 (N) destination IPv4 address.
|
||
Must be unicast address
|
||
OF_DPA_DST_IP_MASK 4 (N) IP mask. Must be prefix mask
|
||
OF_DPA_DST_IPV6 16 (N) destination IPv6 address.
|
||
Must be unicast address
|
||
OF_DPA_DST_IPV6_MASK 16 (N) IPv6 mask. Must be prefix mask
|
||
OF_DPA_GOTO_TBL 2 goto table ID; zero to drop
|
||
OF_DPA_GROUP_ID 4 data for GROUP action must
|
||
be an L3 Unicast group entry
|
||
|
||
Table ID 40: multicast routing
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd
|
||
OF_DPA_VLAN_ID 2 (N) vlan ID
|
||
OF_DPA_SRC_IP 4 (N) source IPv4. Optional,
|
||
can contain IPv4 address,
|
||
must be completely masked
|
||
if not used
|
||
OF_DPA_SRC_IP_MASK 4 (N) IP Mask
|
||
OF_DPA_DST_IP 4 (N) destination IPv4 address.
|
||
Must be multicast address
|
||
OF_DPA_SRC_IPV6 16 (N) source IPv6 Address. Optional.
|
||
Can contain IPv6 address,
|
||
must be completely masked
|
||
if not used
|
||
OF_DPA_SRC_IPV6_MASK 16 (N) IPv6 mask.
|
||
OF_DPA_DST_IPV6 16 (N) destination IPv6 Address. Must
|
||
be multicast address
|
||
Must be multicast address
|
||
OF_DPA_GOTO_TBL 2 goto table ID; zero to drop
|
||
OF_DPA_GROUP_ID 4 data for GROUP action must
|
||
be an L3 multicast group entry
|
||
|
||
Table ID 50: bridging
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_VLAN_ID 2 (N) vlan ID
|
||
OF_DPA_TUNNEL_ID 4 tunnel ID
|
||
OF_DPA_DST_MAC 6 (N) destination MAC
|
||
OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask
|
||
OF_DPA_GOTO_TBL 2 goto table ID; zero to drop
|
||
OF_DPA_GROUP_ID 4 data for GROUP action must
|
||
be a L2 Interface, L2
|
||
Multicast, L2 Flood,
|
||
or L2 Overlay group entry
|
||
as appropriate
|
||
OF_DPA_TUNNEL_LPORT 4 unicast Tenant Bridging
|
||
flows specify a tunnel
|
||
logical port ID
|
||
OF_DPA_OUT_PPORT 2 data for OUTPUT action,
|
||
restricted to CONTROLLER,
|
||
set to 0 otherwise
|
||
|
||
Table ID 60: acl policy
|
||
|
||
field width description
|
||
----------------------------------------------------
|
||
OF_DPA_IN_PPORT 4 ingress physical port number
|
||
OF_DPA_IN_PPORT_MASK 4 ingress physical port number mask
|
||
OF_DPA_ETHERTYPE 2 (N) ethertype
|
||
OF_DPA_VLAN_ID 2 (N) vlan ID
|
||
OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask
|
||
OF_DPA_VLAN_PCP 2 (N) vlan Priority Code Point
|
||
OF_DPA_VLAN_PCP_MASK 2 (N) vlan Priority Code Point mask
|
||
OF_DPA_SRC_MAC 6 (N) source MAC
|
||
OF_DPA_SRC_MAC_MASK 6 (N) source MAC mask
|
||
OF_DPA_DST_MAC 6 (N) destination MAC
|
||
OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask
|
||
OF_DPA_TUNNEL_ID 4 tunnel ID
|
||
OF_DPA_SRC_IP 4 (N) source IPv4. Optional,
|
||
can contain IPv4 address,
|
||
must be completely masked
|
||
if not used
|
||
OF_DPA_SRC_IP_MASK 4 (N) IP Mask
|
||
OF_DPA_DST_IP 4 (N) destination IPv4 address.
|
||
Must be multicast address
|
||
OF_DPA_DST_IP_MASK 4 (N) IP Mask
|
||
OF_DPA_SRC_IPV6 16 (N) source IPv6 Address. Optional.
|
||
Can contain IPv6 address,
|
||
must be completely masked
|
||
if not used
|
||
OF_DPA_SRC_IPV6_MASK 16 (N) IPv6 mask
|
||
OF_DPA_DST_IPV6 16 (N) destination IPv6 Address. Must
|
||
be multicast address.
|
||
OF_DPA_DST_IPV6_MASK 16 (N) IPv6 mask
|
||
OF_DPA_SRC_ARP_IP 4 (N) source IPv4 address in the ARP
|
||
payload. Only used if ethertype
|
||
== 0x0806.
|
||
OF_DPA_SRC_ARP_IP_MASK 4 (N) IP Mask
|
||
OF_DPA_IP_PROTO 1 IP protocol
|
||
OF_DPA_IP_PROTO_MASK 1 IP protocol mask
|
||
OF_DPA_IP_DSCP 1 DSCP
|
||
OF_DPA_IP_DSCP_MASK 1 DSCP mask
|
||
OF_DPA_IP_ECN 1 ECN
|
||
OF_DPA_IP_ECN_MASK 1 ECN mask
|
||
OF_DPA_L4_SRC_PORT 2 (N) L4 source port, only for
|
||
TCP, UDP, or SCTP
|
||
OF_DPA_L4_SRC_PORT_MASK 2 (N) L4 source port mask
|
||
OF_DPA_L4_DST_PORT 2 (N) L4 source port, only for
|
||
TCP, UDP, or SCTP
|
||
OF_DPA_L4_DST_PORT_MASK 2 (N) L4 source port mask
|
||
OF_DPA_ICMP_TYPE 1 ICMP type, only if IP
|
||
protocol is 1
|
||
OF_DPA_ICMP_TYPE_MASK 1 ICMP type mask
|
||
OF_DPA_ICMP_CODE 1 ICMP code
|
||
OF_DPA_ICMP_CODE_MASK 1 ICMP code mask
|
||
OF_DPA_IPV6_LABEL 4 (N) IPv6 flow label
|
||
OF_DPA_IPV6_LABEL_MASK 4 (N) IPv6 flow label mask
|
||
OF_DPA_GROUP_ID 4 data for GROUP action
|
||
OF_DPA_QUEUE_ID_ACTION 1 write the queue ID
|
||
OF_DPA_NEW_QUEUE_ID 1 queue ID
|
||
OF_DPA_VLAN_PCP_ACTION 1 write the VLAN priority
|
||
OF_DPA_NEW_VLAN_PCP 1 VLAN priority
|
||
OF_DPA_IP_DSCP_ACTION 1 write the DSCP
|
||
OF_DPA_NEW_IP_DSCP 1 new DSCP
|
||
OF_DPA_TUNNEL_LPORT 4 restrct to valid tunnel
|
||
logical port, set to 0
|
||
otherwise.
|
||
OF_DPA_OUT_PPORT 2 data for OUTPUT action,
|
||
restricted to CONTROLLER,
|
||
set to 0 otherwise
|
||
OF_DPA_CLEAR_ACTIONS 4 if 1 packets matching flow are
|
||
dropped (all other instructions
|
||
ignored)
|
||
|
||
TLVs for flow delete and get stats command are:
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
OF_DPA_CMD 2 CMD_[DEL|GET_STATS]
|
||
OF_DPA_COOKIE 8 Cookie
|
||
|
||
On completion of get stats command, the descriptor buffer is written back with
|
||
the following TLVs:
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
OF_DPA_STAT_DURATION 4 Flow duration
|
||
OF_DPA_STAT_RX_PKTS 8 Received packets
|
||
OF_DPA_STAT_TX_PKTS 8 Transmit packets
|
||
|
||
Possible status return codes in descriptor on completion are:
|
||
|
||
DESC_COMP_ERR command reason
|
||
--------------------------------------------------------------------
|
||
0 all OK
|
||
-ROCKER_EFAULT all head or tail index outside
|
||
of ring
|
||
-ROCKER_ENXIO all address or data read err on
|
||
desc buf
|
||
-ROCKER_EMSGSIZE GET_STATS cmd descriptor buffer wasn't
|
||
big enough to contain write-back
|
||
TLVs
|
||
-ROCKER_EINVAL all invalid parameters passed in
|
||
-ROCKER_EEXIST ADD entry already exists
|
||
-ROCKER_ENOSPC ADD no space left in flow table
|
||
-ROCKER_ENOENT MOD|DEL|GET_STATS cookie invalid
|
||
|
||
Group Table Interface
|
||
---------------------
|
||
|
||
There are commands to add, modify, delete, and get stats of group table
|
||
entries. The commands are issued using the DMA CMD descriptor ring. The
|
||
following commands are defined:
|
||
|
||
CMD_ADD: add an entry to group table
|
||
CMD_MOD: modify an entry in group table
|
||
CMD_DEL: delete an entry from group table
|
||
CMD_GET_STATS: get stats for group entry
|
||
|
||
TLVs for add and modify commands are:
|
||
|
||
field width description
|
||
-----------------------------------------------------------
|
||
FLOW_GROUP_CMD 2 CMD_[ADD|MOD]
|
||
FLOW_GROUP_ID 2 Flow group ID
|
||
FLOW_GROUP_TYPE 1 Group type:
|
||
0: L2 interface
|
||
1: L2 rewrite
|
||
2: L3 unicast
|
||
3: L2 multicast
|
||
4: L2 flood
|
||
5: L3 interface
|
||
6: L3 multicast
|
||
7: L3 ECMP
|
||
8: L2 overlay
|
||
FLOW_VLAN_ID 2 Vlan ID (types 0, 3, 4, 6)
|
||
FLOW_L2_PORT 2 Port (types 0)
|
||
FLOW_INDEX 4 Index (all types but 0)
|
||
FLOW_OVERLAY_TYPE 1 Overlay sub-type (type 8):
|
||
0: Flood unicast tunnel
|
||
1: Flood multicast tunnel
|
||
2: Multicast unicast tunnel
|
||
3: Multicast multicast tunnel
|
||
FLOW_GROUP_ACTION nest
|
||
FLOW_GROUP_ID 2 next group ID in chain (all
|
||
types except 0)
|
||
FLOW_OUT_PORT 4 egress port (types 0, 8)
|
||
FLOW_POP_VLAN_TAG 1 strip outer VLAN tag (type 1
|
||
only)
|
||
FLOW_VLAN_ID 2 (types 1, 5)
|
||
FLOW_SRC_MAC 6 (types 1, 2, 5)
|
||
FLOW_DST_MAC 6 (types 1, 2)
|
||
|
||
TLVs for flow delete and get stats command are:
|
||
|
||
field width description
|
||
-----------------------------------------------------------
|
||
FLOW_GROUP_CMD 2 CMD_[DEL|GET_STATS]
|
||
FLOW_GROUP_ID 2 Flow group ID
|
||
|
||
On completion of get stats command, the descriptor buffer is written back with
|
||
the following TLVs:
|
||
|
||
field width description
|
||
---------------------------------------------------
|
||
FLOW_GROUP_ID 2 Flow group ID
|
||
FLOW_STAT_DURATION 4 Flow duration
|
||
FLOW_STAT_REF_COUNT 4 Flow reference count
|
||
FLOW_STAT_BUCKET_COUNT 4 Flow bucket count
|
||
|
||
Possible status return codes in descriptor on completion are:
|
||
|
||
DESC_COMP_ERR command reason
|
||
--------------------------------------------------------------------
|
||
0 all OK
|
||
-ROCKER_EFAULT all head or tail index outside
|
||
of ring
|
||
-ROCKER_ENXIO all address or data read err on
|
||
desc buf
|
||
-ROCKER_ENOSPC GET_STATS cmd descriptor buffer wasn't
|
||
big enough to contain write-back
|
||
TLVs
|
||
-ROCKER_EINVAL ADD|MOD invalid parameters passed in
|
||
-ROCKER_EEXIST ADD entry already exists
|
||
-ROCKER_ENOSPC ADD no space left in flow table
|
||
-ROCKER_ENOENT MOD|DEL|GET_STATS group ID invalid
|
||
-ROCKER_EBUSY DEL group reference count non-zero
|
||
-ROCKER_ENODEV ADD next group ID doesn't exist
|
||
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
[1] OpenFlow Data Plane Abstraction (OF-DPA) Abstract Switch Specification,
|
||
Version 1.0, from Broadcom Corporation, February 21, 2014.
|