OpenE2K
/
linux
6
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge branch 'chelsio-inline-tls' 

Atul Gupta says:

====================
Chelsio Inline TLS

Series for Chelsio Inline TLS driver (chtls)

Use tls ULP infrastructure to register chtls as Inline TLS driver.
Chtls use TCP Sockets to Tx/Rx TLS records.
TCP sk_proto APIs are enhanced to offload TLS record.

T6 adapter provides the following features:
        -TLS record offload, TLS header, encrypt, digest and transmit
        -TLS record receive and decrypt
        -TLS keys store
        -TCP/IP engine
        -TLS engine
        -GCM crypto engine [support CBC also]

TLS provides security at the transport layer. It uses TCP to provide
reliable end-to-end transport of application data.
It relies on TCP for any retransmission.
TLS session comprises of three parts:
a. TCP/IP connection
b. TLS handshake
c. Record layer processing

TLS handshake state machine is executed in host (refer standard
implementation eg. OpenSSL).  Setsockopt [SOL_TCP, TCP_ULP]
initialize TCP proto-ops for Chelsio inline tls support.
setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls"));

Tx and Rx Keys are decided during handshake and programmed on
the chip after CCS is exchanged.
struct tls12_crypto_info_aes_gcm_128 crypto_info
setsockopt(sock, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info))
Finish is the first encrypted/decrypted message tx/rx inline.

On the Tx path TLS engine receive plain text from openssl, insert IV,
fetches the tx key, create cipher text records and generate MAC.

TLS header is added to cipher text and forward to TCP/IP engine for
transport layer processing and transmission on wire.
TX PATH:
Apps--openssl--chtls---TLS engine---encrypt/auth---TCP/IP engine---wire

On the Rx side, data received is PDU aligned at record boundaries.
TLS processes only the complete record. If rx key is programmed on
CCS receive, data is decrypted and plain text is posted to host.
RX PATH:
Wire--cipher-text--TCP/IP engine [PDU align]---TLS engine---
decrypt/auth---plain-text--chtls--openssl--application

v15: indent fix in mark_urg
     -removed unwanted checks in sendmsg, sendpage, recvmsg,
      close, disconnect,shutdown, destroy sock [Sabrina]
     - removed unused chtls_free_kmap [chtls.h]
     - rebase to top of net-next

v14: -Reverse christmas tree style for variable declarations for
     various functions in chtls_hw.c, chtls_io.c [Stefano Brivio]
     - replaced break with return in tcp_state_to_flowc_state
       [Stefano Brivio]
     - renamed tlstx_seq_number to tlstx_incr_seqnum [Stefano Brivio]
     - use bool for corked, should_push and send_should_push
       [Stefano Brivio]
     - removed "Reviewed-by" tag for Stefano, Sabrina, Dave Watson

v13: handle clean ctx free for HW_RECORD in tls_sk_proto_close
    -removed SOCK_INLINE [chtls.h], using csk_conn_inline instead
     in send_abort_rpl,chtls_send_abort_rpl,chtls_sendmsg,chtls_sendpage
    -removed sk_no_receive [chtls_io.c] replaced with sk_shutdown &
     RCV_SHUTDOWN in chtls_pt_recvmsg, peekmsg and chtls_recvmsg
    -cleaned chtls_expansion_size [Stefano Brivio]
    - u8 conf:3 in tls_sw_context to add TLS_HW_RECORD
    -removed is_tls_skb, using tls_skb_inline [Stefano Brivio]
    -reverse christmas tree formatting in chtls_io.c, chtls_cm.c
     [Stefano Brivio]
    -fixed build warning reported by kbuild robot
    -retained ctx conf enum in chtls_main vs earlier versions, tls_prots
     not used in chtls.
    -cleanup [removed syn_sent, base_prot, added synq] [Michael Werner]
    - passing struct fw_wr_hdr * to ofldtxq_stop [Casey]
    - rebased on top of the current net-next

v12: patch against net-next
    -fixed build error [reported by Julia]
    -replace set_queue with skb_set_queue_mapping [Sabrina]
    -copyright year correction [chtls]

v11: formatting and cleanup, few function rename and error
     handling [Stefano Brivio]
     - ctx freed later for TLS_HW_RECORD
     - split tx and rx in different patch

v10: fixed following based on the review comments of Sabrina Dubroca
     -docs header added for struct tls_device [tls.h]
     -changed TLS_FULL_HW to TLS_HW_RECORD
     -similary using tls-hw-record instead of tls-inline for
     ethtool feature config
     -added more description to patch sets
     -replaced kmalloc/vmalloc/kfree with kvzalloc/kvfree
     -reordered the patch sequence
     -formatted entire patch for func return values

v9: corrected __u8 and similar usage
    -create_ctx to alloc tls_context
    -tls_hw_prot before sk !establish check

v8: tls_main.c cleanup comment [Dave Watson]

v7: func name change, use sk->sk_prot where required

v6: modify prot only for FULL_HW
   -corrected commit message for patch 11

v5: set TLS_FULL_HW for registered inline tls drivers
   -set TLS_FULL_HW prot for offload connection else move
    to TLS_SW_TX
   -Case handled for interface with same IP [Dave Miller]
   -Removed Specific IP and INADDR_ANY handling [v4]

v4: removed chtls ULP type, retained tls ULP
   -registered chtls with net tls
   -defined struct tls_device to register the Inline drivers
   -ethtool interface tls-inline to enable Inline TLS for interface
   -prot update to support inline TLS

v3: fixed the kbuild test issues
   -made few funtions static
   -initialized few variables

v2: fixed the following based on the review comments of Stephan Mueller,
    Stefano Brivio and Hannes Frederic
    -Added more details in cover letter
    -Fixed indentation and formating issues
    -Using aes instead of aes-generic
    -memset key info after programing the key on chip
    -reordered the patch sequence
====================

Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
David S. Miller 2018-03-31 23:37:33 -04:00
parent d4069fe6fc bd7f485766
commit 06b19fe9a6
22 changed files with 6295 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
drivers/crypto/chelsio/Kconfig
View File

@ -29,3 +29,14 @@ config CHELSIO_IPSEC_INLINE
default n
---help---
Enable support for IPSec Tx Inline.
config CRYPTO_DEV_CHELSIO_TLS
tristate "Chelsio Crypto Inline TLS Driver"
depends on CHELSIO_T4
depends on TLS
select CRYPTO_DEV_CHELSIO
---help---
Support Chelsio Inline TLS with Chelsio crypto accelerator.
To compile this driver as a module, choose M here: the module
will be called chtls.

1
drivers/crypto/chelsio/Makefile
View File

@ -3,3 +3,4 @@ ccflags-y := -Idrivers/net/ethernet/chelsio/cxgb4
obj-$(CONFIG_CRYPTO_DEV_CHELSIO) += chcr.o
chcr-objs := chcr_core.o chcr_algo.o
chcr-$(CONFIG_CHELSIO_IPSEC_INLINE) += chcr_ipsec.o
obj-$(CONFIG_CRYPTO_DEV_CHELSIO_TLS) += chtls/

42
drivers/crypto/chelsio/chcr_algo.h
View File

@ -86,6 +86,39 @@
KEY_CONTEXT_OPAD_PRESENT_M)
#define KEY_CONTEXT_OPAD_PRESENT_F KEY_CONTEXT_OPAD_PRESENT_V(1U)
#define TLS_KEYCTX_RXFLIT_CNT_S 24
#define TLS_KEYCTX_RXFLIT_CNT_V(x) ((x) << TLS_KEYCTX_RXFLIT_CNT_S)
#define TLS_KEYCTX_RXPROT_VER_S 20
#define TLS_KEYCTX_RXPROT_VER_M 0xf
#define TLS_KEYCTX_RXPROT_VER_V(x) ((x) << TLS_KEYCTX_RXPROT_VER_S)
#define TLS_KEYCTX_RXCIPH_MODE_S 16
#define TLS_KEYCTX_RXCIPH_MODE_M 0xf
#define TLS_KEYCTX_RXCIPH_MODE_V(x) ((x) << TLS_KEYCTX_RXCIPH_MODE_S)
#define TLS_KEYCTX_RXAUTH_MODE_S 12
#define TLS_KEYCTX_RXAUTH_MODE_M 0xf
#define TLS_KEYCTX_RXAUTH_MODE_V(x) ((x) << TLS_KEYCTX_RXAUTH_MODE_S)
#define TLS_KEYCTX_RXCIAU_CTRL_S 11
#define TLS_KEYCTX_RXCIAU_CTRL_V(x) ((x) << TLS_KEYCTX_RXCIAU_CTRL_S)
#define TLS_KEYCTX_RX_SEQCTR_S 9
#define TLS_KEYCTX_RX_SEQCTR_M 0x3
#define TLS_KEYCTX_RX_SEQCTR_V(x) ((x) << TLS_KEYCTX_RX_SEQCTR_S)
#define TLS_KEYCTX_RX_VALID_S 8
#define TLS_KEYCTX_RX_VALID_V(x) ((x) << TLS_KEYCTX_RX_VALID_S)
#define TLS_KEYCTX_RXCK_SIZE_S 3
#define TLS_KEYCTX_RXCK_SIZE_M 0x7
#define TLS_KEYCTX_RXCK_SIZE_V(x) ((x) << TLS_KEYCTX_RXCK_SIZE_S)
#define TLS_KEYCTX_RXMK_SIZE_S 0
#define TLS_KEYCTX_RXMK_SIZE_M 0x7
#define TLS_KEYCTX_RXMK_SIZE_V(x) ((x) << TLS_KEYCTX_RXMK_SIZE_S)
#define CHCR_HASH_MAX_DIGEST_SIZE 64
#define CHCR_MAX_SHA_DIGEST_SIZE 64
@ -176,6 +209,15 @@
KEY_CONTEXT_SALT_PRESENT_V(1) | \
KEY_CONTEXT_CTX_LEN_V((ctx_len)))
#define FILL_KEY_CRX_HDR(ck_size, mk_size, d_ck, opad, ctx_len) \
htonl(TLS_KEYCTX_RXMK_SIZE_V(mk_size) | \
TLS_KEYCTX_RXCK_SIZE_V(ck_size) | \
TLS_KEYCTX_RX_VALID_V(1) | \
TLS_KEYCTX_RX_SEQCTR_V(3) | \
TLS_KEYCTX_RXAUTH_MODE_V(4) | \
TLS_KEYCTX_RXCIPH_MODE_V(2) | \
TLS_KEYCTX_RXFLIT_CNT_V((ctx_len)))
#define FILL_WR_OP_CCTX_SIZE \
htonl( \
FW_CRYPTO_LOOKASIDE_WR_OPCODE_V( \

55
drivers/crypto/chelsio/chcr_core.h
View File

@ -65,10 +65,58 @@ struct uld_ctx;
struct _key_ctx {
__be32 ctx_hdr;
u8 salt[MAX_SALT];
__be64 reserverd;
__be64 iv_to_auth;
unsigned char key[0];
};
#define KEYCTX_TX_WR_IV_S 55
#define KEYCTX_TX_WR_IV_M 0x1ffULL
#define KEYCTX_TX_WR_IV_V(x) ((x) << KEYCTX_TX_WR_IV_S)
#define KEYCTX_TX_WR_IV_G(x) \
(((x) >> KEYCTX_TX_WR_IV_S) & KEYCTX_TX_WR_IV_M)
#define KEYCTX_TX_WR_AAD_S 47
#define KEYCTX_TX_WR_AAD_M 0xffULL
#define KEYCTX_TX_WR_AAD_V(x) ((x) << KEYCTX_TX_WR_AAD_S)
#define KEYCTX_TX_WR_AAD_G(x) (((x) >> KEYCTX_TX_WR_AAD_S) & \
KEYCTX_TX_WR_AAD_M)
#define KEYCTX_TX_WR_AADST_S 39
#define KEYCTX_TX_WR_AADST_M 0xffULL
#define KEYCTX_TX_WR_AADST_V(x) ((x) << KEYCTX_TX_WR_AADST_S)
#define KEYCTX_TX_WR_AADST_G(x) \
(((x) >> KEYCTX_TX_WR_AADST_S) & KEYCTX_TX_WR_AADST_M)
#define KEYCTX_TX_WR_CIPHER_S 30
#define KEYCTX_TX_WR_CIPHER_M 0x1ffULL
#define KEYCTX_TX_WR_CIPHER_V(x) ((x) << KEYCTX_TX_WR_CIPHER_S)
#define KEYCTX_TX_WR_CIPHER_G(x) \
(((x) >> KEYCTX_TX_WR_CIPHER_S) & KEYCTX_TX_WR_CIPHER_M)
#define KEYCTX_TX_WR_CIPHERST_S 23
#define KEYCTX_TX_WR_CIPHERST_M 0x7f
#define KEYCTX_TX_WR_CIPHERST_V(x) ((x) << KEYCTX_TX_WR_CIPHERST_S)
#define KEYCTX_TX_WR_CIPHERST_G(x) \
(((x) >> KEYCTX_TX_WR_CIPHERST_S) & KEYCTX_TX_WR_CIPHERST_M)
#define KEYCTX_TX_WR_AUTH_S 14
#define KEYCTX_TX_WR_AUTH_M 0x1ff
#define KEYCTX_TX_WR_AUTH_V(x) ((x) << KEYCTX_TX_WR_AUTH_S)
#define KEYCTX_TX_WR_AUTH_G(x) \
(((x) >> KEYCTX_TX_WR_AUTH_S) & KEYCTX_TX_WR_AUTH_M)
#define KEYCTX_TX_WR_AUTHST_S 7
#define KEYCTX_TX_WR_AUTHST_M 0x7f
#define KEYCTX_TX_WR_AUTHST_V(x) ((x) << KEYCTX_TX_WR_AUTHST_S)
#define KEYCTX_TX_WR_AUTHST_G(x) \
(((x) >> KEYCTX_TX_WR_AUTHST_S) & KEYCTX_TX_WR_AUTHST_M)
#define KEYCTX_TX_WR_AUTHIN_S 0
#define KEYCTX_TX_WR_AUTHIN_M 0x7f
#define KEYCTX_TX_WR_AUTHIN_V(x) ((x) << KEYCTX_TX_WR_AUTHIN_S)
#define KEYCTX_TX_WR_AUTHIN_G(x) \
(((x) >> KEYCTX_TX_WR_AUTHIN_S) & KEYCTX_TX_WR_AUTHIN_M)
struct chcr_wr {
struct fw_crypto_lookaside_wr wreq;
struct ulp_txpkt ulptx;
@ -90,6 +138,11 @@ struct uld_ctx {
struct chcr_dev *dev;
};
struct sge_opaque_hdr {
void *dev;
dma_addr_t addr[MAX_SKB_FRAGS + 1];
};
struct chcr_ipsec_req {
struct ulp_txpkt ulptx;
struct ulptx_idata sc_imm;

4
drivers/crypto/chelsio/chtls/Makefile Normal file
View File

@ -0,0 +1,4 @@
ccflags-y := -Idrivers/net/ethernet/chelsio/cxgb4 -Idrivers/crypto/chelsio/
obj-$(CONFIG_CRYPTO_DEV_CHELSIO_TLS) += chtls.o
chtls-objs := chtls_main.o chtls_cm.o chtls_io.o chtls_hw.o

482
drivers/crypto/chelsio/chtls/chtls.h Normal file
View File

@ -0,0 +1,482 @@
/*
* Copyright (c) 2018 Chelsio Communications, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef __CHTLS_H__
#define __CHTLS_H__
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/hash.h>
#include <crypto/sha.h>
#include <crypto/authenc.h>
#include <crypto/ctr.h>
#include <crypto/gf128mul.h>
#include <crypto/internal/aead.h>
#include <crypto/null.h>
#include <crypto/internal/skcipher.h>
#include <crypto/aead.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/hash.h>
#include <linux/tls.h>
#include <net/tls.h>
#include "t4fw_api.h"
#include "t4_msg.h"
#include "cxgb4.h"
#include "cxgb4_uld.h"
#include "l2t.h"
#include "chcr_algo.h"
#include "chcr_core.h"
#include "chcr_crypto.h"
#define MAX_IVS_PAGE 256
#define TLS_KEY_CONTEXT_SZ 64
#define CIPHER_BLOCK_SIZE 16
#define GCM_TAG_SIZE 16
#define KEY_ON_MEM_SZ 16
#define AEAD_EXPLICIT_DATA_SIZE 8
#define TLS_HEADER_LENGTH 5
#define SCMD_CIPH_MODE_AES_GCM 2
/* Any MFS size should work and come from openssl */
#define TLS_MFS 16384
#define RSS_HDR sizeof(struct rss_header)
#define TLS_WR_CPL_LEN \
(sizeof(struct fw_tlstx_data_wr) + sizeof(struct cpl_tx_tls_sfo))
enum {
CHTLS_KEY_CONTEXT_DSGL,
CHTLS_KEY_CONTEXT_IMM,
CHTLS_KEY_CONTEXT_DDR,
};
enum {
CHTLS_LISTEN_START,
CHTLS_LISTEN_STOP,
};
/* Flags for return value of CPL message handlers */
enum {
CPL_RET_BUF_DONE = 1, /* buffer processing done */
CPL_RET_BAD_MSG = 2, /* bad CPL message */
CPL_RET_UNKNOWN_TID = 4 /* unexpected unknown TID */
};
#define TLS_RCV_ST_READ_HEADER 0xF0
#define TLS_RCV_ST_READ_BODY 0xF1
#define TLS_RCV_ST_READ_DONE 0xF2
#define TLS_RCV_ST_READ_NB 0xF3
#define LISTEN_INFO_HASH_SIZE 32
#define RSPQ_HASH_BITS 5
struct listen_info {
struct listen_info *next; /* Link to next entry */
struct sock *sk; /* The listening socket */
unsigned int stid; /* The server TID */
};
enum {
T4_LISTEN_START_PENDING,
T4_LISTEN_STARTED
};
enum csk_flags {
CSK_CALLBACKS_CHKD, /* socket callbacks have been sanitized */
CSK_ABORT_REQ_RCVD, /* received one ABORT_REQ_RSS message */
CSK_TX_MORE_DATA, /* sending ULP data; don't set SHOVE bit */
CSK_TX_WAIT_IDLE, /* suspend Tx until in-flight data is ACKed */
CSK_ABORT_SHUTDOWN, /* shouldn't send more abort requests */
CSK_ABORT_RPL_PENDING, /* expecting an abort reply */
CSK_CLOSE_CON_REQUESTED,/* we've sent a close_conn_req */
CSK_TX_DATA_SENT, /* sent a TX_DATA WR on this connection */
CSK_TX_FAILOVER, /* Tx traffic failing over */
CSK_UPDATE_RCV_WND, /* Need to update rcv window */
CSK_RST_ABORTED, /* outgoing RST was aborted */
CSK_TLS_HANDSHK, /* TLS Handshake */
CSK_CONN_INLINE, /* Connection on HW */
};
struct listen_ctx {
struct sock *lsk;
struct chtls_dev *cdev;
struct sk_buff_head synq;
u32 state;
};
struct key_map {
unsigned long *addr;
unsigned int start;
unsigned int available;
unsigned int size;
spinlock_t lock; /* lock for key id request from map */
} __packed;
struct tls_scmd {
u32 seqno_numivs;
u32 ivgen_hdrlen;
};
struct chtls_dev {
struct tls_device tlsdev;
struct list_head list;
struct cxgb4_lld_info *lldi;
struct pci_dev *pdev;
struct listen_info *listen_hash_tab[LISTEN_INFO_HASH_SIZE];
spinlock_t listen_lock; /* lock for listen list */
struct net_device **ports;
struct tid_info *tids;
unsigned int pfvf;
const unsigned short *mtus;
struct idr hwtid_idr;
struct idr stid_idr;
spinlock_t idr_lock ____cacheline_aligned_in_smp;
struct net_device *egr_dev[NCHAN * 2];
struct sk_buff *rspq_skb_cache[1 << RSPQ_HASH_BITS];
struct sk_buff *askb;
struct sk_buff_head deferq;
struct work_struct deferq_task;
struct list_head list_node;
struct list_head rcu_node;
struct list_head na_node;
unsigned int send_page_order;
struct key_map kmap;
};
struct chtls_hws {
struct sk_buff_head sk_recv_queue;
u8 txqid;
u8 ofld;
u16 type;
u16 rstate;
u16 keyrpl;
u16 pldlen;
u16 rcvpld;
u16 compute;
u16 expansion;
u16 keylen;
u16 pdus;
u16 adjustlen;
u16 ivsize;
u16 txleft;
u32 mfs;
s32 txkey;
s32 rxkey;
u32 fcplenmax;
u32 copied_seq;
u64 tx_seq_no;
struct tls_scmd scmd;
struct tls12_crypto_info_aes_gcm_128 crypto_info;
};
struct chtls_sock {
struct sock *sk;
struct chtls_dev *cdev;
struct l2t_entry *l2t_entry; /* pointer to the L2T entry */
struct net_device *egress_dev; /* TX_CHAN for act open retry */
struct sk_buff_head txq;
struct sk_buff *wr_skb_head;
struct sk_buff *wr_skb_tail;
struct sk_buff *ctrl_skb_cache;
struct sk_buff *txdata_skb_cache; /* abort path messages */
struct kref kref;
unsigned long flags;
u32 opt2;
u32 wr_credits;
u32 wr_unacked;
u32 wr_max_credits;
u32 wr_nondata;
u32 hwtid; /* TCP Control Block ID */
u32 txq_idx;
u32 rss_qid;
u32 tid;
u32 idr;
u32 mss;
u32 ulp_mode;
u32 tx_chan;
u32 rx_chan;
u32 sndbuf;
u32 txplen_max;
u32 mtu_idx; /* MTU table index */
u32 smac_idx;
u8 port_id;
u8 tos;
u16 resv2;
u32 delack_mode;
u32 delack_seq;
void *passive_reap_next; /* placeholder for passive */
struct chtls_hws tlshws;
struct synq {
struct sk_buff *next;
struct sk_buff *prev;
} synq;
struct listen_ctx *listen_ctx;
};
struct tls_hdr {
u8 type;
u16 version;
u16 length;
} __packed;
struct tlsrx_cmp_hdr {
u8 type;
u16 version;
u16 length;
u64 tls_seq;
u16 reserved1;
u8 res_to_mac_error;
} __packed;
/* res_to_mac_error fields */
#define TLSRX_HDR_PKT_INT_ERROR_S 4
#define TLSRX_HDR_PKT_INT_ERROR_M 0x1
#define TLSRX_HDR_PKT_INT_ERROR_V(x) \
((x) << TLSRX_HDR_PKT_INT_ERROR_S)
#define TLSRX_HDR_PKT_INT_ERROR_G(x) \
(((x) >> TLSRX_HDR_PKT_INT_ERROR_S) & TLSRX_HDR_PKT_INT_ERROR_M)
#define TLSRX_HDR_PKT_INT_ERROR_F TLSRX_HDR_PKT_INT_ERROR_V(1U)
#define TLSRX_HDR_PKT_SPP_ERROR_S 3
#define TLSRX_HDR_PKT_SPP_ERROR_M 0x1
#define TLSRX_HDR_PKT_SPP_ERROR_V(x) ((x) << TLSRX_HDR_PKT_SPP_ERROR)
#define TLSRX_HDR_PKT_SPP_ERROR_G(x) \
(((x) >> TLSRX_HDR_PKT_SPP_ERROR_S) & TLSRX_HDR_PKT_SPP_ERROR_M)
#define TLSRX_HDR_PKT_SPP_ERROR_F TLSRX_HDR_PKT_SPP_ERROR_V(1U)
#define TLSRX_HDR_PKT_CCDX_ERROR_S 2
#define TLSRX_HDR_PKT_CCDX_ERROR_M 0x1
#define TLSRX_HDR_PKT_CCDX_ERROR_V(x) ((x) << TLSRX_HDR_PKT_CCDX_ERROR_S)
#define TLSRX_HDR_PKT_CCDX_ERROR_G(x) \
(((x) >> TLSRX_HDR_PKT_CCDX_ERROR_S) & TLSRX_HDR_PKT_CCDX_ERROR_M)
#define TLSRX_HDR_PKT_CCDX_ERROR_F TLSRX_HDR_PKT_CCDX_ERROR_V(1U)
#define TLSRX_HDR_PKT_PAD_ERROR_S 1
#define TLSRX_HDR_PKT_PAD_ERROR_M 0x1
#define TLSRX_HDR_PKT_PAD_ERROR_V(x) ((x) << TLSRX_HDR_PKT_PAD_ERROR_S)
#define TLSRX_HDR_PKT_PAD_ERROR_G(x) \
(((x) >> TLSRX_HDR_PKT_PAD_ERROR_S) & TLSRX_HDR_PKT_PAD_ERROR_M)
#define TLSRX_HDR_PKT_PAD_ERROR_F TLSRX_HDR_PKT_PAD_ERROR_V(1U)
#define TLSRX_HDR_PKT_MAC_ERROR_S 0
#define TLSRX_HDR_PKT_MAC_ERROR_M 0x1
#define TLSRX_HDR_PKT_MAC_ERROR_V(x) ((x) << TLSRX_HDR_PKT_MAC_ERROR)
#define TLSRX_HDR_PKT_MAC_ERROR_G(x) \
(((x) >> S_TLSRX_HDR_PKT_MAC_ERROR_S) & TLSRX_HDR_PKT_MAC_ERROR_M)
#define TLSRX_HDR_PKT_MAC_ERROR_F TLSRX_HDR_PKT_MAC_ERROR_V(1U)
#define TLSRX_HDR_PKT_ERROR_M 0x1F
struct ulp_mem_rw {
__be32 cmd;
__be32 len16; /* command length */
__be32 dlen; /* data length in 32-byte units */
__be32 lock_addr;
};
struct tls_key_wr {
__be32 op_to_compl;
__be32 flowid_len16;
__be32 ftid;
u8 reneg_to_write_rx;
u8 protocol;
__be16 mfs;
};
struct tls_key_req {
struct tls_key_wr wr;
struct ulp_mem_rw req;
struct ulptx_idata sc_imm;
};
/*
* This lives in skb->cb and is used to chain WRs in a linked list.
*/
struct wr_skb_cb {
struct l2t_skb_cb l2t; /* reserve space for l2t CB */
struct sk_buff *next_wr; /* next write request */
};
/* Per-skb backlog handler. Run when a socket's backlog is processed. */
struct blog_skb_cb {
void (*backlog_rcv)(struct sock *sk, struct sk_buff *skb);
struct chtls_dev *cdev;
};
/*
* Similar to tcp_skb_cb but with ULP elements added to support TLS,
* etc.
*/
struct ulp_skb_cb {
struct wr_skb_cb wr; /* reserve space for write request */
u16 flags; /* TCP-like flags */
u8 psh;
u8 ulp_mode; /* ULP mode/submode of sk_buff */
u32 seq; /* TCP sequence number */
union { /* ULP-specific fields */
struct {
u8 type;
u8 ofld;
u8 iv;
} tls;
} ulp;
};
#define ULP_SKB_CB(skb) ((struct ulp_skb_cb *)&((skb)->cb[0]))
#define BLOG_SKB_CB(skb) ((struct blog_skb_cb *)(skb)->cb)
/*
* Flags for ulp_skb_cb.flags.
*/
enum {
ULPCB_FLAG_NEED_HDR = 1 << 0, /* packet needs a TX_DATA_WR header */
ULPCB_FLAG_NO_APPEND = 1 << 1, /* don't grow this skb */
ULPCB_FLAG_BARRIER = 1 << 2, /* set TX_WAIT_IDLE after sending */
ULPCB_FLAG_HOLD = 1 << 3, /* skb not ready for Tx yet */
ULPCB_FLAG_COMPL = 1 << 4, /* request WR completion */
ULPCB_FLAG_URG = 1 << 5, /* urgent data */
ULPCB_FLAG_TLS_ND = 1 << 6, /* payload of zero length */
ULPCB_FLAG_NO_HDR = 1 << 7, /* not a ofld wr */
};
/* The ULP mode/submode of an skbuff */
#define skb_ulp_mode(skb) (ULP_SKB_CB(skb)->ulp_mode)
#define TCP_PAGE(sk) (sk->sk_frag.page)
#define TCP_OFF(sk) (sk->sk_frag.offset)
static inline struct chtls_dev *to_chtls_dev(struct tls_device *tlsdev)
{
return container_of(tlsdev, struct chtls_dev, tlsdev);
}
static inline void csk_set_flag(struct chtls_sock *csk,
enum csk_flags flag)
{
__set_bit(flag, &csk->flags);
}
static inline void csk_reset_flag(struct chtls_sock *csk,
enum csk_flags flag)
{
__clear_bit(flag, &csk->flags);
}
static inline bool csk_conn_inline(const struct chtls_sock *csk)
{
return test_bit(CSK_CONN_INLINE, &csk->flags);
}
static inline int csk_flag(const struct sock *sk, enum csk_flags flag)
{
struct chtls_sock *csk = rcu_dereference_sk_user_data(sk);
if (!csk_conn_inline(csk))
return 0;
return test_bit(flag, &csk->flags);
}
static inline int csk_flag_nochk(const struct chtls_sock *csk,
enum csk_flags flag)
{
return test_bit(flag, &csk->flags);
}
static inline void *cplhdr(struct sk_buff *skb)
{
return skb->data;
}
static inline int is_neg_adv(unsigned int status)
{
return status == CPL_ERR_RTX_NEG_ADVICE ||
status == CPL_ERR_KEEPALV_NEG_ADVICE ||
status == CPL_ERR_PERSIST_NEG_ADVICE;
}
static inline void process_cpl_msg(void (*fn)(struct sock *, struct sk_buff *),
struct sock *sk,
struct sk_buff *skb)
{
skb_reset_mac_header(skb);
skb_reset_network_header(skb);
skb_reset_transport_header(skb);
bh_lock_sock(sk);
if (unlikely(sock_owned_by_user(sk))) {
BLOG_SKB_CB(skb)->backlog_rcv = fn;
__sk_add_backlog(sk, skb);
} else {
fn(sk, skb);
}
bh_unlock_sock(sk);
}
static inline void chtls_sock_free(struct kref *ref)
{
struct chtls_sock *csk = container_of(ref, struct chtls_sock,
kref);
kfree(csk);
}
static inline void __chtls_sock_put(const char *fn, struct chtls_sock *csk)
{
kref_put(&csk->kref, chtls_sock_free);
}
static inline void __chtls_sock_get(const char *fn,
struct chtls_sock *csk)
{
kref_get(&csk->kref);
}
static inline void send_or_defer(struct sock *sk, struct tcp_sock *tp,
struct sk_buff *skb, int through_l2t)
{
struct chtls_sock *csk = rcu_dereference_sk_user_data(sk);
if (through_l2t) {
/* send through L2T */
cxgb4_l2t_send(csk->egress_dev, skb, csk->l2t_entry);
} else {
/* send directly */
cxgb4_ofld_send(csk->egress_dev, skb);
}
}
typedef int (*chtls_handler_func)(struct chtls_dev *, struct sk_buff *);
extern chtls_handler_func chtls_handlers[NUM_CPL_CMDS];
void chtls_install_cpl_ops(struct sock *sk);
int chtls_init_kmap(struct chtls_dev *cdev, struct cxgb4_lld_info *lldi);
void chtls_listen_stop(struct chtls_dev *cdev, struct sock *sk);
int chtls_listen_start(struct chtls_dev *cdev, struct sock *sk);
void chtls_close(struct sock *sk, long timeout);
int chtls_disconnect(struct sock *sk, int flags);
void chtls_shutdown(struct sock *sk, int how);
void chtls_destroy_sock(struct sock *sk);
int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
int chtls_recvmsg(struct sock *sk, struct msghdr *msg,
size_t len, int nonblock, int flags, int *addr_len);
int chtls_sendpage(struct sock *sk, struct page *page,
int offset, size_t size, int flags);
int send_tx_flowc_wr(struct sock *sk, int compl,
u32 snd_nxt, u32 rcv_nxt);
void chtls_tcp_push(struct sock *sk, int flags);
int chtls_push_frames(struct chtls_sock *csk, int comp);
int chtls_set_tcb_tflag(struct sock *sk, unsigned int bit_pos, int val);
int chtls_setkey(struct chtls_sock *csk, u32 keylen, u32 mode);
void skb_entail(struct sock *sk, struct sk_buff *skb, int flags);
unsigned int keyid_to_addr(int start_addr, int keyid);
void free_tls_keyid(struct sock *sk);
#endif

2126
drivers/crypto/chelsio/chtls/chtls_cm.c Normal file
View File

File diff suppressed because it is too large Load Diff

203
drivers/crypto/chelsio/chtls/chtls_cm.h Normal file
View File

@ -0,0 +1,203 @@
/*
* Copyright (c) 2018 Chelsio Communications, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef __CHTLS_CM_H__
#define __CHTLS_CM_H__
/*
* TCB settings
*/
/* 3:0 */
#define TCB_ULP_TYPE_W 0
#define TCB_ULP_TYPE_S 0
#define TCB_ULP_TYPE_M 0xfULL
#define TCB_ULP_TYPE_V(x) ((x) << TCB_ULP_TYPE_S)
/* 11:4 */
#define TCB_ULP_RAW_W 0
#define TCB_ULP_RAW_S 4
#define TCB_ULP_RAW_M 0xffULL
#define TCB_ULP_RAW_V(x) ((x) << TCB_ULP_RAW_S)
#define TF_TLS_KEY_SIZE_S 7
#define TF_TLS_KEY_SIZE_V(x) ((x) << TF_TLS_KEY_SIZE_S)
#define TF_TLS_CONTROL_S 2
#define TF_TLS_CONTROL_V(x) ((x) << TF_TLS_CONTROL_S)
#define TF_TLS_ACTIVE_S 1
#define TF_TLS_ACTIVE_V(x) ((x) << TF_TLS_ACTIVE_S)
#define TF_TLS_ENABLE_S 0
#define TF_TLS_ENABLE_V(x) ((x) << TF_TLS_ENABLE_S)
#define TF_RX_QUIESCE_S 15
#define TF_RX_QUIESCE_V(x) ((x) << TF_RX_QUIESCE_S)
/*
* Max receive window supported by HW in bytes. Only a small part of it can
* be set through option0, the rest needs to be set through RX_DATA_ACK.
*/
#define MAX_RCV_WND ((1U << 27) - 1)
#define MAX_MSS 65536
/*
* Min receive window. We want it to be large enough to accommodate receive
* coalescing, handle jumbo frames, and not trigger sender SWS avoidance.
*/
#define MIN_RCV_WND (24 * 1024U)
#define LOOPBACK(x) (((x) & htonl(0xff000000)) == htonl(0x7f000000))
/* ulp_mem_io + ulptx_idata + payload + padding */
#define MAX_IMM_ULPTX_WR_LEN (32 + 8 + 256 + 8)
/* for TX: a skb must have a headroom of at least TX_HEADER_LEN bytes */
#define TX_HEADER_LEN \
(sizeof(struct fw_ofld_tx_data_wr) + sizeof(struct sge_opaque_hdr))
#define TX_TLSHDR_LEN \
(sizeof(struct fw_tlstx_data_wr) + sizeof(struct cpl_tx_tls_sfo) + \
sizeof(struct sge_opaque_hdr))
#define TXDATA_SKB_LEN 128
enum {
CPL_TX_TLS_SFO_TYPE_CCS,
CPL_TX_TLS_SFO_TYPE_ALERT,
CPL_TX_TLS_SFO_TYPE_HANDSHAKE,
CPL_TX_TLS_SFO_TYPE_DATA,
CPL_TX_TLS_SFO_TYPE_HEARTBEAT,
};
enum {
TLS_HDR_TYPE_CCS = 20,
TLS_HDR_TYPE_ALERT,
TLS_HDR_TYPE_HANDSHAKE,
TLS_HDR_TYPE_RECORD,
TLS_HDR_TYPE_HEARTBEAT,
};
typedef void (*defer_handler_t)(struct chtls_dev *dev, struct sk_buff *skb);
extern struct request_sock_ops chtls_rsk_ops;
struct deferred_skb_cb {
defer_handler_t handler;
struct chtls_dev *dev;
};
#define DEFERRED_SKB_CB(skb) ((struct deferred_skb_cb *)(skb)->cb)
#define failover_flowc_wr_len offsetof(struct fw_flowc_wr, mnemval[3])
#define WR_SKB_CB(skb) ((struct wr_skb_cb *)(skb)->cb)
#define ACCEPT_QUEUE(sk) (&inet_csk(sk)->icsk_accept_queue.rskq_accept_head)
#define SND_WSCALE(tp) ((tp)->rx_opt.snd_wscale)
#define RCV_WSCALE(tp) ((tp)->rx_opt.rcv_wscale)
#define USER_MSS(tp) ((tp)->rx_opt.user_mss)
#define TS_RECENT_STAMP(tp) ((tp)->rx_opt.ts_recent_stamp)
#define WSCALE_OK(tp) ((tp)->rx_opt.wscale_ok)
#define TSTAMP_OK(tp) ((tp)->rx_opt.tstamp_ok)
#define SACK_OK(tp) ((tp)->rx_opt.sack_ok)
#define INC_ORPHAN_COUNT(sk) percpu_counter_inc((sk)->sk_prot->orphan_count)
/* TLS SKB */
#define skb_ulp_tls_inline(skb) (ULP_SKB_CB(skb)->ulp.tls.ofld)
#define skb_ulp_tls_iv_imm(skb) (ULP_SKB_CB(skb)->ulp.tls.iv)
void chtls_defer_reply(struct sk_buff *skb, struct chtls_dev *dev,
defer_handler_t handler);
/*
* Returns true if the socket is in one of the supplied states.
*/
static inline unsigned int sk_in_state(const struct sock *sk,
unsigned int states)
{
return states & (1 << sk->sk_state);
}
static void chtls_rsk_destructor(struct request_sock *req)
{
/* do nothing */
}
static inline void chtls_init_rsk_ops(struct proto *chtls_tcp_prot,
struct request_sock_ops *chtls_tcp_ops,
struct proto *tcp_prot, int family)
{
memset(chtls_tcp_ops, 0, sizeof(*chtls_tcp_ops));
chtls_tcp_ops->family = family;
chtls_tcp_ops->obj_size = sizeof(struct tcp_request_sock);
chtls_tcp_ops->destructor = chtls_rsk_destructor;
chtls_tcp_ops->slab = tcp_prot->rsk_prot->slab;
chtls_tcp_prot->rsk_prot = chtls_tcp_ops;
}
static inline void chtls_reqsk_free(struct request_sock *req)
{
if (req->rsk_listener)
sock_put(req->rsk_listener);
kmem_cache_free(req->rsk_ops->slab, req);
}
#define DECLARE_TASK_FUNC(task, task_param) \
static void task(struct work_struct *task_param)
static inline void sk_wakeup_sleepers(struct sock *sk, bool interruptable)
{
struct socket_wq *wq;
rcu_read_lock();
wq = rcu_dereference(sk->sk_wq);
if (skwq_has_sleeper(wq)) {
if (interruptable)
wake_up_interruptible(sk_sleep(sk));
else
wake_up_all(sk_sleep(sk));
}
rcu_read_unlock();
}
static inline void chtls_set_req_port(struct request_sock *oreq,
__be16 source, __be16 dest)
{
inet_rsk(oreq)->ir_rmt_port = source;
inet_rsk(oreq)->ir_num = ntohs(dest);
}
static inline void chtls_set_req_addr(struct request_sock *oreq,
__be32 local_ip, __be32 peer_ip)
{
inet_rsk(oreq)->ir_loc_addr = local_ip;
inet_rsk(oreq)->ir_rmt_addr = peer_ip;
}
static inline void chtls_free_skb(struct sock *sk, struct sk_buff *skb)
{
skb_dst_set(skb, NULL);
__skb_unlink(skb, &sk->sk_receive_queue);
__kfree_skb(skb);
}
static inline void chtls_kfree_skb(struct sock *sk, struct sk_buff *skb)
{
skb_dst_set(skb, NULL);
__skb_unlink(skb, &sk->sk_receive_queue);
kfree_skb(skb);
}
static inline void enqueue_wr(struct chtls_sock *csk, struct sk_buff *skb)
{
WR_SKB_CB(skb)->next_wr = NULL;
skb_get(skb);
if (!csk->wr_skb_head)
csk->wr_skb_head = skb;
else
WR_SKB_CB(csk->wr_skb_tail)->next_wr = skb;
csk->wr_skb_tail = skb;
}
#endif

412
drivers/crypto/chelsio/chtls/chtls_hw.c Normal file
View File

@ -0,0 +1,412 @@
/*
* Copyright (c) 2018 Chelsio Communications, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* Written by: Atul Gupta (atul.gupta@chelsio.com)
*/
#include <linux/module.h>
#include <linux/list.h>
#include <linux/workqueue.h>
#include <linux/skbuff.h>
#include <linux/timer.h>
#include <linux/notifier.h>
#include <linux/inetdevice.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/tls.h>
#include <net/tls.h>
#include "chtls.h"
#include "chtls_cm.h"
static void __set_tcb_field_direct(struct chtls_sock *csk,
struct cpl_set_tcb_field *req, u16 word,
u64 mask, u64 val, u8 cookie, int no_reply)
{
struct ulptx_idata *sc;
INIT_TP_WR_CPL(req, CPL_SET_TCB_FIELD, csk->tid);
req->wr.wr_mid |= htonl(FW_WR_FLOWID_V(csk->tid));
req->reply_ctrl = htons(NO_REPLY_V(no_reply) |
QUEUENO_V(csk->rss_qid));
req->word_cookie = htons(TCB_WORD_V(word) | TCB_COOKIE_V(cookie));
req->mask = cpu_to_be64(mask);
req->val = cpu_to_be64(val);
sc = (struct ulptx_idata *)(req + 1);
sc->cmd_more = htonl(ULPTX_CMD_V(ULP_TX_SC_NOOP));
sc->len = htonl(0);
}
static void __set_tcb_field(struct sock *sk, struct sk_buff *skb, u16 word,
u64 mask, u64 val, u8 cookie, int no_reply)
{
struct cpl_set_tcb_field *req;
struct chtls_sock *csk;
struct ulptx_idata *sc;
unsigned int wrlen;
wrlen = roundup(sizeof(*req) + sizeof(*sc), 16);
csk = rcu_dereference_sk_user_data(sk);
req = (struct cpl_set_tcb_field *)__skb_put(skb, wrlen);
__set_tcb_field_direct(csk, req, word, mask, val, cookie, no_reply);
set_wr_txq(skb, CPL_PRIORITY_CONTROL, csk->port_id);
}
/*
* Send control message to HW, message go as immediate data and packet
* is freed immediately.
*/
static int chtls_set_tcb_field(struct sock *sk, u16 word, u64 mask, u64 val)
{
struct cpl_set_tcb_field *req;
unsigned int credits_needed;
struct chtls_sock *csk;
struct ulptx_idata *sc;
struct sk_buff *skb;
unsigned int wrlen;
int ret;
wrlen = roundup(sizeof(*req) + sizeof(*sc), 16);
skb = alloc_skb(wrlen, GFP_ATOMIC);
if (!skb)
return -ENOMEM;
credits_needed = DIV_ROUND_UP(wrlen, 16);
csk = rcu_dereference_sk_user_data(sk);
__set_tcb_field(sk, skb, word, mask, val, 0, 1);
skb_set_queue_mapping(skb, (csk->txq_idx << 1) | CPL_PRIORITY_DATA);
csk->wr_credits -= credits_needed;
csk->wr_unacked += credits_needed;
enqueue_wr(csk, skb);
ret = cxgb4_ofld_send(csk->egress_dev, skb);
if (ret < 0)
kfree_skb(skb);
return ret < 0 ? ret : 0;
}
/*
* Set one of the t_flags bits in the TCB.
*/
int chtls_set_tcb_tflag(struct sock *sk, unsigned int bit_pos, int val)
{
return chtls_set_tcb_field(sk, 1, 1ULL << bit_pos,
val << bit_pos);
}
static int chtls_set_tcb_keyid(struct sock *sk, int keyid)
{
return chtls_set_tcb_field(sk, 31, 0xFFFFFFFFULL, keyid);
}
static int chtls_set_tcb_seqno(struct sock *sk)
{
return chtls_set_tcb_field(sk, 28, ~0ULL, 0);
}
static int chtls_set_tcb_quiesce(struct sock *sk, int val)
{
return chtls_set_tcb_field(sk, 1, (1ULL << TF_RX_QUIESCE_S),
TF_RX_QUIESCE_V(val));
}
/* TLS Key bitmap processing */
int chtls_init_kmap(struct chtls_dev *cdev, struct cxgb4_lld_info *lldi)
{
unsigned int num_key_ctx, bsize;
int ksize;
num_key_ctx = (lldi->vr->key.size / TLS_KEY_CONTEXT_SZ);
bsize = BITS_TO_LONGS(num_key_ctx);
cdev->kmap.size = num_key_ctx;
cdev->kmap.available = bsize;
ksize = sizeof(*cdev->kmap.addr) * bsize;
cdev->kmap.addr = kvzalloc(ksize, GFP_KERNEL);
if (!cdev->kmap.addr)
return -ENOMEM;
cdev->kmap.start = lldi->vr->key.start;
spin_lock_init(&cdev->kmap.lock);
return 0;
}
static int get_new_keyid(struct chtls_sock *csk, u32 optname)
{
struct net_device *dev = csk->egress_dev;
struct chtls_dev *cdev = csk->cdev;
struct chtls_hws *hws;
struct adapter *adap;
int keyid;
adap = netdev2adap(dev);
hws = &csk->tlshws;
spin_lock_bh(&cdev->kmap.lock);
keyid = find_first_zero_bit(cdev->kmap.addr, cdev->kmap.size);
if (keyid < cdev->kmap.size) {
__set_bit(keyid, cdev->kmap.addr);
if (optname == TLS_RX)
hws->rxkey = keyid;
else
hws->txkey = keyid;
atomic_inc(&adap->chcr_stats.tls_key);
} else {
keyid = -1;
}
spin_unlock_bh(&cdev->kmap.lock);
return keyid;
}
void free_tls_keyid(struct sock *sk)
{
struct chtls_sock *csk = rcu_dereference_sk_user_data(sk);
struct net_device *dev = csk->egress_dev;
struct chtls_dev *cdev = csk->cdev;
struct chtls_hws *hws;
struct adapter *adap;
if (!cdev->kmap.addr)
return;
adap = netdev2adap(dev);
hws = &csk->tlshws;
spin_lock_bh(&cdev->kmap.lock);
if (hws->rxkey >= 0) {
__clear_bit(hws->rxkey, cdev->kmap.addr);
atomic_dec(&adap->chcr_stats.tls_key);
hws->rxkey = -1;
}
if (hws->txkey >= 0) {
__clear_bit(hws->txkey, cdev->kmap.addr);
atomic_dec(&adap->chcr_stats.tls_key);
hws->txkey = -1;
}
spin_unlock_bh(&cdev->kmap.lock);
}
unsigned int keyid_to_addr(int start_addr, int keyid)
{
return (start_addr + (keyid * TLS_KEY_CONTEXT_SZ)) >> 5;
}
static void chtls_rxkey_ivauth(struct _key_ctx *kctx)
{
kctx->iv_to_auth = cpu_to_be64(KEYCTX_TX_WR_IV_V(6ULL) |
KEYCTX_TX_WR_AAD_V(1ULL) |
KEYCTX_TX_WR_AADST_V(5ULL) |
KEYCTX_TX_WR_CIPHER_V(14ULL) |
KEYCTX_TX_WR_CIPHERST_V(0ULL) |
KEYCTX_TX_WR_AUTH_V(14ULL) |
KEYCTX_TX_WR_AUTHST_V(16ULL) |
KEYCTX_TX_WR_AUTHIN_V(16ULL));
}
static int chtls_key_info(struct chtls_sock *csk,
struct _key_ctx *kctx,
u32 keylen, u32 optname)
{
unsigned char key[CHCR_KEYCTX_CIPHER_KEY_SIZE_256];
struct tls12_crypto_info_aes_gcm_128 *gcm_ctx;
unsigned char ghash_h[AEAD_H_SIZE];
struct crypto_cipher *cipher;
int ck_size, key_ctx_size;
int ret;
gcm_ctx = (struct tls12_crypto_info_aes_gcm_128 *)
&csk->tlshws.crypto_info;
key_ctx_size = sizeof(struct _key_ctx) +
roundup(keylen, 16) + AEAD_H_SIZE;
if (keylen == AES_KEYSIZE_128) {
ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_128;
} else if (keylen == AES_KEYSIZE_192) {
ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_192;
} else if (keylen == AES_KEYSIZE_256) {
ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_256;
} else {
pr_err("GCM: Invalid key length %d\n", keylen);
return -EINVAL;
}
memcpy(key, gcm_ctx->key, keylen);
/* Calculate the H = CIPH(K, 0 repeated 16 times).
* It will go in key context
*/
cipher = crypto_alloc_cipher("aes", 0, 0);
if (IS_ERR(cipher)) {
ret = -ENOMEM;
goto out;
}
ret = crypto_cipher_setkey(cipher, key, keylen);
if (ret)
goto out1;
memset(ghash_h, 0, AEAD_H_SIZE);
crypto_cipher_encrypt_one(cipher, ghash_h, ghash_h);
csk->tlshws.keylen = key_ctx_size;
/* Copy the Key context */
if (optname == TLS_RX) {
int key_ctx;
key_ctx = ((key_ctx_size >> 4) << 3);
kctx->ctx_hdr = FILL_KEY_CRX_HDR(ck_size,
CHCR_KEYCTX_MAC_KEY_SIZE_128,
0, 0, key_ctx);
chtls_rxkey_ivauth(kctx);
} else {
kctx->ctx_hdr = FILL_KEY_CTX_HDR(ck_size,
CHCR_KEYCTX_MAC_KEY_SIZE_128,
0, 0, key_ctx_size >> 4);
}
memcpy(kctx->salt, gcm_ctx->salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE);
memcpy(kctx->key, gcm_ctx->key, keylen);
memcpy(kctx->key + keylen, ghash_h, AEAD_H_SIZE);
/* erase key info from driver */
memset(gcm_ctx->key, 0, keylen);
out1:
crypto_free_cipher(cipher);
out:
return ret;
}
static void chtls_set_scmd(struct chtls_sock *csk)
{
struct chtls_hws *hws = &csk->tlshws;
hws->scmd.seqno_numivs =
SCMD_SEQ_NO_CTRL_V(3) |
SCMD_PROTO_VERSION_V(0) |
SCMD_ENC_DEC_CTRL_V(0) |
SCMD_CIPH_AUTH_SEQ_CTRL_V(1) |
SCMD_CIPH_MODE_V(2) |
SCMD_AUTH_MODE_V(4) |
SCMD_HMAC_CTRL_V(0) |
SCMD_IV_SIZE_V(4) |
SCMD_NUM_IVS_V(1);
hws->scmd.ivgen_hdrlen =
SCMD_IV_GEN_CTRL_V(1) |
SCMD_KEY_CTX_INLINE_V(0) |
SCMD_TLS_FRAG_ENABLE_V(1);
}
int chtls_setkey(struct chtls_sock *csk, u32 keylen, u32 optname)
{
struct tls_key_req *kwr;
struct chtls_dev *cdev;
struct _key_ctx *kctx;
int wrlen, klen, len;
struct sk_buff *skb;
struct sock *sk;
int keyid;
int kaddr;
int ret;
cdev = csk->cdev;
sk = csk->sk;
klen = roundup((keylen + AEAD_H_SIZE) + sizeof(*kctx), 32);
wrlen = roundup(sizeof(*kwr), 16);
len = klen + wrlen;
/* Flush out-standing data before new key takes effect */
if (optname == TLS_TX) {
lock_sock(sk);
if (skb_queue_len(&csk->txq))
chtls_push_frames(csk, 0);
release_sock(sk);
}
skb = alloc_skb(len, GFP_KERNEL);
if (!skb)
return -ENOMEM;
keyid = get_new_keyid(csk, optname);
if (keyid < 0) {
ret = -ENOSPC;
goto out_nokey;
}
kaddr = keyid_to_addr(cdev->kmap.start, keyid);
kwr = (struct tls_key_req *)__skb_put_zero(skb, len);
kwr->wr.op_to_compl =
cpu_to_be32(FW_WR_OP_V(FW_ULPTX_WR) | FW_WR_COMPL_F |
FW_WR_ATOMIC_V(1U));
kwr->wr.flowid_len16 =
cpu_to_be32(FW_WR_LEN16_V(DIV_ROUND_UP(len, 16) |
FW_WR_FLOWID_V(csk->tid)));
kwr->wr.protocol = 0;
kwr->wr.mfs = htons(TLS_MFS);
kwr->wr.reneg_to_write_rx = optname;
/* ulptx command */
kwr->req.cmd = cpu_to_be32(ULPTX_CMD_V(ULP_TX_MEM_WRITE) |
T5_ULP_MEMIO_ORDER_V(1) |
T5_ULP_MEMIO_IMM_V(1));
kwr->req.len16 = cpu_to_be32((csk->tid << 8) |
DIV_ROUND_UP(len - sizeof(kwr->wr), 16));
kwr->req.dlen = cpu_to_be32(ULP_MEMIO_DATA_LEN_V(klen >> 5));
kwr->req.lock_addr = cpu_to_be32(ULP_MEMIO_ADDR_V(kaddr));
/* sub command */
kwr->sc_imm.cmd_more = cpu_to_be32(ULPTX_CMD_V(ULP_TX_SC_IMM));
kwr->sc_imm.len = cpu_to_be32(klen);
/* key info */
kctx = (struct _key_ctx *)(kwr + 1);
ret = chtls_key_info(csk, kctx, keylen, optname);
if (ret)
goto out_notcb;
set_wr_txq(skb, CPL_PRIORITY_DATA, csk->tlshws.txqid);
csk->wr_credits -= DIV_ROUND_UP(len, 16);
csk->wr_unacked += DIV_ROUND_UP(len, 16);
enqueue_wr(csk, skb);
cxgb4_ofld_send(csk->egress_dev, skb);
chtls_set_scmd(csk);
/* Clear quiesce for Rx key */
if (optname == TLS_RX) {
ret = chtls_set_tcb_keyid(sk, keyid);
if (ret)
goto out_notcb;
ret = chtls_set_tcb_field(sk, 0,
TCB_ULP_RAW_V(TCB_ULP_RAW_M),
TCB_ULP_RAW_V((TF_TLS_KEY_SIZE_V(1) |
TF_TLS_CONTROL_V(1) |
TF_TLS_ACTIVE_V(1) |
TF_TLS_ENABLE_V(1))));
if (ret)
goto out_notcb;
ret = chtls_set_tcb_seqno(sk);
if (ret)
goto out_notcb;
ret = chtls_set_tcb_quiesce(sk, 0);
if (ret)
goto out_notcb;
csk->tlshws.rxkey = keyid;
} else {
csk->tlshws.tx_seq_no = 0;
csk->tlshws.txkey = keyid;
}
return ret;
out_notcb:
free_tls_keyid(sk);
out_nokey:
kfree_skb(skb);
return ret;
}

1822
drivers/crypto/chelsio/chtls/chtls_io.c Normal file
View File

File diff suppressed because it is too large Load Diff

578
drivers/crypto/chelsio/chtls/chtls_main.c Normal file
View File

@ -0,0 +1,578 @@
/*
* Copyright (c) 2018 Chelsio Communications, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* Written by: Atul Gupta (atul.gupta@chelsio.com)
*/
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/skbuff.h>
#include <linux/socket.h>
#include <linux/hash.h>
#include <linux/in.h>
#include <linux/net.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <net/tcp.h>
#include <net/tls.h>
#include "chtls.h"
#include "chtls_cm.h"
#define DRV_NAME "chtls"
/*
* chtls device management
* maintains a list of the chtls devices
*/
static LIST_HEAD(cdev_list);
static DEFINE_MUTEX(cdev_mutex);
static DEFINE_MUTEX(cdev_list_lock);
static DEFINE_MUTEX(notify_mutex);
static RAW_NOTIFIER_HEAD(listen_notify_list);
static struct proto chtls_cpl_prot;
struct request_sock_ops chtls_rsk_ops;
static uint send_page_order = (14 - PAGE_SHIFT < 0) ? 0 : 14 - PAGE_SHIFT;
static void register_listen_notifier(struct notifier_block *nb)
{
mutex_lock(&notify_mutex);
raw_notifier_chain_register(&listen_notify_list, nb);
mutex_unlock(&notify_mutex);
}
static void unregister_listen_notifier(struct notifier_block *nb)
{
mutex_lock(&notify_mutex);
raw_notifier_chain_unregister(&listen_notify_list, nb);
mutex_unlock(&notify_mutex);
}
static int listen_notify_handler(struct notifier_block *this,
unsigned long event, void *data)
{
struct chtls_dev *cdev;
struct sock *sk;
int ret;
sk = data;
ret = NOTIFY_DONE;
switch (event) {
case CHTLS_LISTEN_START:
case CHTLS_LISTEN_STOP:
mutex_lock(&cdev_list_lock);
list_for_each_entry(cdev, &cdev_list, list) {
if (event == CHTLS_LISTEN_START)
ret = chtls_listen_start(cdev, sk);
else
chtls_listen_stop(cdev, sk);
}
mutex_unlock(&cdev_list_lock);
break;
}
return ret;
}
static struct notifier_block listen_notifier = {
.notifier_call = listen_notify_handler
};
static int listen_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
if (likely(skb_transport_header(skb) != skb_network_header(skb)))
return tcp_v4_do_rcv(sk, skb);
BLOG_SKB_CB(skb)->backlog_rcv(sk, skb);
return 0;
}
static int chtls_start_listen(struct sock *sk)
{
int err;
if (sk->sk_protocol != IPPROTO_TCP)
return -EPROTONOSUPPORT;
if (sk->sk_family == PF_INET &&
LOOPBACK(inet_sk(sk)->inet_rcv_saddr))
return -EADDRNOTAVAIL;
sk->sk_backlog_rcv = listen_backlog_rcv;
mutex_lock(&notify_mutex);
err = raw_notifier_call_chain(&listen_notify_list,
CHTLS_LISTEN_START, sk);
mutex_unlock(&notify_mutex);
return err;
}
static void chtls_stop_listen(struct sock *sk)
{
if (sk->sk_protocol != IPPROTO_TCP)
return;
mutex_lock(&notify_mutex);
raw_notifier_call_chain(&listen_notify_list,
CHTLS_LISTEN_STOP, sk);
mutex_unlock(&notify_mutex);
}
static int chtls_inline_feature(struct tls_device *dev)
{
struct net_device *netdev;
struct chtls_dev *cdev;
int i;
cdev = to_chtls_dev(dev);
for (i = 0; i < cdev->lldi->nports; i++) {
netdev = cdev->ports[i];
if (netdev->features & NETIF_F_HW_TLS_RECORD)
return 1;
}
return 0;
}
static int chtls_create_hash(struct tls_device *dev, struct sock *sk)
{
if (sk->sk_state == TCP_LISTEN)
return chtls_start_listen(sk);
return 0;
}
static void chtls_destroy_hash(struct tls_device *dev, struct sock *sk)
{
if (sk->sk_state == TCP_LISTEN)
chtls_stop_listen(sk);
}
static void chtls_register_dev(struct chtls_dev *cdev)
{
struct tls_device *tlsdev = &cdev->tlsdev;
strlcpy(tlsdev->name, "chtls", TLS_DEVICE_NAME_MAX);
strlcat(tlsdev->name, cdev->lldi->ports[0]->name,
TLS_DEVICE_NAME_MAX);
tlsdev->feature = chtls_inline_feature;
tlsdev->hash = chtls_create_hash;
tlsdev->unhash = chtls_destroy_hash;
tls_register_device(&cdev->tlsdev);
}
static void chtls_unregister_dev(struct chtls_dev *cdev)
{
tls_unregister_device(&cdev->tlsdev);
}
static void process_deferq(struct work_struct *task_param)
{
struct chtls_dev *cdev = container_of(task_param,
struct chtls_dev, deferq_task);
struct sk_buff *skb;
spin_lock_bh(&cdev->deferq.lock);
while ((skb = __skb_dequeue(&cdev->deferq)) != NULL) {
spin_unlock_bh(&cdev->deferq.lock);
DEFERRED_SKB_CB(skb)->handler(cdev, skb);
spin_lock_bh(&cdev->deferq.lock);
}
spin_unlock_bh(&cdev->deferq.lock);
}
static int chtls_get_skb(struct chtls_dev *cdev)
{
cdev->askb = alloc_skb(sizeof(struct tcphdr), GFP_KERNEL);
if (!cdev->askb)
return -ENOMEM;
skb_put(cdev->askb, sizeof(struct tcphdr));
skb_reset_transport_header(cdev->askb);
memset(cdev->askb->data, 0, cdev->askb->len);
return 0;
}
static void *chtls_uld_add(const struct cxgb4_lld_info *info)
{
struct cxgb4_lld_info *lldi;
struct chtls_dev *cdev;
int i, j;
cdev = kzalloc(sizeof(*cdev) + info->nports *
(sizeof(struct net_device *)), GFP_KERNEL);
if (!cdev)
goto out;
lldi = kzalloc(sizeof(*lldi), GFP_KERNEL);
if (!lldi)
goto out_lldi;
if (chtls_get_skb(cdev))
goto out_skb;
*lldi = *info;
cdev->lldi = lldi;
cdev->pdev = lldi->pdev;
cdev->tids = lldi->tids;
cdev->ports = (struct net_device **)(cdev + 1);
cdev->ports = lldi->ports;
cdev->mtus = lldi->mtus;
cdev->tids = lldi->tids;
cdev->pfvf = FW_VIID_PFN_G(cxgb4_port_viid(lldi->ports[0]))
<< FW_VIID_PFN_S;
for (i = 0; i < (1 << RSPQ_HASH_BITS); i++) {
unsigned int size = 64 - sizeof(struct rsp_ctrl) - 8;
cdev->rspq_skb_cache[i] = __alloc_skb(size,
gfp_any(), 0,
lldi->nodeid);
if (unlikely(!cdev->rspq_skb_cache[i]))
goto out_rspq_skb;
}
idr_init(&cdev->hwtid_idr);
INIT_WORK(&cdev->deferq_task, process_deferq);
spin_lock_init(&cdev->listen_lock);
spin_lock_init(&cdev->idr_lock);
cdev->send_page_order = min_t(uint, get_order(32768),
send_page_order);
if (lldi->vr->key.size)
if (chtls_init_kmap(cdev, lldi))
goto out_rspq_skb;
mutex_lock(&cdev_mutex);
list_add_tail(&cdev->list, &cdev_list);
mutex_unlock(&cdev_mutex);
return cdev;
out_rspq_skb:
for (j = 0; j <= i; j++)
kfree_skb(cdev->rspq_skb_cache[j]);
kfree_skb(cdev->askb);
out_skb:
kfree(lldi);
out_lldi:
kfree(cdev);
out:
return NULL;
}
static void chtls_free_uld(struct chtls_dev *cdev)
{
int i;
chtls_unregister_dev(cdev);
kvfree(cdev->kmap.addr);
idr_destroy(&cdev->hwtid_idr);
for (i = 0; i < (1 << RSPQ_HASH_BITS); i++)
kfree_skb(cdev->rspq_skb_cache[i]);
kfree(cdev->lldi);
if (cdev->askb)
kfree_skb(cdev->askb);
kfree(cdev);
}
static void chtls_free_all_uld(void)
{
struct chtls_dev *cdev, *tmp;
mutex_lock(&cdev_mutex);
list_for_each_entry_safe(cdev, tmp, &cdev_list, list)
chtls_free_uld(cdev);
mutex_unlock(&cdev_mutex);
}
static int chtls_uld_state_change(void *handle, enum cxgb4_state new_state)
{
struct chtls_dev *cdev = handle;
switch (new_state) {
case CXGB4_STATE_UP:
chtls_register_dev(cdev);
break;
case CXGB4_STATE_DOWN:
break;
case CXGB4_STATE_START_RECOVERY:
break;
case CXGB4_STATE_DETACH:
mutex_lock(&cdev_mutex);
list_del(&cdev->list);
mutex_unlock(&cdev_mutex);
chtls_free_uld(cdev);
break;
default:
break;
}
return 0;
}
static struct sk_buff *copy_gl_to_skb_pkt(const struct pkt_gl *gl,
const __be64 *rsp,
u32 pktshift)
{
struct sk_buff *skb;
/* Allocate space for cpl_pass_accpet_req which will be synthesized by
* driver. Once driver synthesizes cpl_pass_accpet_req the skb will go
* through the regular cpl_pass_accept_req processing in TOM.
*/
skb = alloc_skb(gl->tot_len + sizeof(struct cpl_pass_accept_req)
- pktshift, GFP_ATOMIC);
if (unlikely(!skb))
return NULL;
__skb_put(skb, gl->tot_len + sizeof(struct cpl_pass_accept_req)
- pktshift);
/* For now we will copy cpl_rx_pkt in the skb */
skb_copy_to_linear_data(skb, rsp, sizeof(struct cpl_rx_pkt));
skb_copy_to_linear_data_offset(skb, sizeof(struct cpl_pass_accept_req)
, gl->va + pktshift,
gl->tot_len - pktshift);
return skb;
}
static int chtls_recv_packet(struct chtls_dev *cdev,
const struct pkt_gl *gl, const __be64 *rsp)
{
unsigned int opcode = *(u8 *)rsp;
struct sk_buff *skb;
int ret;
skb = copy_gl_to_skb_pkt(gl, rsp, cdev->lldi->sge_pktshift);
if (!skb)
return -ENOMEM;
ret = chtls_handlers[opcode](cdev, skb);
if (ret & CPL_RET_BUF_DONE)
kfree_skb(skb);
return 0;
}
static int chtls_recv_rsp(struct chtls_dev *cdev, const __be64 *rsp)
{
unsigned long rspq_bin;
unsigned int opcode;
struct sk_buff *skb;
unsigned int len;
int ret;
len = 64 - sizeof(struct rsp_ctrl) - 8;
opcode = *(u8 *)rsp;
rspq_bin = hash_ptr((void *)rsp, RSPQ_HASH_BITS);
skb = cdev->rspq_skb_cache[rspq_bin];
if (skb && !skb_is_nonlinear(skb) &&
!skb_shared(skb) && !skb_cloned(skb)) {
refcount_inc(&skb->users);
if (refcount_read(&skb->users) == 2) {
__skb_trim(skb, 0);
if (skb_tailroom(skb) >= len)
goto copy_out;
}
refcount_dec(&skb->users);
}
skb = alloc_skb(len, GFP_ATOMIC);
if (unlikely(!skb))
return -ENOMEM;
copy_out:
__skb_put(skb, len);
skb_copy_to_linear_data(skb, rsp, len);
skb_reset_network_header(skb);
skb_reset_transport_header(skb);
ret = chtls_handlers[opcode](cdev, skb);
if (ret & CPL_RET_BUF_DONE)
kfree_skb(skb);
return 0;
}
static void chtls_recv(struct chtls_dev *cdev,
struct sk_buff **skbs, const __be64 *rsp)
{
struct sk_buff *skb = *skbs;
unsigned int opcode;
int ret;
opcode = *(u8 *)rsp;
__skb_push(skb, sizeof(struct rss_header));
skb_copy_to_linear_data(skb, rsp, sizeof(struct rss_header));
ret = chtls_handlers[opcode](cdev, skb);
if (ret & CPL_RET_BUF_DONE)
kfree_skb(skb);
}
static int chtls_uld_rx_handler(void *handle, const __be64 *rsp,
const struct pkt_gl *gl)
{
struct chtls_dev *cdev = handle;
unsigned int opcode;
struct sk_buff *skb;
opcode = *(u8 *)rsp;
if (unlikely(opcode == CPL_RX_PKT)) {
if (chtls_recv_packet(cdev, gl, rsp) < 0)
goto nomem;
return 0;
}
if (!gl)
return chtls_recv_rsp(cdev, rsp);
#define RX_PULL_LEN 128
skb = cxgb4_pktgl_to_skb(gl, RX_PULL_LEN, RX_PULL_LEN);
if (unlikely(!skb))
goto nomem;
chtls_recv(cdev, &skb, rsp);
return 0;
nomem:
return -ENOMEM;
}
static int do_chtls_getsockopt(struct sock *sk, char __user *optval,
int __user *optlen)
{
struct tls_crypto_info crypto_info;
crypto_info.version = TLS_1_2_VERSION;
if (copy_to_user(optval, &crypto_info, sizeof(struct tls_crypto_info)))
return -EFAULT;
return 0;
}
static int chtls_getsockopt(struct sock *sk, int level, int optname,
char __user *optval, int __user *optlen)
{
struct tls_context *ctx = tls_get_ctx(sk);
if (level != SOL_TLS)
return ctx->getsockopt(sk, level, optname, optval, optlen);
return do_chtls_getsockopt(sk, optval, optlen);
}
static int do_chtls_setsockopt(struct sock *sk, int optname,
char __user *optval, unsigned int optlen)
{
struct tls_crypto_info *crypto_info, tmp_crypto_info;
struct chtls_sock *csk;
int keylen;
int rc = 0;
csk = rcu_dereference_sk_user_data(sk);
if (!optval || optlen < sizeof(*crypto_info)) {
rc = -EINVAL;
goto out;
}
rc = copy_from_user(&tmp_crypto_info, optval, sizeof(*crypto_info));
if (rc) {
rc = -EFAULT;
goto out;
}
/* check version */
if (tmp_crypto_info.version != TLS_1_2_VERSION) {
rc = -ENOTSUPP;
goto out;
}
crypto_info = (struct tls_crypto_info *)&csk->tlshws.crypto_info;
switch (tmp_crypto_info.cipher_type) {
case TLS_CIPHER_AES_GCM_128: {
rc = copy_from_user(crypto_info, optval,
sizeof(struct
tls12_crypto_info_aes_gcm_128));
if (rc) {
rc = -EFAULT;
goto out;
}
keylen = TLS_CIPHER_AES_GCM_128_KEY_SIZE;
rc = chtls_setkey(csk, keylen, optname);
break;
}
default:
rc = -EINVAL;
goto out;
}
out:
return rc;
}
static int chtls_setsockopt(struct sock *sk, int level, int optname,
char __user *optval, unsigned int optlen)
{
struct tls_context *ctx = tls_get_ctx(sk);
if (level != SOL_TLS)
return ctx->setsockopt(sk, level, optname, optval, optlen);
return do_chtls_setsockopt(sk, optname, optval, optlen);
}
static struct cxgb4_uld_info chtls_uld_info = {
.name = DRV_NAME,
.nrxq = MAX_ULD_QSETS,
.ntxq = MAX_ULD_QSETS,
.rxq_size = 1024,
.add = chtls_uld_add,
.state_change = chtls_uld_state_change,
.rx_handler = chtls_uld_rx_handler,
};
void chtls_install_cpl_ops(struct sock *sk)
{
sk->sk_prot = &chtls_cpl_prot;
}
static void __init chtls_init_ulp_ops(void)
{
chtls_cpl_prot = tcp_prot;
chtls_init_rsk_ops(&chtls_cpl_prot, &chtls_rsk_ops,
&tcp_prot, PF_INET);
chtls_cpl_prot.close = chtls_close;
chtls_cpl_prot.disconnect = chtls_disconnect;
chtls_cpl_prot.destroy = chtls_destroy_sock;
chtls_cpl_prot.shutdown = chtls_shutdown;
chtls_cpl_prot.sendmsg = chtls_sendmsg;
chtls_cpl_prot.sendpage = chtls_sendpage;
chtls_cpl_prot.recvmsg = chtls_recvmsg;
chtls_cpl_prot.setsockopt = chtls_setsockopt;
chtls_cpl_prot.getsockopt = chtls_getsockopt;
}
static int __init chtls_register(void)
{
chtls_init_ulp_ops();
register_listen_notifier(&listen_notifier);
cxgb4_register_uld(CXGB4_ULD_TLS, &chtls_uld_info);
return 0;
}
static void __exit chtls_unregister(void)
{
unregister_listen_notifier(&listen_notifier);
chtls_free_all_uld();
cxgb4_unregister_uld(CXGB4_ULD_TLS);
}
module_init(chtls_register);
module_exit(chtls_unregister);
MODULE_DESCRIPTION("Chelsio TLS Inline driver");
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Chelsio Communications");
MODULE_VERSION(DRV_VERSION);

32
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
View File

@ -4549,18 +4549,32 @@ static int adap_init0(struct adapter *adap)
adap->num_ofld_uld += 2;
}
if (caps_cmd.cryptocaps) {
/* Should query params here...TODO */
params[0] = FW_PARAM_PFVF(NCRYPTO_LOOKASIDE);
ret = t4_query_params(adap, adap->mbox, adap->pf, 0, 2,
params, val);
if (ret < 0) {
if (ret != -EINVAL)
if (ntohs(caps_cmd.cryptocaps) &
FW_CAPS_CONFIG_CRYPTO_LOOKASIDE) {
params[0] = FW_PARAM_PFVF(NCRYPTO_LOOKASIDE);
ret = t4_query_params(adap, adap->mbox, adap->pf, 0,
2, params, val);
if (ret < 0) {
if (ret != -EINVAL)
goto bye;
} else {
adap->vres.ncrypto_fc = val[0];
}
adap->num_ofld_uld += 1;
}
if (ntohs(caps_cmd.cryptocaps) &
FW_CAPS_CONFIG_TLS_INLINE) {
params[0] = FW_PARAM_PFVF(TLS_START);
params[1] = FW_PARAM_PFVF(TLS_END);
ret = t4_query_params(adap, adap->mbox, adap->pf, 0,
2, params, val);
if (ret < 0)
goto bye;
} else {
adap->vres.ncrypto_fc = val[0];
adap->vres.key.start = val[0];
adap->vres.key.size = val[1] - val[0] + 1;
adap->num_uld += 1;
}
adap->params.crypto = ntohs(caps_cmd.cryptocaps);
adap->num_uld += 1;
}
#undef FW_PARAM_PFVF
#undef FW_PARAM_DEV

7
drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
View File

@ -237,6 +237,7 @@ enum cxgb4_uld {
CXGB4_ULD_ISCSI,
CXGB4_ULD_ISCSIT,
CXGB4_ULD_CRYPTO,
CXGB4_ULD_TLS,
CXGB4_ULD_MAX
};
@ -289,6 +290,7 @@ struct cxgb4_virt_res { /* virtualized HW resources */
struct cxgb4_range qp;
struct cxgb4_range cq;
struct cxgb4_range ocq;
struct cxgb4_range key;
unsigned int ncrypto_fc;
};
@ -300,6 +302,9 @@ struct chcr_stats_debug {
atomic_t error;
atomic_t fallback;
atomic_t ipsec_cnt;
atomic_t tls_pdu_tx;
atomic_t tls_pdu_rx;
atomic_t tls_key;
};
#define OCQ_WIN_OFFSET(pdev, vres) \
@ -382,6 +387,8 @@ struct cxgb4_uld_info {
int cxgb4_register_uld(enum cxgb4_uld type, const struct cxgb4_uld_info *p);
int cxgb4_unregister_uld(enum cxgb4_uld type);
int cxgb4_ofld_send(struct net_device *dev, struct sk_buff *skb);
int cxgb4_immdata_send(struct net_device *dev, unsigned int idx,
const void *src, unsigned int len);
int cxgb4_crypto_send(struct net_device *dev, struct sk_buff *skb);
unsigned int cxgb4_dbfifo_count(const struct net_device *dev, int lpfifo);
unsigned int cxgb4_port_chan(const struct net_device *dev);

107
drivers/net/ethernet/chelsio/cxgb4/sge.c
View File

@ -1019,8 +1019,8 @@ EXPORT_SYMBOL(cxgb4_ring_tx_db);
void cxgb4_inline_tx_skb(const struct sk_buff *skb,
const struct sge_txq *q, void *pos)
{
u64 *p;
int left = (void *)q->stat - pos;
u64 *p;
if (likely(skb->len <= left)) {
if (likely(!skb->data_len))
@ -1735,15 +1735,13 @@ static void txq_stop_maperr(struct sge_uld_txq *q)
/**
* ofldtxq_stop - stop an offload Tx queue that has become full
* @q: the queue to stop
* @skb: the packet causing the queue to become full
* @wr: the Work Request causing the queue to become full
*
* Stops an offload Tx queue that has become full and modifies the packet
* being written to request a wakeup.
*/
static void ofldtxq_stop(struct sge_uld_txq *q, struct sk_buff *skb)
static void ofldtxq_stop(struct sge_uld_txq *q, struct fw_wr_hdr *wr)
{
struct fw_wr_hdr *wr = (struct fw_wr_hdr *)skb->data;
wr->lo |= htonl(FW_WR_EQUEQ_F | FW_WR_EQUIQ_F);
q->q.stops++;
q->full = 1;
@ -1804,7 +1802,7 @@ static void service_ofldq(struct sge_uld_txq *q)
credits = txq_avail(&q->q) - ndesc;
BUG_ON(credits < 0);
if (unlikely(credits < TXQ_STOP_THRES))
ofldtxq_stop(q, skb);
ofldtxq_stop(q, (struct fw_wr_hdr *)skb->data);
pos = (u64 *)&q->q.desc[q->q.pidx];
if (is_ofld_imm(skb))
@ -2005,6 +2003,103 @@ int cxgb4_ofld_send(struct net_device *dev, struct sk_buff *skb)
}
EXPORT_SYMBOL(cxgb4_ofld_send);
static void *inline_tx_header(const void *src,
const struct sge_txq *q,
void *pos, int length)
{
int left = (void *)q->stat - pos;
u64 *p;
if (likely(length <= left)) {
memcpy(pos, src, length);
pos += length;
} else {
memcpy(pos, src, left);
memcpy(q->desc, src + left, length - left);
pos = (void *)q->desc + (length - left);
}
/* 0-pad to multiple of 16 */
p = PTR_ALIGN(pos, 8);
if ((uintptr_t)p & 8) {
*p = 0;
return p + 1;
}
return p;
}
/**
* ofld_xmit_direct - copy a WR into offload queue
* @q: the Tx offload queue
* @src: location of WR
* @len: WR length
*
* Copy an immediate WR into an uncontended SGE offload queue.
*/
static int ofld_xmit_direct(struct sge_uld_txq *q, const void *src,
unsigned int len)
{
unsigned int ndesc;
int credits;
u64 *pos;
/* Use the lower limit as the cut-off */
if (len > MAX_IMM_OFLD_TX_DATA_WR_LEN) {
WARN_ON(1);
return NET_XMIT_DROP;
}
/* Don't return NET_XMIT_CN here as the current
* implementation doesn't queue the request
* using an skb when the following conditions not met
*/
if (!spin_trylock(&q->sendq.lock))
return NET_XMIT_DROP;
if (q->full || !skb_queue_empty(&q->sendq) ||
q->service_ofldq_running) {
spin_unlock(&q->sendq.lock);
return NET_XMIT_DROP;
}
ndesc = flits_to_desc(DIV_ROUND_UP(len, 8));
credits = txq_avail(&q->q) - ndesc;
pos = (u64 *)&q->q.desc[q->q.pidx];
/* ofldtxq_stop modifies WR header in-situ */
inline_tx_header(src, &q->q, pos, len);
if (unlikely(credits < TXQ_STOP_THRES))
ofldtxq_stop(q, (struct fw_wr_hdr *)pos);
txq_advance(&q->q, ndesc);
cxgb4_ring_tx_db(q->adap, &q->q, ndesc);
spin_unlock(&q->sendq.lock);
return NET_XMIT_SUCCESS;
}
int cxgb4_immdata_send(struct net_device *dev, unsigned int idx,
const void *src, unsigned int len)
{
struct sge_uld_txq_info *txq_info;
struct sge_uld_txq *txq;
struct adapter *adap;
int ret;
adap = netdev2adap(dev);
local_bh_disable();
txq_info = adap->sge.uld_txq_info[CXGB4_TX_OFLD];
if (unlikely(!txq_info)) {
WARN_ON(true);
local_bh_enable();
return NET_XMIT_DROP;
}
txq = &txq_info->uldtxq[idx];
ret = ofld_xmit_direct(txq, src, len);
local_bh_enable();
return net_xmit_eval(ret);
}
EXPORT_SYMBOL(cxgb4_immdata_send);
/**
* t4_crypto_send - send crypto packet
* @adap: the adapter

122
drivers/net/ethernet/chelsio/cxgb4/t4_msg.h
View File

@ -82,13 +82,15 @@ enum {
CPL_RX_ISCSI_CMP = 0x45,
CPL_TRACE_PKT_T5 = 0x48,
CPL_RX_ISCSI_DDP = 0x49,
CPL_RX_TLS_CMP = 0x4E,
CPL_RDMA_READ_REQ = 0x60,
CPL_PASS_OPEN_REQ6 = 0x81,
CPL_ACT_OPEN_REQ6 = 0x83,
CPL_TX_TLS_PDU = 0x88,
CPL_TX_TLS_PDU = 0x88,
CPL_TX_TLS_SFO = 0x89,
CPL_TX_SEC_PDU = 0x8A,
CPL_TX_TLS_ACK = 0x8B,
@ -98,6 +100,7 @@ enum {
CPL_RX_MPS_PKT = 0xAF,
CPL_TRACE_PKT = 0xB0,
CPL_TLS_DATA = 0xB1,
CPL_ISCSI_DATA = 0xB2,
CPL_FW4_MSG = 0xC0,
@ -155,6 +158,7 @@ enum {
ULP_MODE_RDMA = 4,
ULP_MODE_TCPDDP = 5,
ULP_MODE_FCOE = 6,
ULP_MODE_TLS = 8,
};
enum {
@ -1445,6 +1449,14 @@ struct cpl_tx_data {
#define T6_TX_FORCE_V(x) ((x) << T6_TX_FORCE_S)
#define T6_TX_FORCE_F T6_TX_FORCE_V(1U)
#define TX_SHOVE_S 14
#define TX_SHOVE_V(x) ((x) << TX_SHOVE_S)
#define TX_ULP_MODE_S 10
#define TX_ULP_MODE_M 0x7
#define TX_ULP_MODE_V(x) ((x) << TX_ULP_MODE_S)
#define TX_ULP_MODE_G(x) (((x) >> TX_ULP_MODE_S) & TX_ULP_MODE_M)
enum {
ULP_TX_MEM_READ = 2,
ULP_TX_MEM_WRITE = 3,
@ -1455,12 +1467,21 @@ enum {
ULP_TX_SC_NOOP = 0x80,
ULP_TX_SC_IMM = 0x81,
ULP_TX_SC_DSGL = 0x82,
ULP_TX_SC_ISGL = 0x83
ULP_TX_SC_ISGL = 0x83,
ULP_TX_SC_MEMRD = 0x86
};
#define ULPTX_CMD_S 24
#define ULPTX_CMD_V(x) ((x) << ULPTX_CMD_S)
#define ULPTX_LEN16_S 0
#define ULPTX_LEN16_M 0xFF
#define ULPTX_LEN16_V(x) ((x) << ULPTX_LEN16_S)
#define ULP_TX_SC_MORE_S 23
#define ULP_TX_SC_MORE_V(x) ((x) << ULP_TX_SC_MORE_S)
#define ULP_TX_SC_MORE_F ULP_TX_SC_MORE_V(1U)
struct ulptx_sge_pair {
__be32 len[2];
__be64 addr[2];
@ -2183,4 +2204,101 @@ struct cpl_srq_table_rpl {
#define SRQT_IDX_V(x) ((x) << SRQT_IDX_S)
#define SRQT_IDX_G(x) (((x) >> SRQT_IDX_S) & SRQT_IDX_M)
struct cpl_tx_tls_sfo {
__be32 op_to_seg_len;
__be32 pld_len;
__be32 type_protover;
__be32 r1_lo;
__be32 seqno_numivs;
__be32 ivgen_hdrlen;
__be64 scmd1;
};
/* cpl_tx_tls_sfo macros */
#define CPL_TX_TLS_SFO_OPCODE_S 24
#define CPL_TX_TLS_SFO_OPCODE_V(x) ((x) << CPL_TX_TLS_SFO_OPCODE_S)
#define CPL_TX_TLS_SFO_DATA_TYPE_S 20
#define CPL_TX_TLS_SFO_DATA_TYPE_V(x) ((x) << CPL_TX_TLS_SFO_DATA_TYPE_S)
#define CPL_TX_TLS_SFO_CPL_LEN_S 16
#define CPL_TX_TLS_SFO_CPL_LEN_V(x) ((x) << CPL_TX_TLS_SFO_CPL_LEN_S)
#define CPL_TX_TLS_SFO_SEG_LEN_S 0
#define CPL_TX_TLS_SFO_SEG_LEN_M 0xffff
#define CPL_TX_TLS_SFO_SEG_LEN_V(x) ((x) << CPL_TX_TLS_SFO_SEG_LEN_S)
#define CPL_TX_TLS_SFO_SEG_LEN_G(x) \
(((x) >> CPL_TX_TLS_SFO_SEG_LEN_S) & CPL_TX_TLS_SFO_SEG_LEN_M)
#define CPL_TX_TLS_SFO_TYPE_S 24
#define CPL_TX_TLS_SFO_TYPE_M 0xff
#define CPL_TX_TLS_SFO_TYPE_V(x) ((x) << CPL_TX_TLS_SFO_TYPE_S)
#define CPL_TX_TLS_SFO_TYPE_G(x) \
(((x) >> CPL_TX_TLS_SFO_TYPE_S) & CPL_TX_TLS_SFO_TYPE_M)
#define CPL_TX_TLS_SFO_PROTOVER_S 8
#define CPL_TX_TLS_SFO_PROTOVER_M 0xffff
#define CPL_TX_TLS_SFO_PROTOVER_V(x) ((x) << CPL_TX_TLS_SFO_PROTOVER_S)
#define CPL_TX_TLS_SFO_PROTOVER_G(x) \
(((x) >> CPL_TX_TLS_SFO_PROTOVER_S) & CPL_TX_TLS_SFO_PROTOVER_M)
struct cpl_tls_data {
struct rss_header rsshdr;
union opcode_tid ot;
__be32 length_pkd;
__be32 seq;
__be32 r1;
};
#define CPL_TLS_DATA_OPCODE_S 24
#define CPL_TLS_DATA_OPCODE_M 0xff
#define CPL_TLS_DATA_OPCODE_V(x) ((x) << CPL_TLS_DATA_OPCODE_S)
#define CPL_TLS_DATA_OPCODE_G(x) \
(((x) >> CPL_TLS_DATA_OPCODE_S) & CPL_TLS_DATA_OPCODE_M)
#define CPL_TLS_DATA_TID_S 0
#define CPL_TLS_DATA_TID_M 0xffffff
#define CPL_TLS_DATA_TID_V(x) ((x) << CPL_TLS_DATA_TID_S)
#define CPL_TLS_DATA_TID_G(x) \
(((x) >> CPL_TLS_DATA_TID_S) & CPL_TLS_DATA_TID_M)
#define CPL_TLS_DATA_LENGTH_S 0
#define CPL_TLS_DATA_LENGTH_M 0xffff
#define CPL_TLS_DATA_LENGTH_V(x) ((x) << CPL_TLS_DATA_LENGTH_S)
#define CPL_TLS_DATA_LENGTH_G(x) \
(((x) >> CPL_TLS_DATA_LENGTH_S) & CPL_TLS_DATA_LENGTH_M)
struct cpl_rx_tls_cmp {
struct rss_header rsshdr;
union opcode_tid ot;
__be32 pdulength_length;
__be32 seq;
__be32 ddp_report;
__be32 r;
__be32 ddp_valid;
};
#define CPL_RX_TLS_CMP_OPCODE_S 24
#define CPL_RX_TLS_CMP_OPCODE_M 0xff
#define CPL_RX_TLS_CMP_OPCODE_V(x) ((x) << CPL_RX_TLS_CMP_OPCODE_S)
#define CPL_RX_TLS_CMP_OPCODE_G(x) \
(((x) >> CPL_RX_TLS_CMP_OPCODE_S) & CPL_RX_TLS_CMP_OPCODE_M)
#define CPL_RX_TLS_CMP_TID_S 0
#define CPL_RX_TLS_CMP_TID_M 0xffffff
#define CPL_RX_TLS_CMP_TID_V(x) ((x) << CPL_RX_TLS_CMP_TID_S)
#define CPL_RX_TLS_CMP_TID_G(x) \
(((x) >> CPL_RX_TLS_CMP_TID_S) & CPL_RX_TLS_CMP_TID_M)
#define CPL_RX_TLS_CMP_PDULENGTH_S 16
#define CPL_RX_TLS_CMP_PDULENGTH_M 0xffff
#define CPL_RX_TLS_CMP_PDULENGTH_V(x) ((x) << CPL_RX_TLS_CMP_PDULENGTH_S)
#define CPL_RX_TLS_CMP_PDULENGTH_G(x) \
(((x) >> CPL_RX_TLS_CMP_PDULENGTH_S) & CPL_RX_TLS_CMP_PDULENGTH_M)
#define CPL_RX_TLS_CMP_LENGTH_S 0
#define CPL_RX_TLS_CMP_LENGTH_M 0xffff
#define CPL_RX_TLS_CMP_LENGTH_V(x) ((x) << CPL_RX_TLS_CMP_LENGTH_S)
#define CPL_RX_TLS_CMP_LENGTH_G(x) \
(((x) >> CPL_RX_TLS_CMP_LENGTH_S) & CPL_RX_TLS_CMP_LENGTH_M)
#endif /* __T4_MSG_H */

2
drivers/net/ethernet/chelsio/cxgb4/t4_regs.h
View File

@ -2775,6 +2775,8 @@
#define ULP_RX_LA_RDPTR_A 0x19240
#define ULP_RX_LA_RDDATA_A 0x19244
#define ULP_RX_LA_WRPTR_A 0x19248
#define ULP_RX_TLS_KEY_LLIMIT_A 0x192ac
#define ULP_RX_TLS_KEY_ULIMIT_A 0x192b0
#define HPZ3_S 24
#define HPZ3_V(x) ((x) << HPZ3_S)

165
drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
View File

@ -105,6 +105,7 @@ enum fw_wr_opcodes {
FW_RI_INV_LSTAG_WR = 0x1a,
FW_ISCSI_TX_DATA_WR = 0x45,
FW_PTP_TX_PKT_WR = 0x46,
FW_TLSTX_DATA_WR = 0x68,
FW_CRYPTO_LOOKASIDE_WR = 0X6d,
FW_LASTC2E_WR = 0x70,
FW_FILTER2_WR = 0x77
@ -635,6 +636,30 @@ struct fw_ofld_connection_wr {
#define FW_OFLD_CONNECTION_WR_CPLPASSACCEPTRPL_F \
FW_OFLD_CONNECTION_WR_CPLPASSACCEPTRPL_V(1U)
enum fw_flowc_mnem_tcpstate {
FW_FLOWC_MNEM_TCPSTATE_CLOSED = 0, /* illegal */
FW_FLOWC_MNEM_TCPSTATE_LISTEN = 1, /* illegal */
FW_FLOWC_MNEM_TCPSTATE_SYNSENT = 2, /* illegal */
FW_FLOWC_MNEM_TCPSTATE_SYNRECEIVED = 3, /* illegal */
FW_FLOWC_MNEM_TCPSTATE_ESTABLISHED = 4, /* default */
FW_FLOWC_MNEM_TCPSTATE_CLOSEWAIT = 5, /* got peer close already */
FW_FLOWC_MNEM_TCPSTATE_FINWAIT1 = 6, /* haven't gotten ACK for FIN and
* will resend FIN - equiv ESTAB
*/
FW_FLOWC_MNEM_TCPSTATE_CLOSING = 7, /* haven't gotten ACK for FIN and
* will resend FIN but have
* received FIN
*/
FW_FLOWC_MNEM_TCPSTATE_LASTACK = 8, /* haven't gotten ACK for FIN and
* will resend FIN but have
* received FIN
*/
FW_FLOWC_MNEM_TCPSTATE_FINWAIT2 = 9, /* sent FIN and got FIN + ACK,
* waiting for FIN
*/
FW_FLOWC_MNEM_TCPSTATE_TIMEWAIT = 10, /* not expected */
};
enum fw_flowc_mnem {
FW_FLOWC_MNEM_PFNVFN, /* PFN [15:8] VFN [7:0] */
FW_FLOWC_MNEM_CH,
@ -651,6 +676,8 @@ enum fw_flowc_mnem {
FW_FLOWC_MNEM_DCBPRIO,
FW_FLOWC_MNEM_SND_SCALE,
FW_FLOWC_MNEM_RCV_SCALE,
FW_FLOWC_MNEM_ULD_MODE,
FW_FLOWC_MNEM_MAX,
};
struct fw_flowc_mnemval {
@ -675,6 +702,14 @@ struct fw_ofld_tx_data_wr {
__be32 tunnel_to_proxy;
};
#define FW_OFLD_TX_DATA_WR_ALIGNPLD_S 30
#define FW_OFLD_TX_DATA_WR_ALIGNPLD_V(x) ((x) << FW_OFLD_TX_DATA_WR_ALIGNPLD_S)
#define FW_OFLD_TX_DATA_WR_ALIGNPLD_F FW_OFLD_TX_DATA_WR_ALIGNPLD_V(1U)
#define FW_OFLD_TX_DATA_WR_SHOVE_S 29
#define FW_OFLD_TX_DATA_WR_SHOVE_V(x) ((x) << FW_OFLD_TX_DATA_WR_SHOVE_S)
#define FW_OFLD_TX_DATA_WR_SHOVE_F FW_OFLD_TX_DATA_WR_SHOVE_V(1U)
#define FW_OFLD_TX_DATA_WR_TUNNEL_S 19
#define FW_OFLD_TX_DATA_WR_TUNNEL_V(x) ((x) << FW_OFLD_TX_DATA_WR_TUNNEL_S)
@ -691,10 +726,6 @@ struct fw_ofld_tx_data_wr {
#define FW_OFLD_TX_DATA_WR_MORE_S 15
#define FW_OFLD_TX_DATA_WR_MORE_V(x) ((x) << FW_OFLD_TX_DATA_WR_MORE_S)
#define FW_OFLD_TX_DATA_WR_SHOVE_S 14
#define FW_OFLD_TX_DATA_WR_SHOVE_V(x) ((x) << FW_OFLD_TX_DATA_WR_SHOVE_S)
#define FW_OFLD_TX_DATA_WR_SHOVE_F FW_OFLD_TX_DATA_WR_SHOVE_V(1U)
#define FW_OFLD_TX_DATA_WR_ULPMODE_S 10
#define FW_OFLD_TX_DATA_WR_ULPMODE_V(x) ((x) << FW_OFLD_TX_DATA_WR_ULPMODE_S)
@ -1121,6 +1152,12 @@ enum fw_caps_config_iscsi {
FW_CAPS_CONFIG_ISCSI_TARGET_CNXOFLD = 0x00000008,
};
enum fw_caps_config_crypto {
FW_CAPS_CONFIG_CRYPTO_LOOKASIDE = 0x00000001,
FW_CAPS_CONFIG_TLS_INLINE = 0x00000002,
FW_CAPS_CONFIG_IPSEC_INLINE = 0x00000004,
};
enum fw_caps_config_fcoe {
FW_CAPS_CONFIG_FCOE_INITIATOR = 0x00000001,
FW_CAPS_CONFIG_FCOE_TARGET = 0x00000002,
@ -1266,6 +1303,8 @@ enum fw_params_param_pfvf {
FW_PARAMS_PARAM_PFVF_CPLFW4MSG_ENCAP = 0x31,
FW_PARAMS_PARAM_PFVF_HPFILTER_START = 0x32,
FW_PARAMS_PARAM_PFVF_HPFILTER_END = 0x33,
FW_PARAMS_PARAM_PFVF_TLS_START = 0x34,
FW_PARAMS_PARAM_PFVF_TLS_END = 0x35,
FW_PARAMS_PARAM_PFVF_NCRYPTO_LOOKASIDE = 0x39,
FW_PARAMS_PARAM_PFVF_PORT_CAPS32 = 0x3A,
};
@ -3839,4 +3878,122 @@ struct fw_crypto_lookaside_wr {
(((x) >> FW_CRYPTO_LOOKASIDE_WR_HASH_SIZE_S) & \
FW_CRYPTO_LOOKASIDE_WR_HASH_SIZE_M)
struct fw_tlstx_data_wr {
__be32 op_to_immdlen;
__be32 flowid_len16;
__be32 plen;
__be32 lsodisable_to_flags;
__be32 r5;
__be32 ctxloc_to_exp;
__be16 mfs;
__be16 adjustedplen_pkd;
__be16 expinplenmax_pkd;
u8 pdusinplenmax_pkd;
u8 r10;
};
#define FW_TLSTX_DATA_WR_OPCODE_S 24
#define FW_TLSTX_DATA_WR_OPCODE_M 0xff
#define FW_TLSTX_DATA_WR_OPCODE_V(x) ((x) << FW_TLSTX_DATA_WR_OPCODE_S)
#define FW_TLSTX_DATA_WR_OPCODE_G(x) \
(((x) >> FW_TLSTX_DATA_WR_OPCODE_S) & FW_TLSTX_DATA_WR_OPCODE_M)
#define FW_TLSTX_DATA_WR_COMPL_S 21
#define FW_TLSTX_DATA_WR_COMPL_M 0x1
#define FW_TLSTX_DATA_WR_COMPL_V(x) ((x) << FW_TLSTX_DATA_WR_COMPL_S)
#define FW_TLSTX_DATA_WR_COMPL_G(x) \
(((x) >> FW_TLSTX_DATA_WR_COMPL_S) & FW_TLSTX_DATA_WR_COMPL_M)
#define FW_TLSTX_DATA_WR_COMPL_F FW_TLSTX_DATA_WR_COMPL_V(1U)
#define FW_TLSTX_DATA_WR_IMMDLEN_S 0
#define FW_TLSTX_DATA_WR_IMMDLEN_M 0xff
#define FW_TLSTX_DATA_WR_IMMDLEN_V(x) ((x) << FW_TLSTX_DATA_WR_IMMDLEN_S)
#define FW_TLSTX_DATA_WR_IMMDLEN_G(x) \
(((x) >> FW_TLSTX_DATA_WR_IMMDLEN_S) & FW_TLSTX_DATA_WR_IMMDLEN_M)
#define FW_TLSTX_DATA_WR_FLOWID_S 8
#define FW_TLSTX_DATA_WR_FLOWID_M 0xfffff
#define FW_TLSTX_DATA_WR_FLOWID_V(x) ((x) << FW_TLSTX_DATA_WR_FLOWID_S)
#define FW_TLSTX_DATA_WR_FLOWID_G(x) \
(((x) >> FW_TLSTX_DATA_WR_FLOWID_S) & FW_TLSTX_DATA_WR_FLOWID_M)
#define FW_TLSTX_DATA_WR_LEN16_S 0
#define FW_TLSTX_DATA_WR_LEN16_M 0xff
#define FW_TLSTX_DATA_WR_LEN16_V(x) ((x) << FW_TLSTX_DATA_WR_LEN16_S)
#define FW_TLSTX_DATA_WR_LEN16_G(x) \
(((x) >> FW_TLSTX_DATA_WR_LEN16_S) & FW_TLSTX_DATA_WR_LEN16_M)
#define FW_TLSTX_DATA_WR_LSODISABLE_S 31
#define FW_TLSTX_DATA_WR_LSODISABLE_M 0x1
#define FW_TLSTX_DATA_WR_LSODISABLE_V(x) \
((x) << FW_TLSTX_DATA_WR_LSODISABLE_S)
#define FW_TLSTX_DATA_WR_LSODISABLE_G(x) \
(((x) >> FW_TLSTX_DATA_WR_LSODISABLE_S) & FW_TLSTX_DATA_WR_LSODISABLE_M)
#define FW_TLSTX_DATA_WR_LSODISABLE_F FW_TLSTX_DATA_WR_LSODISABLE_V(1U)
#define FW_TLSTX_DATA_WR_ALIGNPLD_S 30
#define FW_TLSTX_DATA_WR_ALIGNPLD_M 0x1
#define FW_TLSTX_DATA_WR_ALIGNPLD_V(x) ((x) << FW_TLSTX_DATA_WR_ALIGNPLD_S)
#define FW_TLSTX_DATA_WR_ALIGNPLD_G(x) \
(((x) >> FW_TLSTX_DATA_WR_ALIGNPLD_S) & FW_TLSTX_DATA_WR_ALIGNPLD_M)
#define FW_TLSTX_DATA_WR_ALIGNPLD_F FW_TLSTX_DATA_WR_ALIGNPLD_V(1U)
#define FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_S 29
#define FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_M 0x1
#define FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_V(x) \
((x) << FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_S)
#define FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_G(x) \
(((x) >> FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_S) & \
FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_M)
#define FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_F FW_TLSTX_DATA_WR_ALIGNPLDSHOVE_V(1U)
#define FW_TLSTX_DATA_WR_FLAGS_S 0
#define FW_TLSTX_DATA_WR_FLAGS_M 0xfffffff
#define FW_TLSTX_DATA_WR_FLAGS_V(x) ((x) << FW_TLSTX_DATA_WR_FLAGS_S)
#define FW_TLSTX_DATA_WR_FLAGS_G(x) \
(((x) >> FW_TLSTX_DATA_WR_FLAGS_S) & FW_TLSTX_DATA_WR_FLAGS_M)
#define FW_TLSTX_DATA_WR_CTXLOC_S 30
#define FW_TLSTX_DATA_WR_CTXLOC_M 0x3
#define FW_TLSTX_DATA_WR_CTXLOC_V(x) ((x) << FW_TLSTX_DATA_WR_CTXLOC_S)
#define FW_TLSTX_DATA_WR_CTXLOC_G(x) \
(((x) >> FW_TLSTX_DATA_WR_CTXLOC_S) & FW_TLSTX_DATA_WR_CTXLOC_M)
#define FW_TLSTX_DATA_WR_IVDSGL_S 29
#define FW_TLSTX_DATA_WR_IVDSGL_M 0x1
#define FW_TLSTX_DATA_WR_IVDSGL_V(x) ((x) << FW_TLSTX_DATA_WR_IVDSGL_S)
#define FW_TLSTX_DATA_WR_IVDSGL_G(x) \
(((x) >> FW_TLSTX_DATA_WR_IVDSGL_S) & FW_TLSTX_DATA_WR_IVDSGL_M)
#define FW_TLSTX_DATA_WR_IVDSGL_F FW_TLSTX_DATA_WR_IVDSGL_V(1U)
#define FW_TLSTX_DATA_WR_KEYSIZE_S 24
#define FW_TLSTX_DATA_WR_KEYSIZE_M 0x1f
#define FW_TLSTX_DATA_WR_KEYSIZE_V(x) ((x) << FW_TLSTX_DATA_WR_KEYSIZE_S)
#define FW_TLSTX_DATA_WR_KEYSIZE_G(x) \
(((x) >> FW_TLSTX_DATA_WR_KEYSIZE_S) & FW_TLSTX_DATA_WR_KEYSIZE_M)
#define FW_TLSTX_DATA_WR_NUMIVS_S 14
#define FW_TLSTX_DATA_WR_NUMIVS_M 0xff
#define FW_TLSTX_DATA_WR_NUMIVS_V(x) ((x) << FW_TLSTX_DATA_WR_NUMIVS_S)
#define FW_TLSTX_DATA_WR_NUMIVS_G(x) \
(((x) >> FW_TLSTX_DATA_WR_NUMIVS_S) & FW_TLSTX_DATA_WR_NUMIVS_M)
#define FW_TLSTX_DATA_WR_EXP_S 0
#define FW_TLSTX_DATA_WR_EXP_M 0x3fff
#define FW_TLSTX_DATA_WR_EXP_V(x) ((x) << FW_TLSTX_DATA_WR_EXP_S)
#define FW_TLSTX_DATA_WR_EXP_G(x) \
(((x) >> FW_TLSTX_DATA_WR_EXP_S) & FW_TLSTX_DATA_WR_EXP_M)
#define FW_TLSTX_DATA_WR_ADJUSTEDPLEN_S 1
#define FW_TLSTX_DATA_WR_ADJUSTEDPLEN_V(x) \
((x) << FW_TLSTX_DATA_WR_ADJUSTEDPLEN_S)
#define FW_TLSTX_DATA_WR_EXPINPLENMAX_S 4
#define FW_TLSTX_DATA_WR_EXPINPLENMAX_V(x) \
((x) << FW_TLSTX_DATA_WR_EXPINPLENMAX_S)
#define FW_TLSTX_DATA_WR_PDUSINPLENMAX_S 2
#define FW_TLSTX_DATA_WR_PDUSINPLENMAX_V(x) \
((x) << FW_TLSTX_DATA_WR_PDUSINPLENMAX_S)
#endif /* _T4FW_INTERFACE_H_ */

2
include/linux/netdev_features.h
View File

@ -79,6 +79,7 @@ enum {
NETIF_F_RX_UDP_TUNNEL_PORT_BIT, /* Offload of RX port for UDP tunnels */
NETIF_F_GRO_HW_BIT, /* Hardware Generic receive offload */
NETIF_F_HW_TLS_RECORD_BIT, /* Offload TLS record */
/*
* Add your fresh new feature above and remember to update
@ -145,6 +146,7 @@ enum {
#define NETIF_F_HW_ESP __NETIF_F(HW_ESP)
#define NETIF_F_HW_ESP_TX_CSUM __NETIF_F(HW_ESP_TX_CSUM)
#define NETIF_F_RX_UDP_TUNNEL_PORT __NETIF_F(RX_UDP_TUNNEL_PORT)
#define NETIF_F_HW_TLS_RECORD __NETIF_F(HW_TLS_RECORD)
#define for_each_netdev_feature(mask_addr, bit) \
for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)

32
include/net/tls.h
View File

@ -56,6 +56,32 @@
#define TLS_RECORD_TYPE_DATA 0x17
#define TLS_AAD_SPACE_SIZE 13
#define TLS_DEVICE_NAME_MAX 32
/*
* This structure defines the routines for Inline TLS driver.
* The following routines are optional and filled with a
* null pointer if not defined.
*
* @name: Its the name of registered Inline tls device
* @dev_list: Inline tls device list
* int (*feature)(struct tls_device *device);
* Called to return Inline TLS driver capability
*
* int (*hash)(struct tls_device *device, struct sock *sk);
* This function sets Inline driver for listen and program
* device specific functioanlity as required
*
* void (*unhash)(struct tls_device *device, struct sock *sk);
* This function cleans listen state set by Inline TLS driver
*/
struct tls_device {
char name[TLS_DEVICE_NAME_MAX];
struct list_head dev_list;
int (*feature)(struct tls_device *device);
int (*hash)(struct tls_device *device, struct sock *sk);
void (*unhash)(struct tls_device *device, struct sock *sk);
};
struct tls_sw_context {
struct crypto_aead *aead_send;
@ -114,7 +140,7 @@ struct tls_context {
void *priv_ctx;
u8 conf:2;
u8 conf:3;
struct cipher_context tx;
struct cipher_context rx;
@ -135,6 +161,8 @@ struct tls_context {
int (*getsockopt)(struct sock *sk, int level,
int optname, char __user *optval,
int __user *optlen);
int (*hash)(struct sock *sk);
void (*unhash)(struct sock *sk);
};
int wait_on_pending_writer(struct sock *sk, long *timeo);
@ -283,5 +311,7 @@ static inline struct tls_offload_context *tls_offload_ctx(
int tls_proccess_cmsg(struct sock *sk, struct msghdr *msg,
unsigned char *record_type);
void tls_register_device(struct tls_device *device);
void tls_unregister_device(struct tls_device *device);
#endif /* _TLS_OFFLOAD_H */

1
net/core/ethtool.c
View File

@ -108,6 +108,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
[NETIF_F_HW_ESP_BIT] = "esp-hw-offload",
[NETIF_F_HW_ESP_TX_CSUM_BIT] = "esp-tx-csum-hw-offload",
[NETIF_F_RX_UDP_TUNNEL_PORT_BIT] = "rx-udp_tunnel-port-offload",
[NETIF_F_HW_TLS_RECORD_BIT] = "tls-hw-record",
};
static const char

1
net/ipv4/tcp_minisocks.c
View File

@ -332,6 +332,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
tcp_update_metrics(sk);
tcp_done(sk);
}
EXPORT_SYMBOL(tcp_time_wait);
void tcp_twsk_destructor(struct sock *sk)
{

114
net/tls/tls_main.c
View File

@ -38,6 +38,7 @@
#include <linux/highmem.h>
#include <linux/netdevice.h>
#include <linux/sched/signal.h>
#include <linux/inetdevice.h>
#include <net/tls.h>
@ -56,11 +57,14 @@ enum {
TLS_SW_TX,
TLS_SW_RX,
TLS_SW_RXTX,
TLS_HW_RECORD,
TLS_NUM_CONFIG,
};
static struct proto *saved_tcpv6_prot;
static DEFINE_MUTEX(tcpv6_prot_mutex);
static LIST_HEAD(device_list);
static DEFINE_MUTEX(device_mutex);
static struct proto tls_prots[TLS_NUM_PROTS][TLS_NUM_CONFIG];
static struct proto_ops tls_sw_proto_ops;
@ -241,8 +245,12 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
lock_sock(sk);
sk_proto_close = ctx->sk_proto_close;
if (ctx->conf == TLS_HW_RECORD)
goto skip_tx_cleanup;
if (ctx->conf == TLS_BASE) {
kfree(ctx);
ctx = NULL;
goto skip_tx_cleanup;
}
@ -276,6 +284,11 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
skip_tx_cleanup:
release_sock(sk);
sk_proto_close(sk, timeout);
/* free ctx for TLS_HW_RECORD, used by tcp_set_state
* for sk->sk_prot->unhash [tls_hw_unhash]
*/
if (ctx && ctx->conf == TLS_HW_RECORD)
kfree(ctx);
}
static int do_tls_getsockopt_tx(struct sock *sk, char __user *optval,
@ -493,6 +506,79 @@ static int tls_setsockopt(struct sock *sk, int level, int optname,
return do_tls_setsockopt(sk, optname, optval, optlen);
}
static struct tls_context *create_ctx(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tls_context *ctx;
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (!ctx)
return NULL;
icsk->icsk_ulp_data = ctx;
return ctx;
}
static int tls_hw_prot(struct sock *sk)
{
struct tls_context *ctx;
struct tls_device *dev;
int rc = 0;
mutex_lock(&device_mutex);
list_for_each_entry(dev, &device_list, dev_list) {
if (dev->feature && dev->feature(dev)) {
ctx = create_ctx(sk);
if (!ctx)
goto out;
ctx->hash = sk->sk_prot->hash;
ctx->unhash = sk->sk_prot->unhash;
ctx->sk_proto_close = sk->sk_prot->close;
ctx->conf = TLS_HW_RECORD;
update_sk_prot(sk, ctx);
rc = 1;
break;
}
}
out:
mutex_unlock(&device_mutex);
return rc;
}
static void tls_hw_unhash(struct sock *sk)
{
struct tls_context *ctx = tls_get_ctx(sk);
struct tls_device *dev;
mutex_lock(&device_mutex);
list_for_each_entry(dev, &device_list, dev_list) {
if (dev->unhash)
dev->unhash(dev, sk);
}
mutex_unlock(&device_mutex);
ctx->unhash(sk);
}
static int tls_hw_hash(struct sock *sk)
{
struct tls_context *ctx = tls_get_ctx(sk);
struct tls_device *dev;
int err;
err = ctx->hash(sk);
mutex_lock(&device_mutex);
list_for_each_entry(dev, &device_list, dev_list) {
if (dev->hash)
err |= dev->hash(dev, sk);
}
mutex_unlock(&device_mutex);
if (err)
tls_hw_unhash(sk);
return err;
}
static void build_protos(struct proto *prot, struct proto *base)
{
prot[TLS_BASE] = *base;
@ -511,15 +597,22 @@ static void build_protos(struct proto *prot, struct proto *base)
prot[TLS_SW_RXTX] = prot[TLS_SW_TX];
prot[TLS_SW_RXTX].recvmsg = tls_sw_recvmsg;
prot[TLS_SW_RXTX].close = tls_sk_proto_close;
prot[TLS_HW_RECORD] = *base;
prot[TLS_HW_RECORD].hash = tls_hw_hash;
prot[TLS_HW_RECORD].unhash = tls_hw_unhash;
prot[TLS_HW_RECORD].close = tls_sk_proto_close;
}
static int tls_init(struct sock *sk)
{
int ip_ver = sk->sk_family == AF_INET6 ? TLSV6 : TLSV4;
struct inet_connection_sock *icsk = inet_csk(sk);
struct tls_context *ctx;
int rc = 0;
if (tls_hw_prot(sk))
goto out;
/* The TLS ulp is currently supported only for TCP sockets
* in ESTABLISHED state.
* Supporting sockets in LISTEN state will require us
@ -530,12 +623,11 @@ static int tls_init(struct sock *sk)
return -ENOTSUPP;
/* allocate tls context */
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
ctx = create_ctx(sk);
if (!ctx) {
rc = -ENOMEM;
goto out;
}
icsk->icsk_ulp_data = ctx;
ctx->setsockopt = sk->sk_prot->setsockopt;
ctx->getsockopt = sk->sk_prot->getsockopt;
ctx->sk_proto_close = sk->sk_prot->close;
@ -557,6 +649,22 @@ out:
return rc;
}
void tls_register_device(struct tls_device *device)
{
mutex_lock(&device_mutex);
list_add_tail(&device->dev_list, &device_list);
mutex_unlock(&device_mutex);
}
EXPORT_SYMBOL(tls_register_device);
void tls_unregister_device(struct tls_device *device)
{
mutex_lock(&device_mutex);
list_del(&device->dev_list);
mutex_unlock(&device_mutex);
}
EXPORT_SYMBOL(tls_unregister_device);
static struct tcp_ulp_ops tcp_tls_ulp_ops __read_mostly = {
.name = "tls",
.uid = TCP_ULP_TLS,