Commit Graph

28 Commits

Author SHA1 Message Date
David S. Miller 62ba63dc89 sparc64: Set CRYPTO_TFM_REQ_MAY_SLEEP consistently in CAMELLIA code.
We use the FPU and therefore cannot sleep during the crypto
loops.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:44:31 -08:00
David S. Miller b3a3794707 sparc64: Set CRYPTO_TFM_REQ_MAY_SLEEP consistently in DES code.
We use the FPU and therefore cannot sleep during the crypto
loops.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:43:38 -08:00
David S. Miller ce6889515d sparc64: Fix ECB looping constructs in AES code.
Things works better when you increment the source buffer pointer
properly.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:30:07 -08:00
David S. Miller b35d282ef7 sparc64: Set CRYPTO_TFM_REQ_MAY_SLEEP consistently in AES code.
We use the FPU and therefore cannot sleep during the crypto
loops.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:22:03 -08:00
David S. Miller a8d97cef21 sparc64: Fix AES ctr mode block size.
Like the generic versions, we need to support a block size
of '1' for CTR mode AES.

This was discovered thanks to all of the new test cases added by
Jussi Kivilinna.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:20:23 -08:00
David S. Miller 9f28ffc03e sparc64: Fix unrolled AES 256-bit key loops.
The basic scheme of the block mode assembler is that we start by
enabling the FPU, loading the key into the floating point registers,
then iterate calling the encrypt/decrypt routine for each block.

For the 256-bit key cases, we run short on registers in the unrolled
loops.

So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that
get clobbered.

The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this.

So if we have a mix of multi-block and single-block calls, the
single-block unrolled 256-bit encrypt/decrypt can run with some
of the key registers clobbered.

Handle this by always explicitly loading those registers before using
the non-unrolled 256-bit macro.

This was discovered thanks to all of the new test cases added by
Jussi Kivilinna.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 15:19:11 -08:00
David S. Miller 226f7cea94 sparc64: Fix build with mix of modular vs. non-modular crypto drivers.
We tried linking in a single built object to hold the device table,
but only works if all of the sparc64 crypto modules get built the same
way (modular vs. non-modular).

Just include the device ID stub into each driver source file so that
the table gets compiled into the correct result in all cases.

Reported-by: Meelis Roos <mroos@linux.ee>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-11-09 20:55:16 -08:00
David S. Miller 274504f5e6 sparc64: Fix function argument comment in camellia_sparc64_key_expand asm.
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-21 12:04:51 -07:00
David S. Miller 65d5fcf5c2 sparc64: Fix IV handling bug in des_sparc64_cbc_decrypt
The IV wasn't being propagated properly past the first loop
iteration.

This bug lived only because the crypto layer tests for
cbc(des) do not have any cases that go more than one loop.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-17 19:57:30 -07:00
David S. Miller 979e93ca24 sparc64: Add auto-loading mechanism to crypto-opcode drivers.
Just simply provide a device table containing an entry for sun4v cpus,
the capability mask checks in the drivers themselves will take care of
the rest.

This makes the bootup logs on pre-T4 cpus slightly more verbose, with
each driver indicating lack of support for the associated opcode(s).

But this isn't too much of a real problem.

I toyed with the idea of using explicit entries with compatability
fields of "SPARC-T4", "SPARC-T5", etc. but all future cpus will have
some subset of these opcodes available and this would just be one more
pointless thing to do as each new cpu is released with a new string.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-15 09:46:25 -07:00
David S. Miller 71741680a9 sparc64: Add missing pr_fmt define to crypto opcode drivers.
The hashes and crc32c had it, only the AES/DES/CAMELLIA drivers were
missing it.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-15 09:17:10 -07:00
David S. Miller 1080362425 sparc64: Adjust crypto priorities.
Make the crypto opcode implementations have a higher priority than
those provides by the ring buffer based Niagara crypto device.

Also, several crypto opcode hashes were not setting the priority value
at all.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-15 09:06:30 -07:00
David S. Miller 699871bc94 sparc64: Fix bugs in unrolled 256-bit loops.
Some dm-crypt testing revealed several bugs in the 256-bit unrolled
loops.

The DECRYPT_256_2() macro had two errors:

1) Missing reload of KEY registers %f60 and %f62

2) Missing "\" in penultimate line of definition.

In aes_sparc64_ecb_decrypt_256, we were storing the second half of the
encryption result from the wrong source registers.

In aes_sparc64_ctr_crypt_256 we have to be careful when we fall out of
the 32-byte-at-a-time loop and handle a trailing 16-byte chunk.  In
that case we've clobbered the final key holding registers and have to
restore them before executing the ENCRYPT_256() macro.  Inside of the
32-byte-at-a-time loop things are OK, because we do this key register
restoring during the first few rounds of the ENCRYPT_256_2() macro.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-01 23:05:43 -07:00
David S. Miller 7cff82f5f4 sparc64: Avoid code duplication in crypto assembler.
Put the opcode macros in a common header

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-31 12:11:51 -07:00
David S. Miller 4e71bb49f2 sparc64: Unroll CTR crypt loops in AES driver.
Before:

testing speed of ctr(aes) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes)

testing speed of ctr(aes) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes)

After:

testing speed of ctr(aes) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes)

testing speed of ctr(aes) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes)

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 08:40:44 -07:00
David S. Miller 301013159e sparc64: Unroll ECB decryption loops in AES driver.
Before:

testing speed of ecb(aes) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes)

After:

testing speed of ecb(aes) decryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes)

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 08:11:01 -07:00
David S. Miller 03d168ad12 sparc64: Unroll ECB encryption loops in AES driver.
The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a
time we avoid a pipeline bubble in between every round.

For the 256-bit key case, it looks like we're doing more work in
order to reload the KEY registers during the loop to make space
for scarce temporaries.  But the load dual issues with the AES
operations so we get the KEY reloads essentially for free.

Before:

testing speed of ecb(aes) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes)

After:

testing speed of ecb(aes) encryption
test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes)
test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes)
test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes)
test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes)
test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes)
test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes)
test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes)
test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes)
test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes)
test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes)
test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes)
test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes)
test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes)
test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes)
test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes)

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 07:51:32 -07:00
David S. Miller 9fd130ecbe sparc64: Add ctr mode support to AES driver.
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 14:49:23 -07:00
David S. Miller 0bdcaf7495 sparc64: Move AES driver over to a methods based implementation.
Instead of testing and branching off of the key size on every
encrypt/decrypt call, use method ops assigned at key set time.

Reverse the order of float registers used for decryption to make
future changes easier.

Align all assembler routines on a 32-byte boundary.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 14:22:23 -07:00
David S. Miller 45dfe237a8 sparc64: Use fsrc2 instead of fsrc1 in sparc64 hash crypto drivers.
On SPARC-T4 fsrc2 has 1 cycle of latency, whereas fsrc1 has 11 cycles.

True story.

Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-28 20:55:19 -07:00
David S. Miller 81658ad0d9 sparc64: Add CAMELLIA driver making use of the new camellia opcodes.
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-28 20:51:24 -07:00
David S. Miller c5aac2df65 sparc64: Add DES driver making use of the new des opcodes.
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-25 22:37:23 -07:00
David S. Miller 442a7c40b1 sparc64: Add CRC32C driver making use of the new crc32c opcode.
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-22 20:47:36 -07:00
David S. Miller 9bf4852d3d sparc64: Add AES driver making use of the new aes opcodes.
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-22 14:29:33 -07:00
David S. Miller fa4dfedcc2 sparc64: Add MD5 driver making use of the 'md5' instruction.
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-20 15:08:50 -07:00
David S. Miller 775e0c6998 sparc64: Add SHA384/SHA512 driver making use of the 'sha512' instruction.
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-20 15:08:50 -07:00
David S. Miller 86c93b24ef sparc64: Add SHA224/SHA256 driver making use of the 'sha256' instruction.
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-20 15:08:49 -07:00
David S. Miller 4ff28d4ca9 sparc64: Add SHA1 driver making use of the 'sha1' instruction.
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-20 15:08:49 -07:00