Apparently, almost everyone else is using little-endian Serpent.
Since we are going to introduce a new transform, we can make our new
transform have conventional little-endian Serpent. But we need to
retain the big-endian one for compatibility.
In this patch:
* Make serpent.h compile-time bytesexual by providing a new definition
of GETPUT_CP. We default to (conventional) little-endian.
* The big-endian and little-endian versions have different names:
decorate the function names. serpent.h declares both sets.
* Provide serpentbe.c which compiles to a .o implementing the big-endian
version under the new names. Build only that one.
* Change all the call sites to use the big-endian Serpent.
* Mention this issue in the documentation.
Ultimately, no functional change in this patch.
Regarding the endianness of Serpent, Mark Wooding writes:
I've understood how my Serpent implementation differs from Secnet's, and
have reproduced [Ian's EAX] test vectors.
NIST have managed to completely screw up their archive of the AES
contest pages, but the WayBack Machine works fine -- even on the various
PDF documents.
The sorry tale begins with the initial Request for Candidate Algorithm
Nominations, wherein the specification[1] for the test files unhelpfully
muddles things by implying a big-endian representation in the test
vector files, e.g., in 3.1,
: Each of the possible key basis vectors is tested in this manner, by
: shifting the "1" a single position at a time, starting at the most
: significant (left-most) bit position of the key.
It doesn't help that the original Serpent C implementations in the
submission package[2] failed to implement the defined API correctly, and
implicitly coerced BYTE pointer (from the defined entry point
`blockEncrypt' to an `unsigned long' pointer as an argument to the
internal `serpent_encrypt' function. The Java version carefully picks
words out of its input byte array in a little-endian order.
Secnet's implementation is derived from this original reference
implementation. Originally, (v0.03) it had the same bug (only with
`uint8_t' and `uint32_t'), but was patched (v0.1.16) to correct the
obvious dependency on host endianness. Unfortunately, this patch is
wrong: it creates a perfectly portable completely-byte-reversed Serpent
compatible with nothing else. I've not found many other
implementations, but where I have, they agree with me and the Java
reference, and not with Secnet.
[1] http://web.archive.org/web/
20000420040415/http://csrc.nist.gov/encryption/aes/katmct/katmct.htm
[2] http://www.cl.cam.ac.uk/~rja14/Papers/serpent.tar.gz
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>