In `modmul', if
* the topmost bit of mod is clear, so mshift is nonzero; and
* both p and q are no more than half as long as mod, so 2*pqlen <=
mlen
then we run this code:
if (mshift) {
for (i = 2 * pqlen - mlen - 1; i < 2 * pqlen - 1; i++)
a[i] = (a[i] << mshift) | (a[i + 1] >> (BIGNUM_INT_BITS - mshift));
But then mlen + 1 > 2*pqlen and therefore i is initially negative
-- and therefore certainly less than 2*pqlen. So the initial access to
a[] is illegal.
Signed-off-by: Mark Wooding <mdw@distorted.org.uk>
pqlen = (p[0] > q[0] ? p[0] : q[0]);
+ /* Make sure that we're allowing enough space. The shifting below will
+ * underflow the vectors we allocate if `pqlen' is too small.
+ */
+ if (2*pqlen <= mlen)
+ pqlen = mlen/2 + 1;
+
/* Allocate n of size pqlen, copy p to n */
n = snewn(pqlen, BignumInt);
i = pqlen - p[0];