diff mbox series

dragonfly: Fix legendre symbol calculation failure handling

Message ID 20231106171915.292787-1-Chaitanya.Tata@nordicsemi.no
State Accepted
Headers show
Series dragonfly: Fix legendre symbol calculation failure handling | expand

Commit Message

Krishna Chaitanya Nov. 6, 2023, 5:19 p.m. UTC 
In case of low-memory conditions, the computation for legendre symbol
can fail and return -2 as per documentation, but the check for that
was missed here. And this can can cause an infinite loop searching for
qr and qnr.

Break the loop if calculation fails, we can leave retry to the callers
or user.

Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
---
 src/common/dragonfly.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Comments

Jouni Malinen Nov. 6, 2023, 6:57 p.m. UTC | #1 
On Mon, Nov 06, 2023 at 10:49:15PM +0530, Chaitanya Tata wrote:
> In case of low-memory conditions, the computation for legendre symbol
> can fail and return -2 as per documentation, but the check for that
> was missed here. And this can can cause an infinite loop searching for
> qr and qnr.
> 
> Break the loop if calculation fails, we can leave retry to the callers
> or user.

Thanks, applied.
diff mbox series

Patch

diff --git a/src/common/dragonfly.c b/src/common/dragonfly.c
index 1e8427166..a0befe537 100644
--- a/src/common/dragonfly.c
+++ b/src/common/dragonfly.c
@@ -71,8 +71,11 @@  int dragonfly_get_random_qr_qnr(const struct crypto_bignum *prime,
 			*qr = tmp;
 		else if (res == -1 && !(*qnr))
 			*qnr = tmp;
-		else
+		else {
 			crypto_bignum_deinit(tmp, 0);
+			if (res == -2)
+				break;
+		}
 	}
 
 	if (*qr && *qnr)