Support OAEP decryption with modulus above 4096

[GabMus]

Riffing off of #210 I wanted to ask if it's possible to add some feature flag or some other opt-in solution to support keys with modulus size bigger than 4096.

I understand the concerns for algorithmic denial of service, but I think having an escape hatch for when you actually want to use this key size for various reasons would be useful, and in fact it's necessary for a particular need I have 😅

[tarcieri]

Which type of keys are you asking about? RsaPublicKey? PKCS#8 or SPKI encoded keys?

RsaPublicKey has new_with_max_size and new_unchecked constructors which can be used to instantiate larger public keys.

PKCS#8 and SPKI are capped to 4096-bits, following recommendations from the X.509 ecosystem. Since these keys are potentially attacker-controlled they're a potential DoS vector if too large. See #166.

[GabMus]

Sorry, I'm in fact referring to PKCS#8 keys in this particular case. Again, as I said I'm well aware of the implication, that's why I was asking specifically for an opt-in solution, something you'd have to explicitly enable. Can this be done?

[tarcieri]

You can parse the PKCS#8 key yourself using the pkcs1 crate and call the appropriate unchecked constructors.

Adding a dedicated feature for this would be rather tricky and seems generally questionable.

Can I ask what use case you have for extremely large PKCS#8 keys?

[GabMus]

In this case it's for keys provided by third parties and we cannot change it.

Regardless the problem does not occur when initializing the RsaPrivateKey but when doing the decryption. Let me show you some example code:

let key = RsaPrivateKey::from_pkcs8_encrypted_pem(my_pem, my_password)?;
let decrypted = key.decrypt(my_oaep_padding_instance, my_encrypted_payload)?; // this errors out with ModulusTooLarge

In particular the error seems to come from RsaPrivateKey::decrypt, calling Oaep::decrypt, calling key::check_public.

This last one then calls key::check_public_with_max_size, and that's where we find the size check code:

    if public_key.n().bits() > max_size {
        return Err(Error::ModulusTooLarge);
    }

Additionally, the max_size value comes from key::RsaPublicKey::MAX_SIZE and it's hardcoded to 4096.

The idea would be to have some sort of override for this functionality, so that we can use whatever we want as a max size, or even an exact expected size.

[tarcieri]

Okay, I agree that's an edge case in OAEP decryption which should be allowed since it's a private key and not a potentially untrusted public key

[GabMus]

Any updates on this? Are you able to give me an ETA, or point me in a direction to propose a PR?

[tarcieri]

You can open a PR which removes this restriction, yes

[Joeydelarago]

@tarcieri I am also running into this issue with max modulus size. I need to verify Pss with RsaPublicKeys that have a larger modulus. I am happy to create a PR for this but I'm not clear on how it should be implemented.

You mention in this PR creating explicit APIs for large modulus. Do you mean to have a separate struct RsaPublicKeyLargeModulus and a separate function verify_large_modulus? Or just to have a RsaPublicKey::from_x_large_modulus() for each of the existing functions that does not limit the size?

Let me know what you expect for this implementation.

Edit: Ah I see the RsaPublicKey new_with_max_size was added a few months ago. I can just decode my key with pkcs1::RsaPublicKey, extract the modulus/exponent from there, and create a rsa::RsaPublicKey.

[tarcieri]

@Joeydelarago this is an issue about using an RSA private key with OAEP decryption. You're asking about PSS verification with an RSA public key, which is only tangentially related. And looks like you found the relevant API.

[tarcieri]

This was fixed in #441