Repository navigation
Fallback md5 is used, when trying hard to only use OpenSSL #118224
Description
Activity
- changed the title
[-]Fallback md5 is used, when trying hard not to allow it[/-][+]Fallback md5 is used, when trying hard to only use OpenSSL[/+]on Apr 24, 2024 - added 4 commits that reference this issue
on Apr 24, 2024 By
--with-builtin-hashlib-hashes=blake2, you are explicitly saying that you do not want Python to providemd5. By setting base & fips inopenssl.cnf, you're saying that you don't want OpenSSL to providemd5. Taken together, you've disabledmd5.I don't see why should Python should not honor those decisions. If you want md5, why don't you enable one of the things that provide it?
By
--with-builtin-hashlib-hashes=blake2, you are explicitly saying that you do not want Python to providemd5. By setting base & fips inopenssl.cnf, you're saying that you don't want OpenSSL to providemd5. Taken together, you've disabledmd5.I don't see why should Python should not honor those decisions. If you want md5, why don't you enable one of the things that provide it?
It is example of how to trigger the bug. The compliance bypass is that python silently fallsback to internal md5 (when compiled --with-builtin-hashlib-hashes=md5, aka default) when openssl one is incorrectly deemed unavailable (due to provider selection). When in fact, for non security purposes it is there and accessible (aka usage with indicator).
Note, openssl here is not compiled with no-md5.
If you believe that this is as security issue in python, please contact the security team -- see the security page (as linked from the Security policy links on GitHub).
But, I don't see the “compliance bypass”. CPython is not FIPS compliant; there isn't anything to bypass. There are some tools/options that make building a FIPS-certifiable CPython easier, but we're not really in a position to test that they do give you a compliant build.
Asking CPython to bypass system OpenSSL configuration, in order to support a build we don't really test in CI, is... a big thing to ask.Note that building with
--with-builtin-hashlib-hashes=blake2could also be a compliance bypass. (Or are you planning to get that implementation FIPS-certified?)See also recent FIPS discussion.
If you believe that this is as security issue in python, please contact the security team -- see the security page (as linked from the Security policy links on GitHub).
But, I don't see the “compliance bypass”. CPython is not FIPS compliant; there isn't anything to bypass. There are some tools/options that make building a FIPS-certifiable CPython easier, but we're not really in a position to test that they do give you a compliant build. Asking CPython to bypass system OpenSSL configuration, in order to support a build we don't really test in CI, is... a big thing to ask.
The CI tests are already there for the FIPS mode. If desired I can add the extra matrix test case to address this issue as an explicit check.
I'm talking about honoring settings for the NIST algorithms, and using accelerated code-paths by default. And ensuring that existing functionality
usedforsecurity=True/Falseapi of hashlib.md5 actually works correctly as documented. Without changing any other behaviour of any other algorithms. When used with openssl as a cryptographic module, or when used with generic one.Note that building with
--with-builtin-hashlib-hashes=blake2could also be a compliance bypass. (Or are you planning to get that implementation FIPS-certified?)FIPS is consent based. Blocking usage of unapproved algorithms is relevant within a cryptographic module only. And those that want to certify cpython they can choose themselves if they want to block blake2 usage out right, or if they want to implement a runtime FIPS indicator w.r.t. usage of blake2 (such that apps that use blake2 are indicated as running in unapproved mode) which is a possibility under FIPS certification.
See also recent FIPS discussion.
That discussion takes a narrow view of FIPS certification. Hashes that are not used for security purposes do not need to be FIPS certified (ie. siphash, xxhash64, and so on are perfectly fine to use for pyc hashing; and for key-value dictionary implementations). As long as said things are not used to protect data at rest (e.g. file encryption) / in transit (e.g. TLS) / used for authentication (e.g. HTTP basic-auth).
If you believe that this is as security issue in python
it's not a CVE in a classic sense of it. As compliance is frankly arbitrary and conflicting. (i.e. FIPS wants to use FIPS algorithms, and somebody else wants to use Cammelia, and someone else wants to use GOST).
My patch here, is scoped to ensure that MD5 (and in the future SHA1) are treated fairly and correctly: honoring system configuration, using accelerated (openssl) codepath, and have
usedforsecurity=True/Falseoperate as documented.I'm not trying to exert any opinion of how one should configure the cpython builds; nor how one should configure their openssl; or how one should configure their FIPS modules. And ensure that
usedforsecurity=Falseworks correctly in more combinations.I'll need to leave this to security experts. Hopefully the discussion is useful.
One thing I don't get: why don't you load “default” in the OpenSSL config?
AFAIK, md5 works (withusedforsecurity=Falseonly) when you use the “default+fips” providers rather than “base+fips”. I don't see a reason to use the latter, other than not wanting to use “default”.6 remaining items
I do not think it's Python's place to silently override these choices.
is it expected for hashlib.md5 to be always available? why is it always loaded upon hashlib import? Can it be made a lazy import? The documentation has some strong words about shipping python without hashlib.md5.
That's a good question. I guess CPython can go two ways:
- Deprecate
hashlib.algorithms_guaranteed, as it's definitely possible to build Python without some of the algorithms, or - Add strong language to
--with-builtin-hashlib-hashes, saying that if you breakhashlib.algorithms_guaranteedoralgorithms_available, you're on the hook for explaining it to your users.
- Deprecate
you're on the hook for explaining it to your users
Let me try to make this prettier, as the current traceback is very confusing and doesn't explain what is happening (it looks like python internals are broken, rather than runtime expectations do not match the ones that were assumed at build time - i.e. w.r.t. availability of md5).
Reacted by Gregory P. Smith and Petr ViktorinFor FIPS, it's essential that
hashlib.md5doesn't get initialized uponimport hashlib. What's the guidance to get around this and prevent this import error?
How do you get around the fact that modules may be directly consuming md5? E.g.
https://github.com/encode/httpx/blob/7c0cda153d301bde9a011e1dd7157d7e2b20889d/httpx/_auth.py#L177That's really up to the organization that seeks FIPS certification for the system that contains the module.
AFAIK, things like “FIPS modes” are just ways to make it easier to create and audit a system that meets FIPS requirements. It's hard to give guidance for a specific module without knowing the rest of the system.
Anyway, I doubt that
hashlib.md5can't get initialized: you can still use md5 withusedforsecurity=False(but you need to audit all such usage).Reacted by Gregory P. SmithFor FIPS, it's essential that
hashlib.md5doesn't get initialized uponimport hashlib. What's the guidance to get around this and prevent this import error? How do you get around the fact that modules may be directly consuming md5? E.g. https://github.com/encode/httpx/blob/7c0cda153d301bde9a011e1dd7157d7e2b20889d/httpx/_auth.py#L177This is going to be a shameless plug - each distributions choose their own way. In Chainguard Python FIPS Image , I chose to implement this runtime behavior:
$ docker run -ti --entrypoint python cgr.dev/chainguard-private/python-fips:latest-dev Python 3.12.4 (tags/v3.12.4-dirty:8e8a4ba, Jun 26 2024, 12:07:40) [GCC 13.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> hashlib.md5 <built-in function openssl_md5> >>> hashlib.md5() Traceback (most recent call last): File "<stdin>", line 1, in <module> _hashlib.UnsupportedDigestmodError: [digital envelope routines] unsupported >>> hashlib.md5(usedforsecurity=False) <md5 _hashlib.HASH object @ 0x7b3e6ba96930> >>>
That's pretty much what RHEL does too.
If that is the consensus among most Linux distributions shipping FIPS and python..... could we get this upstream into cpython? Because vendor specific behaviour, even when all of them agree in FIPS mode, is counter productive.
Reacted by Petr Viktorin, dooafus, Chris Herborth and Robert SturlaChime in here: https://discuss.python.org/t/python-and-openssl-fips-mode/51389/7
Yes, I imagine that having more than one vendor work on this would make it more acceptable.
Currently it's hard to find the boundary between stuff all FIPS distros need and platform-specific quirks. (On top of the problems of a feature that benefits US government suppliers but is at least inconvenient for almost everyone else.)- added a commit that references this issue
on Sep 19, 2024 The solution might be to support a new compiler option to not fallback to a default MD5 implementation when passing
hashlib.md5(usedforsecurity=True)Reacted by Dimitri John LedkovOk! I had more thoughts on how to do things, and also learning approaches other systems are doing. So I think better implementation can be done which is more intuitive.
Independently of how the behaviour question is resolved, the documentation currently contradicts itself about it, and the wrong half is recent.
Doc/library/hashlib.rst:58-61:Any of these may nonetheless be missing or blocked in unusual environments, such as a rare "FIPS compliant" build of Python or when OpenSSL's "FIPS mode" is configured to exclude some algorithms from its default provider. Calling the constructor of an algorithm that is unavailable raises
ValueError.Doc/library/hashlib.rst:92-95, thirty-one lines below:versionchanged:: 3.12 — For any of the MD5, SHA1, SHA2, or SHA3 algorithms that the linked OpenSSL does not provide we fall back to a verified implementation from the HACL* project.
The second is what the code does —
__hash_newcatches theValueErrorfrom_hashlib.new()and returns the builtin constructor. The first is the paragraph a reader hits first, and it is the one that describes the FIPS case specifically.git blame: the incorrect paragraph is8ae1a236fd33, 2026-06-29; the correct one is3a314f7c3df0, 2023-06-10.So a reader following the current documentation would expect exactly the behaviour this issue reports as missing. Fixing the paragraph seems worth doing regardless of the outcome here, and might also explain some of the expectation mismatch in this thread.
Written with AI assistance; both paragraphs and the blame dates were read at
4f3be1b5777.Strictly speaking, "unavailable" should mean "neither OpenSSL nor HACL*" which is possible (you can disable both of them).
Bug report
Bug description:
When
--with-builtin-hashlib-hashes=blake2to exclude fallback implementation of MD5upon importing hashlib fails to create MD5 construct.
Reference implementation is upstream openssl 3.3.0, with enable-fips, fipsinstall completed and openssl.cnf set to
In essence, things work well only when "default + fips" providers are loaded, as then MD5 functions in OpenSSL are detected as available and are used at runtime and correctly get blocked.
When only "base + fips" providers are loaded, ValueError is raised by OpenSSL constructor, and instead fallback implementation used from _md5 module if it was compiled in.
It seems like the above configuration was not tested, however it can be made to work. CPython should try to load the "default" OpenSSL provider, to guarantee access to non-fips hashes.
Security concerns
This is FedRAMP/FIPS compliance by-pass. This issue may allow using md5 without specifying "usedforsecurity=False" on systems otherwise configured to be in FIPS-mode only. And is the primary reason why documentation mentions that certain distributors of python remove md5 module altogether.
CPython versions tested on:
3.10, 3.11, 3.12
Operating systems tested on:
Linux
Linked PRs