obs-outputs: upgrade mbedtls dependency to v4

[dsimidzija]

Description

The official mbedtls migration guide was used for the following:

• update cmake target modules and versions (mbedcrypto => tfpsacrypto)
• remove deprecated entropy configuration (ctr_drbg + entropy)
• migrate MD5 and HMAC to the new PSA API

The official migration guide:

https://github.com/Mbed-TLS/mbedtls/blob/development/docs/4.0-migration-guide.md

I've done the best I can to cover the guide material, but there are some concerns:

• I've never used cmake, so I don't know if any further changes are needed there, for example, there is a message in FindMbedTLS.cmake which states:

  Use of the custom CMake find module for MbedTLS versions >= 3.6.0 is not supported - build errors might occur!

  Does this mean that this file should be removed, as it would now be obsolete?

• I've added calls to psa_crypto_init() for both HMAC and MD5 macros, which should be called before any crypto functions according to the new docs. Should this be moved to obs_module_load, as a counterpart to mbedtls_psa_crypto_free? My gut tells me yes, but I wanted to double check what is the convention in OBS.

• Old code doesn't seem to have any error handling, so I've replicated that, but that makes me uneasy, the new PSA API can fail on each step (you can see it in the hmac_demo.c with PSA_CHECK). However, I'm not sure how to implement that cleanly with the way these macros are used. I guess I'm just asking for confirmation that this is okay as-is.

Motivation and Context

This aims to close #13601 .

How Has This Been Tested?

I've tested RTMP output with a local test server, with and without TLS (self-signed certificate). While I was able to establish a handshake in both versions, I was unable to view the stream when using RTMPS, but I suspect this might be due to the test server malfunctioning, as RTMP ingestion seemed to work without issues. Unfortunately I don't have any other place to test this on a live/prod server.

Additionally, I only have a Linux machine, so I have no means to check if this code works on other platforms.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change) - breaking in the sense of backward compatibility with mbedtls v2/v3

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[tytan652]

Keeping compat with v3 at build time is required.

[dsimidzija]

I was afraid that might be a requirement. 😀

I don't think that will be a problem to accomplish with the code itself (I'm guessing it's enough to just rely on MbedTLS_VERSION_MAJOR). But I might need some help with the cmake stuff, does this mean that FindMbedTLS.cmake shouldn't be changed at all, provided it won't ever be used for v4?

[tytan652]

You should rebase, we recently drop the custom finder (there was already things in the work to remove it).

Compatibility with MbedTLS 3 is required, we guarded to exclude 4 to reduce the number of report about not supporting 4 and having hopefully a better CMake error than include and linking issue.