Support certificate validation TLS alerts for Linux

[rzikm]

Contributes to #18837.

Summary

On Linux, SslStream previously closed the transport without sending a TLS alert when certificate validation failed. This PR makes SslStream send appropriate TLS alerts (e.g. bad_certificate, unknown_ca, certificate_expired, certificate_required) by moving certificate validation inline into the OpenSSL handshake via SSL_CTX_set_cert_verify_cb.

How it works

Instead of validating the peer certificate after the handshake completes (post-handshake), the validation now runs inside OpenSSL's certificate verification callback during the handshake itself. This allows OpenSSL to send the appropriate TLS alert to the peer when validation fails, since the handshake is still in progress.

The CertVerifyCallback (registered on the SSL_CTX) extracts the certificate chain from the X509_STORE_CTX, then calls the existing managed VerifyRemoteCertificate logic — including the user's RemoteCertificateValidationCallback if one is registered. If validation fails, the callback maps the failure reason to an X509_V_ERR_* status code, which OpenSSL translates into the corresponding TLS alert.

Connection info during the callback

Because the callback fires mid-handshake, connection properties like SslProtocol and CipherAlgorithm would normally not be available yet. To preserve backward compatibility with user callbacks that access these properties, _connectionInfo is populated before invoking the user callback. On TLS 1.2, SSL_get_current_cipher() returns NULL at this point (the session cipher is set at ChangeCipherSpec), so the native code falls back to SSL_get_pending_cipher() which is available once ServerHello has been processed.

Propagating exceptions from validation callback

The exception that would normally be thrown from the cert validation (either AuthenticationException, or whatever user code throws) is caught and stored in SafeSslHandle before returning back to OpenSSL code, and when the control flow returns from SSL_do_handshake, the exception gets rethrown.

Client certificates and ClientCertificateRequired

OpenSSL only invokes the cert verify callback when the peer actually sends a certificate. When a server sets ClientCertificateRequired = true but the client sends no certificate, the callback never fires. Two cases are handled:

• No user callback registered: SSL_VERIFY_FAIL_IF_NO_PEER_CERT is set alongside SSL_VERIFY_PEER, so OpenSSL itself rejects the handshake and sends the certificate_required (TLS 1.3) or handshake_failure (TLS 1.2) alert.
• User callback registered: Only SSL_VERIFY_PEER is set. The handshake succeeds, and managed CompleteHandshake code detects that _remoteCertificate is still null, then calls VerifyRemoteCertificate which invokes the user's callback with SslPolicyErrors.RemoteCertificateNotAvailable. This preserves the existing behavior where applications can accept connections without a client certificate via the callback, unfortunately, no TLS alert is sent in this case.

Resumed sessions

On resumed sessions, OpenSSL skips the cert verify callback since the certificate was already validated in the original handshake. CompleteHandshake detects this (_remoteCertificate is null) and retrieves the peer certificate from the SSL handle via GetRemoteCertificate and valites it post-handshake, previous behavior is preserved (no TLS alert is sent if revalidation fails, but this is not expected to happen often in practice).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.

[wfurt]

LGTM. left few comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 1 comment.

[liveans]

LGTM

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

[bartonjs]

Is it possible for the same chain to get here multiple times (since it's a parameter)? If so, this list could get really, really long.

[rzikm]

In theory if there is TLS 1.2 renegotiation (which basically another handshake mid-connection) and the other side decides to switch certificates, but TLS implementations heavily restrict renegotiation (and enforce identity continuity) so it shouldn't happen in practice.

[bartonjs]

Does the restore need to be around the block, or just the if?

[rzikm]

It needs to go on the block

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.