TLS Certificate probes fail ("handshake failure") against Vercel servers
Hi RIPE Atlas folks-- I noticed today that using the "SSL" probe (gathering X.509 certificates from a TLS endpoint) appears to fail against servers operated by Vercel. the failure is a uniform "handshake failure" according to the web interface. I think this might be a bug in the probe's implementation of the client side of TLS. Where could i find the specific implementation of that probe type so that i can debug it further? Or, if this is a known issue, where can i read up on it in more detail? I've demonstrated this by probing Vercel's own servers, since they appear to be eating their own dogfood. You can see this with measurement 49131334: https://atlas.ripe.net/measurements/49131334/#probes here is the definition of the query i made: { "definitions": [ { "target": "vercel.com", "af": 4, "port": 443, "hostname": "vercel.com", "description": "SSL measurement to vercel.com", "resolve_on_probe": true, "skip_dns_check": false, "type": "sslcert" } ], "probes": [ { "type": "area", "value": "WW", "requested": 10 } ], "is_oneoff": true, } This is pretty odd, since normal TLS clients appear to be able to connect to them fine. Here's me running a manual handshake debugger against vercel.com's servers: $ gnutls-cli-debug vercel.com GnuTLS debug client 3.7.8 Checking vercel.com:443 whether the server accepts default record size (512 bytes)... yes whether %ALLOW_SMALL_RECORDS is required... no whether we need to disable TLS 1.2... no whether we need to disable TLS 1.1... no whether we need to disable TLS 1.0... no whether %NO_EXTENSIONS is required... no for TLS 1.0 (RFC2246) support... no for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no for TLS 1.1 (RFC4346) support... no fallback from TLS 1.1 to... failed for TLS 1.2 (RFC5246) support... yes for TLS 1.3 (RFC8446) support... yes for known TLS or SSL protocols support... yes TLS1.2 neg fallback from TLS 1.6 to... failed (server requires fallback dance) for HTTPS server name... Vercel for certificate chain order... sorted for safe renegotiation (RFC5746) support... yes for encrypt-then-MAC (RFC7366) support... no for ext master secret (RFC7627) support... no for heartbeat (RFC6520) support... no for version rollback bug in RSA PMS... yes whether the server ignores the RSA PMS version... yes whether small records (512 bytes) are tolerated on handshake... yes whether cipher suites not in SSL 3.0 spec are accepted... yes whether a bogus TLS record version in the client hello is accepted... yes whether the server understands TLS closure alerts... yes whether the server supports session resumption... yes for anonymous authentication support... no for RSA key exchange support... no |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice for ephemeral Diffie-Hellman support... yes |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice for RFC7919 Diffie-Hellman support... no for ephemeral EC Diffie-Hellman support... yes for VKO GOST-2012 (draft-smyshlyaev-tls12-gost-suites) support... no for curve SECP256r1 (RFC4492)... yes for curve SECP384r1 (RFC4492)... yes for curve SECP521r1 (RFC4492)... yes for curve X25519 (RFC8422)... yes for AES-GCM cipher (RFC5288) support... yes for AES-CCM cipher (RFC6655) support... no for AES-CCM-8 cipher (RFC6655) support... no for AES-CBC cipher (RFC3268) support... no for CAMELLIA-GCM cipher (RFC6367) support... no for CAMELLIA-CBC cipher (RFC5932) support... no for 3DES-CBC cipher (RFC2246) support... no for ARCFOUR 128 cipher (RFC2246) support... no for CHACHA20-POLY1305 cipher (RFC7905) support... yes for GOST28147-CNT cipher (draft-smyshlyaev-tls12-gost-suites) support... no for MD5 MAC support... no for SHA1 MAC support... no for SHA256 MAC support... no for GOST28147-IMIT MAC (draft-smyshlyaev-tls12-gost-suites) support... no for max record size (RFC6066) support... no for OCSP status response (RFC6066) support... yes $ Thanks for any pointers, --dkg
Hi Daniel, I think this may be because the measurement code doesn’t support TLS 1.3 yet, and vercel.com does. It’s a known issue, we’d like to add TLS 1.3 at some point. You can find the relevant code here: https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/blob/7c03fba082e93b7a1f0f14cc3769bb31e83909e3/eperd/sslgetcert.c#L927 ripe-atlas-probe-measurements/sslgetcert.c at 7c03fba082e93b7a1f0f14cc3769bb31e83909e3 · RIPE-NCC/ripe-atlas-probe-measurements github.com Cheers, Michel
On 25 Jan 2023, at 18:16, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
Hi RIPE Atlas folks--
I noticed today that using the "SSL" probe (gathering X.509 certificates from a TLS endpoint) appears to fail against servers operated by Vercel. the failure is a uniform "handshake failure" according to the web interface.
I think this might be a bug in the probe's implementation of the client side of TLS. Where could i find the specific implementation of that probe type so that i can debug it further?
Or, if this is a known issue, where can i read up on it in more detail?
I've demonstrated this by probing Vercel's own servers, since they appear to be eating their own dogfood.
You can see this with measurement 49131334: https://atlas.ripe.net/measurements/49131334/#probes
here is the definition of the query i made:
{ "definitions": [ { "target": "vercel.com", "af": 4, "port": 443, "hostname": "vercel.com", "description": "SSL measurement to vercel.com", "resolve_on_probe": true, "skip_dns_check": false, "type": "sslcert" } ], "probes": [ { "type": "area", "value": "WW", "requested": 10 } ], "is_oneoff": true, }
This is pretty odd, since normal TLS clients appear to be able to connect to them fine. Here's me running a manual handshake debugger against vercel.com's servers:
$ gnutls-cli-debug vercel.com GnuTLS debug client 3.7.8 Checking vercel.com:443 whether the server accepts default record size (512 bytes)... yes whether %ALLOW_SMALL_RECORDS is required... no whether we need to disable TLS 1.2... no whether we need to disable TLS 1.1... no whether we need to disable TLS 1.0... no whether %NO_EXTENSIONS is required... no for TLS 1.0 (RFC2246) support... no for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no for TLS 1.1 (RFC4346) support... no fallback from TLS 1.1 to... failed for TLS 1.2 (RFC5246) support... yes for TLS 1.3 (RFC8446) support... yes for known TLS or SSL protocols support... yes TLS1.2 neg fallback from TLS 1.6 to... failed (server requires fallback dance) for HTTPS server name... Vercel for certificate chain order... sorted for safe renegotiation (RFC5746) support... yes for encrypt-then-MAC (RFC7366) support... no for ext master secret (RFC7627) support... no for heartbeat (RFC6520) support... no for version rollback bug in RSA PMS... yes whether the server ignores the RSA PMS version... yes whether small records (512 bytes) are tolerated on handshake... yes whether cipher suites not in SSL 3.0 spec are accepted... yes whether a bogus TLS record version in the client hello is accepted... yes whether the server understands TLS closure alerts... yes whether the server supports session resumption... yes for anonymous authentication support... no for RSA key exchange support... no |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice for ephemeral Diffie-Hellman support... yes |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice for RFC7919 Diffie-Hellman support... no for ephemeral EC Diffie-Hellman support... yes for VKO GOST-2012 (draft-smyshlyaev-tls12-gost-suites) support... no for curve SECP256r1 (RFC4492)... yes for curve SECP384r1 (RFC4492)... yes for curve SECP521r1 (RFC4492)... yes for curve X25519 (RFC8422)... yes for AES-GCM cipher (RFC5288) support... yes for AES-CCM cipher (RFC6655) support... no for AES-CCM-8 cipher (RFC6655) support... no for AES-CBC cipher (RFC3268) support... no for CAMELLIA-GCM cipher (RFC6367) support... no for CAMELLIA-CBC cipher (RFC5932) support... no for 3DES-CBC cipher (RFC2246) support... no for ARCFOUR 128 cipher (RFC2246) support... no for CHACHA20-POLY1305 cipher (RFC7905) support... yes for GOST28147-CNT cipher (draft-smyshlyaev-tls12-gost-suites) support... no for MD5 MAC support... no for SHA1 MAC support... no for SHA256 MAC support... no for GOST28147-IMIT MAC (draft-smyshlyaev-tls12-gost-suites) support... no for max record size (RFC6066) support... no for OCSP status response (RFC6066) support... yes $
Thanks for any pointers,
--dkg
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi Michael-- Thanks for your followup! More below… On Wed 2023-01-25 18:30:59 +0100, Michel Stam wrote:
I think this may be because the measurement code doesn’t support TLS 1.3 yet, and vercel.com does. It’s a known issue, we’d like to add TLS 1.3 at some point.
Hm, i don't think that's the full story, because the same probe actually succeeds for sites that also support TLS 1.3 (e.g. https://www.aclu.org/). And, when i try to connect to it from a client that has TLS 1.3 deliberately disabled (e.g. "gnutls-cli -priority NORMAL:-VERS-TLS1.3 vercel.com") i still have no problem connecting. Digging into it a bit further, it looks to me like Vercel servers send an alert if we do not emit the ec_point_format TLS extension. This is probably a bug on Vercel's side, but it shouldn't block the Atlas' ability to harvest certificates from it.
You can find the relevant code here: https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/blob/7c03fba082e93...
Thanks for this pointer! I've provided a (mainly untested) pull request with a pretty simple patch that should hopefully fix the issue: https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/pull/15 If anyone on this list has the ability to test this patch and follow up on that issue, i'd appreciate any review. Regards, --dkg
participants (2)
-
Daniel Kahn Gillmor
-
Michel Stam