Identify TLS handshake failures without mistaking TCP errors for TLS errors

The clearest sign of a TLS handshake failure is a decoded fatal TLS error alert during connection setup, before the handshake completes. In Wireshark, start with tls.alert_message.desc, identify the sender and alert description, then inspect the preceding messages in that connection.

A reset, silence after ClientHello, or a missing Finished message is less conclusive. Those observations can reflect a transport problem, an incomplete capture, or encrypted handshake messages—not necessarily a TLS negotiation rejection.

This workflow covers TLS over TCP, not QUIC or DTLS. Inspect only systems and traffic you own or are authorized to troubleshoot.

1. Capture the attempt and isolate one connection

Start capturing before reproducing the failure. Include both directions and keep capturing until the application reports its error. Record that error and its timestamp so you can distinguish the failed attempt from background connections or retries.

Use this display filter to locate ClientHello messages:

tls.handshake.type == 1

Select the relevant packet, then choose Analyze → Follow → TCP Stream. Close the dialog with Close, rather than Back, to leave the stream filter applied. Wireshark documents this as a quick way to isolate a connection in its Follow Stream guide.

Your filter will look like:

tcp.stream == 7

Here, 7 is an example stream index; use the value from your capture. Keep this full-stream view available: filtering only on tls can hide the TCP packets that explain a stall or closure.

If no ClientHello appears:

• Check whether the capture began after connection setup or whether the application reused an existing connection.
• Check the interface, capture point, and whether both directions are visible.
• For a known TLS service on an unusual TCP port, select a packet and use Analyze → Decode As… to choose TLS for that port. This changes Wireshark’s interpretation, not the traffic itself; see its protocol dissection controls.
• If the ordinary TCP connection never establishes, investigate transport connectivity first. A TLS display filter alone cannot reveal that failure.

2. Look for alerts, then read the packet details

Apply these filters within the isolated connection. The fields are documented in Wireshark’s TLS filter reference and TCP filter reference.

What to inspect	Example display filter
Alerts with a decoded description	tcp.stream == 7 && tls.alert_message.desc
Specifically Handshake Failure, code 40	tcp.stream == 7 && tls.alert_message.desc == 40
Decoded fatal-level alerts	tcp.stream == 7 && tls.alert_message.level == 2
Visible alert records, including encrypted TLS 1.2 alerts	tcp.stream == 7 && tls.record.content_type == 21
Handshake messages with a decoded type	tcp.stream == 7 && tls.handshake.type
Resets or FIN closures	tcp.stream == 7 && (tcp.flags.reset == 1 || tcp.flags.fin == 1)

The broader tls.alert_message filter can also match an Encrypted Alert entry without a readable description. Its presence alone does not mean Wireshark decoded the reason; this distinction is visible in Wireshark’s TLS alert dissector.

For a readable alert packet, expand Transport Layer Security → Record Layer → Alert Message in the packet-details pane. Record:

1. Sender: client or server, determined from the ClientHello direction.
2. Description: the reason reported by that sender.
3. Level: warning or fatal, where relevant to the TLS version.
4. Stage: the last handshake message visible before the alert.

In TLS 1.2, a fatal-level alert terminates the connection. In TLS 1.3, error descriptions determine severity regardless of the legacy level field. Neither close_notify nor user_canceled means a protocol negotiation error: the former signals closure, and the latter signals cancellation for a reason unrelated to protocol failure. These distinctions come from the TLS 1.2 alert protocol and TLS 1.3 alert protocol.

Do not search only for code 40. Other alerts can explain why setup failed.

3. Use the alert to choose the next check

An alert reports the sender’s reason for stopping or rejecting the exchange. It is not always a complete root-cause diagnosis.

Alert description	What it tells you	Next check
handshake_failure (40)	The sender could not negotiate acceptable security parameters.	Compare the client’s offers with server policy: versions, cipher suites, signature algorithms, and groups. Do not assume a cipher mismatch from this code alone.
protocol_version (70)	The attempted version is recognized but unsupported.	Compare offered versions with enabled server versions.
unknown_ca (48)	The sender could not accept the peer’s chain against its trust anchors.	Check the chain supplied by the peer and the sender’s trust configuration.
certificate_expired (45)	The sender considers the certificate expired or not yet valid.	Check certificate validity dates and the validating endpoint’s clock.
certificate_required (116, TLS 1.3)	The server required a client certificate but none was provided.	Check mutual-TLS requirements and client certificate configuration.
unrecognized_name (112)	The server did not recognize the name supplied through the server-name extension.	Check the visible SNI and the TLS virtual-host configuration.
no_application_protocol (120)	The client’s ALPN offers contained no protocol supported by the server.	Compare the client’s application-protocol offers with server configuration.

The meanings and numeric codes are defined in RFC 8446’s alert protocol.

Direction matters. A client sending unknown_ca after receiving a server certificate points toward the client’s rejection of that certificate chain. A server sending it after client authentication points toward rejection of the client’s chain.

A schematic sequence illustrates the distinction:

Client → Server: ClientHello
Server → Client: Fatal alert, Handshake Failure (40)
Server → Client: TCP connection closes

That sequence supports “the server rejected TLS negotiation before sending ServerHello.” It does not establish which parameter caused the rejection. Correlate the attempt with the TLS-terminating endpoint’s logs—not necessarily an application server behind a proxy.

4. Account for encrypted handshake messages

TLS 1.3 encrypts all handshake messages after ServerHello, including Certificate and Finished. Protected records use an outer Application Data type, even when their contents are handshake messages or alerts. Consequently:

• Missing Certificate or Finished entries do not prove failure.
• An Application Data label does not prove the handshake completed.
• An empty alert-filter result does not prove no alert was sent.

In TLS 1.2, the initial full handshake exposes more messages before ChangeCipherSpec. After a sender’s ChangeCipherSpec, its Finished and subsequent alerts are protected. Resumed handshakes also omit parts of a full certificate exchange. See the TLS 1.3 record definitions and TLS 1.2 handshake sequences.

Two other normal patterns can look suspicious:

• TLS 1.2 shown in a TLS 1.3 legacy version field: inspect the ServerHello supported_versions extension; TLS 1.3 selects 0x0304 there while retaining 0x0303 in its legacy field.
• HelloRetryRequest followed by another ClientHello: this can be a normal request for a suitable key share, not a failed attempt.

Both behaviors are specified in RFC 8446’s hello-message definitions.

If you need to see the protected messages, reproduce the issue in an authorized test session using an application that supports TLS session-key logging. Configure the matching key-log file under Preferences → Protocols → TLS → (Pre)-Master-Secret log filename. Wireshark’s TLS documentation explains the process and reassembly requirements; a server RSA private key cannot decrypt TLS 1.3 or an (EC)DHE exchange.

Treat key logs and decrypted captures as sensitive. Keep logging narrowly scoped, stop it after the test, and do not publish secrets, payloads, tokens, addresses, or identifiers. For the broader visibility boundary, see what HTTPS hides in a capture.

5. When there is no readable alert, classify the evidence

Return to the full tcp.stream view and inspect the sequence around the stall.

Observation	Defensible interpretation
TCP establishes; ClientHello is followed by silence	No TLS response is visible at this capture point. Check acknowledgments, capture completeness, and endpoint logs before calling it a TLS rejection.
ClientHello bytes are repeatedly retransmitted without acknowledgment	Investigate transport delivery or capture visibility first. Retransmission is not a TLS alert.
ClientHello is acknowledged, but no TLS response follows	TCP acknowledged the bytes; that does not prove the TLS implementation accepted or processed them successfully.
RST follows a handshake message	The transport was reset during setup; the packet alone does not provide a TLS reason or establish the physical origin of the reset.
A TLS 1.2 encrypted alert is visible	An alert was sent, but its description and severity remain unknown without decryption or endpoint logs.
Decoded Finished messages are followed by a decrypted application request and response	The initial handshake progressed into application exchange; investigate a later TLS or application failure separately.

Wireshark labels retransmissions as suspected in its TCP field reference. Missing captured segments can also prevent useful TLS reassembly. For a reset-centered investigation, continue with diagnosing TCP resets.

Report the narrowest finding the evidence supports: “server sent protocol_version after ClientHello,” “connection reset during setup with no readable TLS alert,” or “capture cannot show whether the encrypted handshake completed.” Include the stream index, frame numbers, timestamp, sender, and last visible handshake stage so the packet evidence can be matched to endpoint logs.