SSL/TLS handshake, cipher suites and and schannel error explained

This document provides a comprehensive overview of the SSL/TLS communication handshake. It aims to help IT administrators and support engineers identify, analyze, and resolve issues related to cipher suite mismatches, protocol incompatibilities, and TLS negotiation failures.

SSL/TLS Communication Overview

The client (MFD) securely connects to the server (YSoft SafeQ Terminal Server) as follows:

1. TCP three-way handshake is established:

   • SYN - client sends connection request to the server

   • SYN-ACK - sever confirms it

   • ACK - client confirms everything is all right

     CODE

     Wireshark example
     9067 10:51:16,730720 172.24.225.206 172.24.170.12 TCP 66 1199 → 5012 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
     9075 10:51:16,763308 172.24.170.12 172.24.225.206 TCP 66 5012 → 1199 [SYN, ACK, ECN] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM=1
     9076 10:51:16,763424 172.24.225.206 172.24.170.12 TCP 54 1199 → 5012 [ACK] Seq=1 Ack=1 Win=131328 Len=0

2. SSL/TLS handshake is established:
   ℹ️ This takes place inside the existing TCP connection

   1. ClientHello: client declares the highest supported SSL/TLS version and cipher suites list

   2. ServerHello: server evaluates the ClientHello and responds by one cipher suite supported by both sides
      Server-side evaluation criteria:

      • protocol version validation - the server ensures the client's SSL/TLS version meets its security policy

      • cipher suite negotiation - the server compares the client’s cipher suite list with its own allowed list

        • the most secure matching cipher suite is sent in ServerHello

          

          

        • if no match exists, connection is terminated (RST packet and "schannel" error in the MS Windows System Event log).

          

   3. Certificate exchange and key negotiation follow.

3. Client and Server communicate over encrypted channel.

Mitigating SSL/TLS version or cipher suite mismatch

Client and Server must support the same SSL/TLS version and they must share at least one common cipher suite.

This can be achieved by:

• Using up to date software on both client and server side
  e.g. version of MFD Firmware, MS Windows, software

• Re-configuring sever and the client

MFDs reconfiguration

Consult MFD vendor, use diagnostic tools like Wireshark , SSLscan or Nmap to review cipher suite support.

Software reconfiguration

Software may:

• have its own configuration to define security standards

• inherit security settings from the hosting operating system

• combine both of above

For example YSoft SafeQ Terminal as a .NET based application:

• allows configuring SSL / TLS protocol version for outgoing communication from within the application

• but it inherits cipher suites from Operating System

• for details refer to:
  How to enforce TLS 1.2 and TLS 1.3 for Terminal Server

MS Windows cipher suite described at:
https://docs.microsoft.com/en-gb/windows/win32/secauthn/cipher-suites-in-schannel

• MS Windows cipher suite support can be reordered/modified as described at:
  http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspx

  • The restart of server is always required after the change.

  • On domain-joined systems, Group Policy may override local cipher suite settings. Changes might appear successful but have no effect. Coordinate with the domain administrator to resolve.

• PowerShell may be used to obtain the ordered list of OS supported cipher suites:
  $(Get-TlsCipherSuite).Name

• PowerShell may help with testing the ability to establish SSL/TLS connection from a server to a remote site running on a specific IP address bypassing DNS translation:
  Invoke-WebRequest https://40.108.200.25/_api/v2.0 -Headers @{ host="ysoftcorporation-my.sharepoint.com" }

• Certificates can be based on RSA, ECDSA (or other). The certificate's signature algorithm determines the type of cryptographic algorithm for signing and key exchange:

  • RSA based certificate cannot be used with cipher suite TLS_ECDHE_ECDSA*

  • ECDSA signed certificate cannot be used with cipher suite TLS_ECDHE_RSA*

Useful tools

SSLscan

can be used to list the cipher suites and TLS version supported by the listening software.

• Tool is available for download at:
  https://github.com/rbsec/sslscan/releases

• The cipher suite naming convention of the SSLscan and MS Windows / Wireshark slightly differ, the conversion table can be used for a better understanding:
  https://www.openssl.org/docs/man1.1.1/man1/ciphers.html

  • for example Wireshark/MS Windows name TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA equals to the ECDHE-ECDSA-AES256-SHA in SSLscan.

• Example of usage: sslscan.exe <DestinationAddress>:<DestinationPort>

• The image below demonstrates the default cipher suites and how the behavior changes when the order is modified in a group policy editor:

  

  

Nmap

is an alternative to SSLScan, it also lists the cipher suites and TLS version supported by the listening software.

• Tool is available for download at:
  https://nmap.org

• The disadvantage is it requires installation.

• The benefit is it lists the ciphers exactly as shown in Windows and Wireshark, there is no need for a conversion table like with SSLScan.

• Example of usage: nmap --script ssl-enum-ciphers -p <DestinationPort> <DestinationAddress>

  

Wireshark

Tool is available for download at:
https://www.wireshark.org/download.html

An example of cipher suite review in Wireshark

1
2
3
4