Fixed: TSL Handshake Error in Windows 11

Microsoft has addressed a known issue with TLS handshake failures in Windows 11 systems. This problem, caused by Windows security updates released during the recent Patch Tuesday on October 11th, has been resolved with the release of the KB5018496 preview cumulative update. Users on affected Windows 11 devices may have encountered SEC_E_ILLEGAL_MESSAGE errors in applications when connecting to servers. These errors indicate failed SSL/TLS handshake connections.

To address this issue, Microsoft has specifically targeted the affected connections that might experience handshake failures during SSL and TLS communication. Developers may notice that these connections receive one or more records followed by a partial record with a size of lesser than 5 bytes within a single input buffer.

It is crucial to have a secure connection between the user system, server, and website to prevent potential damage or corruption. When attempting to establish a connection with an insecure website, users may encounter the TLS handshake failed message, often accompanied by error codes like 501 or 525. This failure indicates that the TLS and the visitor’s browser were unable to establish a secure connection, posing a risk to the website.

For users facing this issue, it can be frustrating and confusing. However, there are several fixes available to address the problem. In this post, we have gathered relevant information and solutions to assist users experiencing TLS handshake errors in Windows 11. By following the provided guidance, users can navigate and resolve these connection issues, ensuring secure and uninterrupted browsing experiences.

Table of Contents

How to Fix TSL Handshake Error in Windows 11?

To fix TLS handshake errors in Windows 11, you can try the following solutions:

1. Update Windows:

Ensure that your Windows 11 system is up to date with the latest patches and updates. Microsoft often releases fixes for known issues, including TLS handshake errors, through Windows updates. Here is how you are going to do it:

Click on the Start button and select Settings (the gear icon).
In the Settings window, click on Windows Update from the list.
Click on Check for updates and let Windows search for the latest updates.
If updates are available, click on Download and wait for the process to complete.
Restart your computer if prompted.

2. Install KB5018496 Preview Cumulative Update Manually:

If you haven’t already, install the KB5018496 preview cumulative update. This update addresses the TLS handshake failure issue specifically on Windows 11 systems.

Open the Microsoft Update Catalog website in your web browser.
Search for “KB5018496” using the search bar on the website.
Locate the preview cumulative update for Windows 11 that matches your system architecture (e.g., 64-bit or 32-bit).
Click on the download link next to the update to start downloading the update file.
Once the download is complete, double-click on the downloaded file to install the update.
Follow the on-screen instructions to complete the installation process.
Restart your computer after the update installation is finished.

If you have already installed the latest update and still experience the TLS Handshake error on Windows 11 then use the other methods given.

3. Check Date and Time Settings:

Incorrect date and time settings can cause TLS handshake errors. Verify that your system’s date, time, and time zone settings are accurate.

Right-click on the clock in the taskbar.
Select Adjust date/time from the menu.
In the Date & Time settings window, ensure that the Set time automatically and Set time zone automatically options are turned on.
If the settings are already enabled, toggle them off and then back on to refresh the settings.
Verify that the date, time, and time zone displayed are correct for your location.
If any changes were made, restart your computer to apply the new settings.

This is a simple yet effective way to fix the TLS error on Windows 11.

4. Disable Antivirus/Firewall:

Temporarily disable your antivirus software or firewall to check if they are causing conflicts with the TLS handshake process. If the error disappears after disabling them, consider adjusting the settings or contacting the software vendor for further assistance.

Locate the antivirus or firewall software icon in the system tray or notification area.
Right-click on the icon and look for an option to disable or pause the protection temporarily.
Select the appropriate option to disable the antivirus or firewall software.
Attempt to establish the TLS handshake connection again and see if the error persists.
After troubleshooting, remember to enable the antivirus or firewall software to restore your system’s protection.

5. Clear SSL/TLS Cache:

Clearing the SSL/TLS cache can resolve potential conflicts or corrupted data that may be causing the handshake error. Open the Command Prompt as an administrator and run the following command: “ipconfig /flushdns”.

Press the Windows key + R on your keyboard to open the Run dialog box.
Type “cmd” and press Enter to open the Command Prompt.
In the Command Prompt window, type the following command and press Enter: “ipconfig /flushdns“
Wait for the command to execute and clear the DNS cache.
Close the Command Prompt window.

Check if the error is fixed or not. Users who are still experiencing the issue should use the next method.

6. Reset Internet Options:

Resetting Internet Options to their default settings can help resolve the error.

Press the Windows key + R on your keyboard to open the Run dialog box.
Type “inetcpl.cpl” and press Enter to open the Internet Properties window.
In the Internet Properties window, go to the Advanced tab.
Click on the Reset button under the Reset Internet Explorer settings section.
Check the box that says Delete personal settings and click on the Reset button.
Wait for the process to complete and then click on Close.

Restart your computer to apply the changes.

7. Check Proxy Settings:

If you are using a proxy server, ensure that the settings are configured correctly. Incorrect proxy settings can interfere with TLS handshake connections. Disable the proxy temporarily to see if it resolves the issue.

Press the Windows key + R on your keyboard to open the Run dialog box.
Type “inetcpl.cpl” and press Enter to open the Internet Properties window.
In the Internet Properties window, go to the Connections tab and click on the LAN settings button.
In the LAN Settings window, ensure that the Automatically detect settings option is checked.
- If you are using a proxy server, ensure the proxy server address and port are entered correctly.
- If you need to make changes, modify the proxy server settings as required.
Click on OK to save the changes.
After verifying or adjusting the proxy settings, close the Internet Properties window.
Attempt to establish the TLS handshake connection again and check if the error persists.

If you are still experiencing the TLS handshake error, you may consider contacting your network administrator or internet service provider for further assistance. They can provide specific guidance based on your network configuration.

Additionally, you can search online for the specific error message or error code you are encountering. Often, there are community forums or knowledge-based articles that provide solutions or workarounds for common TLS handshake errors.

8. Disable Non-Essential Services:

NOTE: Remember to exercise caution and create backups of your important data before making any significant changes to your system.

Some background services or processes may interfere with the TLS handshake. Use the Windows Task Manager to identify non-essential services or processes consuming excessive resources and temporarily disable them to troubleshoot the issue.

Here are the steps to disable non-essential services in Windows 11 to troubleshoot TLS handshake errors:

Press the Ctrl+Shift+Esc keys together on your keyboard to open the Task Manager. Alternatively, you can right-click on the taskbar and select Task Manager from the menu.
In the Task Manager window, click on the More Details option at the bottom-left corner of the window. This will expand the Task Manager and show additional details.
Click on the Services tab at the top of the Task Manager window. This will display a list of services running on your system.
By default, the services are listed alphabetically. Look through the list of services and identify any non-essential services that may be consuming excessive resources.

To determine if a service is non-essential, you can perform a quick internet search using the service name as a keyword. This will help you find information about the service and whether it is safe to temporarily disable it.

Once you have identified a non-essential service that you want to disable, right-click on the service in the list and select Stop from the context menu. This will stop the service from running.
If prompted with a warning message about stopping a service, carefully review the message to ensure that you are stopping the correct service. Click on Yes to confirm the action.
Use the same method to disable any other non-essential services you want to disable.

After disabling the non-essential services, attempt to establish the TLS handshake connection again and check if the error persists.

If disabling a particular service resolves the TLS handshake error, it indicates that the service was causing a conflict. In this case, you may want to investigate further or contact the service provider for guidance on how to configure the service to work properly without causing the error.

If needed, you can re-enable the disabled services by restarting your computer. This will allow the services to start running again.

Please note that when disabling services, exercise caution and only disable services that you are confident are non-essential and safe to disable temporarily. Disabling critical system services can potentially impact the functionality of your Windows 11 system. And these were methods to fix TLS errors on Windows 11.