I have a very strange and intermittent issue, I've had several people looking into it and all have draw blanks so far, so I'm hoping this wonderful Umbraco community can help.
Every so often (sometimes once a month, sometimes every day) we get complete outages on a few of our Windows Server 2016 boxes, running several Umbraco installs, using SNI to map SSLs, all SSLs installed on webservers.
The web servers sit behind an HAProxy server which routes traffic to the correct server with passthrough SSL.
During the outages IIS logs are blank, and our front end monitoring shows a range of errors: Server protocol violation, SSL handshake failed, HTTP send failure.
Sometimes nothing but waiting will bring the sites back. Recycled app pools, IIS reset, rebooted server.
Windows logs don't suggest anything obvious. We've stopped Windows Defender, tightened protocols to TLS 1.0, 1.1,1.2 - tightened ciphers to IIS Crypto best practice. Also tried loosening the secuirty on these.
Stopped WinHttpAutoProxySvc as I saw some evidence it can interfere.
Added useUnsafeHeaderParsing="true" against my best instincts!
Any further suggestions would be very much welcomed as I've run out of ideas and have clients going mad.
All sites on the server are perfect 95+% of the time, then all traffic seems to be blocked for a varying period.
I can see some logs for Windows Update failing but the times don't correspond exactly - we have Windows updates disabled so we can control roll out so I would expect the auto to fail - has anyone know this to interfere with web traffic?
We could still do a standard ‘HTTP get’ from port 80 on the the server, which suggests to me that it’s not a networking issue but on the application stack somewhere on IIS
intermittent outages - SSL handshake errors - Server protocol errors
I have a very strange and intermittent issue, I've had several people looking into it and all have draw blanks so far, so I'm hoping this wonderful Umbraco community can help.
Every so often (sometimes once a month, sometimes every day) we get complete outages on a few of our Windows Server 2016 boxes, running several Umbraco installs, using SNI to map SSLs, all SSLs installed on webservers.
The web servers sit behind an HAProxy server which routes traffic to the correct server with passthrough SSL.
During the outages IIS logs are blank, and our front end monitoring shows a range of errors: Server protocol violation, SSL handshake failed, HTTP send failure.
Sometimes nothing but waiting will bring the sites back. Recycled app pools, IIS reset, rebooted server.
Windows logs don't suggest anything obvious. We've stopped Windows Defender, tightened protocols to TLS 1.0, 1.1,1.2 - tightened ciphers to IIS Crypto best practice. Also tried loosening the secuirty on these.
Stopped WinHttpAutoProxySvc as I saw some evidence it can interfere.
Added useUnsafeHeaderParsing="true" against my best instincts!
Any further suggestions would be very much welcomed as I've run out of ideas and have clients going mad.
Thanks all
Could be a network issue. Can you still ping the server?
I will double check next time the issue crops up - hang tight
Make sure you can ping the server while it's still up and it doesn't get blocked by a firewall.
Whilst the issue is happening, I am unable to ping the server from the proxy that routes traffic to it. Would suggest a NIC/VMWare issue?
Windows updates? Backups? Application pool trying to run 32-bit apps? Does the IIS log show anything prior to the outage?
All sites on the server are perfect 95+% of the time, then all traffic seems to be blocked for a varying period.
I can see some logs for Windows Update failing but the times don't correspond exactly - we have Windows updates disabled so we can control roll out so I would expect the auto to fail - has anyone know this to interfere with web traffic?
We could still do a standard ‘HTTP get’ from port 80 on the the server, which suggests to me that it’s not a networking issue but on the application stack somewhere on IIS
is working on a reply...