Load Balancing - Frontend server 100% RAM and CPU consumption
Hi,
we installed Umbraco 7.2.1 following "Traditional load balancing " documentation.
1 backoffice, 3 frontend and 1 database server.
The loading balancig is managed by hardware and a task syncronizes all files (except App_Data) every 5 minutes.
From one year to now everything was fine, the other day editors published some news during high traffic and all three frontend servers had cpu and RAM at 100%. We had to turn off the portal.
The umbraco.config was 20MB.
These are the first error logs:
frontend
2016-07-29 10:49:25,049 [8] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 153] An error occurred with the scheduled publishing
System.Net.WebException: The operation has timed out
at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
at System.Net.WebClient.UploadString(Uri address, String method, String data)
at Umbraco.Web.Scheduling.ScheduledPublishing.Run()
2016-07-29 10:52:18,615 [8] ERROR Umbraco.Web.Scheduling.KeepAlive - [Thread 66] Error in ping
System.Net.WebException: The request was aborted: The operation has timed out.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at Umbraco.Web.Scheduling.KeepAlive.Start(ApplicationContext appContext, IUmbracoSettingsSection settings)
backoffice
2016-07-29 11:01:28,718 [7] ERROR Umbraco.Web.BatchedServerMessenger - [Thread 32] Error refreshing a node in the distributed list, URI attempted: invalid url (responseUri null)
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)
at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
at Umbraco.Web.BatchedServerMessenger.SendMessages(IEnumerable`1 messages)
thank you, I have changed FNCmode attribute value to single.
I don't know if this fix will be useful, because into all frontend servers's log I didn't never found "Application shutdown" log.
I hope anyway this will help server's performance.
Load Balancing - Frontend server 100% RAM and CPU consumption
Hi,
we installed Umbraco 7.2.1 following "Traditional load balancing " documentation.
1 backoffice, 3 frontend and 1 database server. The loading balancig is managed by hardware and a task syncronizes all files (except App_Data) every 5 minutes.
From one year to now everything was fine, the other day editors published some news during high traffic and all three frontend servers had cpu and RAM at 100%. We had to turn off the portal. The umbraco.config was 20MB.
These are the first error logs:
frontend 2016-07-29 10:49:25,049 [8] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 153] An error occurred with the scheduled publishing System.Net.WebException: The operation has timed out at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request) at System.Net.WebClient.UploadString(Uri address, String method, String data) at Umbraco.Web.Scheduling.ScheduledPublishing.Run()
2016-07-29 10:52:18,615 [8] ERROR Umbraco.Web.Scheduling.KeepAlive - [Thread 66] Error in ping System.Net.WebException: The request was aborted: The operation has timed out. at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request) at System.Net.WebClient.DownloadString(Uri address) at Umbraco.Web.Scheduling.KeepAlive.Start(ApplicationContext appContext, IUmbracoSettingsSection settings)
backoffice 2016-07-29 11:01:28,718 [7] ERROR Umbraco.Web.BatchedServerMessenger - [Thread 32] Error refreshing a node in the distributed list, URI attempted: invalid url (responseUri null) System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) at System.Net.Connection.ReadCallback(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Umbraco.Web.BatchedServerMessenger.SendMessages(IEnumerable`1 messages)
any idea?
Thank you very much
Marco
Pure guess: http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/
If you're running with an FCN Mode other than single then this could result in an application pool refresh.
Hi David,
thank you, I have changed FNCmode attribute value to single. I don't know if this fix will be useful, because into all frontend servers's log I didn't never found "Application shutdown" log.
I hope anyway this will help server's performance.
Thanks
Might application shutdown be INFO and you're storing greater severities?
FCN Mode is a shot in the dark.
is working on a reply...