Hi.
We are using Umbraco 7.2.4 with load-balancing for our high load website.
We have 3 servers for website, and shared database on separate server.
And we have a problems when we publish or republish content in umbraco on the main server. Publishing causes hang, freeze, and website becomes unavailable.
This behavior is unacceptable for us.
If take a look at UmbracoTraceLog, log contains lines like this:
DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 36] ReIndexNode with type: content, Provider=InternalIndexer, NodeId=221605
DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 36] AddSingleNodeToIndex with type: content, Provider=InternalIndexer, NodeId=221605
DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 35] Index created for node 221605, Provider=InternalIndexer, NodeId=221605
How can we reduce the load on website when reindex starts, or maybe can we delay this operation?
I have found that publish operations can cause pages to take 5 seconds or so to load. However, it seems as if that's only the case if I open the website in the same browser that has the Umbraco tab open that is publishing. If I open the site in a different browser, the website responds fine. My guess is ASP.NET is preventing multiple requests from the same session, or something along those lines.
Given the above, my theory is that the site appears to be slow to the content editor, but is actually responsive for everybody else.
Thanks for your response, Nicholas.
No, site is unavailable for everybody. Users cant open the site in this time. Reindex may take 5 or 15 minutes. Sometimes it takes longer time.
Basically, publish operations can cause an application pool restart, which can cause your website to be slow for a few minutes.
That issue has info on how to check if you are affected. The fix is to do Windows Update. Alternatively, you could just install this hotfix in particular: https://support.microsoft.com/en-us/kb/3052480
Thanks for suggestion, Nicholas.
But we dont have installed KB3000850 or KB3007507, that can cause this problem. Furthermore, our umbraco version is 7.2.4, not 7.2.1, 6.2.5, 7.2.2.
And also we have problems with loading images throw code. I use this code:
byte []mediaBytes=GetImageBytes();
var newImage = mediaService.CreateMedia(fileName, parentId, "Image");
MemoryStream strm = new MemoryStream(mediaBytes);
newImage.SetValue("umbracoFile", fileName, strm);
mediaService.Save(newImage);
After execution of this code we have following lines in log on main server:
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] ReIndexNode with type: media, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] AddSingleNodeToIndex with type: media, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] ReIndexNode with type: media, Provider=ExternalIndexer, NodeId=429675
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] AddSingleNodeToIndex with type: media, Provider=ExternalIndexer, NodeId=429675
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 84] Index created for node 429675, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 81] Index created for node 429675, Provider=ExternalIndexer, NodeId=429675
2015-10-26 11:02:30,539 [45] DEBUG Umbraco.Web.UmbracoModule - [Thread 84] Begin request: http://srv-01.tdec.local/umbraco/backoffice/UmbracoApi/Authentication/GetRemainingTimeoutSeconds.
2015-10-26 11:02:31,023 [45] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 71] Performing distributed call for refresher Umbraco.Web.Cache.MediaCacheRefresher, message type: RefreshByJson, servers: http://srv-01.tdec.local/umbraco/webservices/cacheRefresher.asmx;http://serv-2.tdec.local/umbraco/webservices/cacheRefresher.asmx;http://serv-3.tdec.local/umbraco/webservices/cacheRefresher.asmx, ids: , json: [{"Path":"-1,1063,429676","Id":429676,"Operation":0}]
2015-10-26 11:02:31,023 [45] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 71] Invoking refresher Umbraco.Web.Cache.MediaCacheRefresher on single server instance, message type RefreshByJson
And lines in log on child (frontend servers):
[Thread 57] ReIndexNode with type: media, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] AddSingleNodeToIndex with type: media, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] ReIndexNode with type: media, Provider=ExternalIndexer, NodeId=429675
2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] AddSingleNodeToIndex with type: media, Provider=ExternalIndexer, NodeId=429675
2015-10-26 11:02:44,434 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 61] Index created for node 429675, Provider=InternalIndexer, NodeId=429675
2015-10-26 11:02:44,512 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 37] Index created for node 429675, Provider=ExternalIndexer, NodeId=429675
When MediaCacheRefresher is running, we have database timeout errors on pages of our site. We use dedicated server with MS SQL 2012.
Timeout errors in log on frontend servers:
[16] WARN NHibernate.Util.ADOExceptionReporter - System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
[16] ERROR NHibernate.Util.ADOExceptionReporter - The wait operation timed out
[16] WARN NHibernate.Util.ADOExceptionReporter - System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
at System.Data.SqlClient.SqlDataReader.Read()
at NHibernate.Driver.NHybridDataReader.Read()
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer)
ClientConnectionId:d6d9e7a2-cef2-4109-88a1-f9505da108b4
[16] ERROR NHibernate.Util.ADOExceptionReporter - Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Can you suggest something how to solve this problems with MediaCacheRefresher?
Content reindex hangs website
Hi. We are using Umbraco 7.2.4 with load-balancing for our high load website. We have 3 servers for website, and shared database on separate server.
And we have a problems when we publish or republish content in umbraco on the main server. Publishing causes hang, freeze, and website becomes unavailable. This behavior is unacceptable for us. If take a look at UmbracoTraceLog, log contains lines like this:
DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 36] ReIndexNode with type: content, Provider=InternalIndexer, NodeId=221605 DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 36] AddSingleNodeToIndex with type: content, Provider=InternalIndexer, NodeId=221605 DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 35] Index created for node 221605, Provider=InternalIndexer, NodeId=221605
How can we reduce the load on website when reindex starts, or maybe can we delay this operation?
How long does it cause a hang of the website?
I have found that publish operations can cause pages to take 5 seconds or so to load. However, it seems as if that's only the case if I open the website in the same browser that has the Umbraco tab open that is publishing. If I open the site in a different browser, the website responds fine. My guess is ASP.NET is preventing multiple requests from the same session, or something along those lines.
Given the above, my theory is that the site appears to be slow to the content editor, but is actually responsive for everybody else.
Thanks for your response, Nicholas. No, site is unavailable for everybody. Users cant open the site in this time. Reindex may take 5 or 15 minutes. Sometimes it takes longer time.
That's not normal and sounds like this bug: http://issues.umbraco.org/issue/U4-6338
Basically, publish operations can cause an application pool restart, which can cause your website to be slow for a few minutes.
That issue has info on how to check if you are affected. The fix is to do Windows Update. Alternatively, you could just install this hotfix in particular: https://support.microsoft.com/en-us/kb/3052480
Thanks for suggestion, Nicholas. But we dont have installed KB3000850 or KB3007507, that can cause this problem. Furthermore, our umbraco version is 7.2.4, not 7.2.1, 6.2.5, 7.2.2. And also we have problems with loading images throw code. I use this code:
After execution of this code we have following lines in log on main server: 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] ReIndexNode with type: media, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] AddSingleNodeToIndex with type: media, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] ReIndexNode with type: media, Provider=ExternalIndexer, NodeId=429675 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 71] AddSingleNodeToIndex with type: media, Provider=ExternalIndexer, NodeId=429675 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 84] Index created for node 429675, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:29,367 [45] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 81] Index created for node 429675, Provider=ExternalIndexer, NodeId=429675 2015-10-26 11:02:30,539 [45] DEBUG Umbraco.Web.UmbracoModule - [Thread 84] Begin request: http://srv-01.tdec.local/umbraco/backoffice/UmbracoApi/Authentication/GetRemainingTimeoutSeconds. 2015-10-26 11:02:31,023 [45] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 71] Performing distributed call for refresher Umbraco.Web.Cache.MediaCacheRefresher, message type: RefreshByJson, servers: http://srv-01.tdec.local/umbraco/webservices/cacheRefresher.asmx;http://serv-2.tdec.local/umbraco/webservices/cacheRefresher.asmx;http://serv-3.tdec.local/umbraco/webservices/cacheRefresher.asmx, ids: , json: [{"Path":"-1,1063,429676","Id":429676,"Operation":0}] 2015-10-26 11:02:31,023 [45] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 71] Invoking refresher Umbraco.Web.Cache.MediaCacheRefresher on single server instance, message type RefreshByJson
And lines in log on child (frontend servers): [Thread 57] ReIndexNode with type: media, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] AddSingleNodeToIndex with type: media, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] ReIndexNode with type: media, Provider=ExternalIndexer, NodeId=429675 2015-10-26 11:02:44,293 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 57] AddSingleNodeToIndex with type: media, Provider=ExternalIndexer, NodeId=429675 2015-10-26 11:02:44,434 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 61] Index created for node 429675, Provider=InternalIndexer, NodeId=429675 2015-10-26 11:02:44,512 [16] DEBUG UmbracoExamine.DataServices.UmbracoLogService - [Thread 37] Index created for node 429675, Provider=ExternalIndexer, NodeId=429675
When MediaCacheRefresher is running, we have database timeout errors on pages of our site. We use dedicated server with MS SQL 2012.
Timeout errors in log on frontend servers: [16] WARN NHibernate.Util.ADOExceptionReporter - System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out [16] ERROR NHibernate.Util.ADOExceptionReporter - The wait operation timed out [16] WARN NHibernate.Util.ADOExceptionReporter - System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more) at System.Data.SqlClient.SqlDataReader.Read() at NHibernate.Driver.NHybridDataReader.Read() at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer) at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer) at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer) ClientConnectionId:d6d9e7a2-cef2-4109-88a1-f9505da108b4 [16] ERROR NHibernate.Util.ADOExceptionReporter - Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Can you suggest something how to solve this problems with MediaCacheRefresher?
is working on a reply...