Content refresh problem on child server(s) on load balanced environment
Hi, I wonder if anyone can help advise regarding our latest content refresh issue on our set-up (v7.9.0). We seem to have got past various ones to date.
We have a 3 server load balanced set-up with defined servers in UmbracoSettings.config and what we'll call server 1 as the master being the top one listed, Server 1 is also where the CMS is accessed and content is updated.
What we have been finding is that when content is updated, although it refreshes onto server 1 where the edit has taken place, it doesn't always on server 2 and 3. After previous updates; we find that the local Umbraco.config cache is updating on all 3 and it seems to be the in memory cache. When we go to the server and restart the application pool it does then appear.
Only recently users have been accessing the CMS via the IP address of server 1 due to a problem where our internal URL for editing has been re-routed through the load balancer. We have further improved the situation by adding the IP addresses to the those with the Culture & Hostnames for site in question. Our client has just had another example where it was updated on 1 and 2, but not 3 though.
It sounds like your using the legacy load balanced setup and not the flexible setup (introduced with 7.3.0) if the CMS is sat on Server 1? The CMS shouldn't be load balanced, so if it is on Server 1, which is load balanced it may be leading to some of the issues you're seeing?
Another thing to maybe check, which is on that link, is also your Examine indexes. If you're saying the content appears after the app pool has been recycled, that could be because the examine indexes have then been rebuilt.
Thanks for the response, I will check out the links and the examine indexes.
CMS users aren't going through the load balancer now they are using the IP address as a workaround. We had an URL that went to server 1 and was the one used for the CMS, but an error on the routing configuration meant that they was re-routed via the load balancer.
This caused a load of problems from 500 errors, corrupt indexes and Umbraco.config cache not updating.
We are now closer to the flexible set-up but are specifying the servers as the flexible master selection seemed to mean that the Umbraco.config was not updating on the master. We have since defined the servers and that has solved this.
We had one more instance of this but it resolved itself before I got to it. Just giving a little time to see if it occurs again and I will confirm answer :)
My index settings largely seem to be similar to those described in the link. I've noticed a couple of things today regarding this....
1) When I rebuild the index on our CMS server WEB01, it is not refreshing to WEB02 and WEB03
2) On many occasions where a user publishes some content we get a Lucene error on the CMS server just after...
An unhandled exception occurred and the process was terminated.
Application ID: DefaultDomain
Process ID: 3736
Exception: System.Runtime.Serialization.SerializationException
Message: Unable to find assembly 'Lucene.Net, Version=2.9.4.1, Culture=neutral, PublicKeyToken=85089178b9ac3181'.
StackTrace: at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm)
at System.AppDomain.Deserialize(Byte[] blob)
at System.AppDomain.UnmarshalObject(Byte[] blob)
3) We do sometimes get an IO examine error on the slave server(s) a bit after a publish..
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/2/ROOT
Process ID: 5580
Exception: System.IO.FileNotFoundException
Message: Could not find file 'E:\App_Data\TEMP\ExamineIndexes\CGMVPSPWEB02\External\Index_46_1.del'.
StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Lucene.Net.Store.SimpleFSDirectory.OpenInput(String name, Int32 bufferSize)
at Lucene.Net.Util.BitVector..ctor(Directory d, String name)
at Lucene.Net.Index.SegmentReader.LoadDeletedDocs()
at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, SegmentInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDivisor)
at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores, Int32 readBufferSize, Int32 termsIndexDivisor)
at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores)
at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos)
at Lucene.Net.Index.IndexWriter.ApplyDeletes()
at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary2 commitUserData, Boolean internal_Renamed)
at Lucene.Net.Index.IndexWriter.Commit(IDictionary2 commitUserData)
at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1710
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()
4) I have noticed an "APP_DATA" permissions issue on local where it hasn't inherited properly. I'm hoping this is related to it but not sure it will help much
We've had another couple of instances of the issue through and on the last I tried rebuilding the indexes but it did not refresh still.
What we have noticed is that there is some user connections through the load balancer at times. These aren't directly the ones that experienced the problems though or even dead around them and are in a minority. A thought was maybe causing issues with locked index files as that seemed to be the error reported again
We have similar problems.
We have 3 servers and umbraco backooffice website on server1.
Server1: website1 and backoffice site
Server2: website2
Server3: website3
Only websites are exposed through loadbalancer and backoffice from local domain.
They all share their own code base except app_data, can this create havoc on our website?
Will the above We are experiencing similar issues as above:
1
Exception: Lucene.Net.Index.MergePolicy+MergeException
Message: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown.
2
Description: The process was terminated due to an unhandled exception. Exception Info: System.IndexOutOfRangeException at Lucene.Net.Index.IndexWriter.HandleMergeException(System.Exception, OneMerge) at Lucene.Net.Index.IndexWriter.Merge(OneMerge) at Lucene.Net.Index.ConcurrentMergeScheduler+MergeThread.Run()
Content refresh problem on child server(s) on load balanced environment
Hi, I wonder if anyone can help advise regarding our latest content refresh issue on our set-up (v7.9.0). We seem to have got past various ones to date.
We have a 3 server load balanced set-up with defined servers in UmbracoSettings.config and what we'll call server 1 as the master being the top one listed, Server 1 is also where the CMS is accessed and content is updated.
What we have been finding is that when content is updated, although it refreshes onto server 1 where the edit has taken place, it doesn't always on server 2 and 3. After previous updates; we find that the local Umbraco.config cache is updating on all 3 and it seems to be the in memory cache. When we go to the server and restart the application pool it does then appear.
Only recently users have been accessing the CMS via the IP address of server 1 due to a problem where our internal URL for editing has been re-routed through the load balancer. We have further improved the situation by adding the IP addresses to the those with the Culture & Hostnames for site in question. Our client has just had another example where it was updated on 1 and 2, but not 3 though.
Any advice/pointers would be much appreciated.
It sounds like your using the legacy load balanced setup and not the flexible setup (introduced with 7.3.0) if the CMS is sat on Server 1? The CMS shouldn't be load balanced, so if it is on Server 1, which is load balanced it may be leading to some of the issues you're seeing?
https://our.umbraco.com/documentation/Getting-Started/Setup/Server-Setup/load-balancing/flexible
Another thing to maybe check, which is on that link, is also your Examine indexes. If you're saying the content appears after the app pool has been recycled, that could be because the examine indexes have then been rebuilt.
Maybe a few other things to look at on this thread too: https://our.umbraco.com/forum/extending-umbraco-and-using-the-api/86822-distributed-cache-is-not-updated-error-in-load-balanced-environment
Thanks for the response, I will check out the links and the examine indexes.
CMS users aren't going through the load balancer now they are using the IP address as a workaround. We had an URL that went to server 1 and was the one used for the CMS, but an error on the routing configuration meant that they was re-routed via the load balancer.
This caused a load of problems from 500 errors, corrupt indexes and Umbraco.config cache not updating.
We are now closer to the flexible set-up but are specifying the servers as the flexible master selection seemed to mean that the Umbraco.config was not updating on the master. We have since defined the servers and that has solved this.
We had one more instance of this but it resolved itself before I got to it. Just giving a little time to see if it occurs again and I will confirm answer :)
My index settings largely seem to be similar to those described in the link. I've noticed a couple of things today regarding this....
1) When I rebuild the index on our CMS server WEB01, it is not refreshing to WEB02 and WEB03
2) On many occasions where a user publishes some content we get a Lucene error on the CMS server just after...
An unhandled exception occurred and the process was terminated. Application ID: DefaultDomain Process ID: 3736 Exception: System.Runtime.Serialization.SerializationException Message: Unable to find assembly 'Lucene.Net, Version=2.9.4.1, Culture=neutral, PublicKeyToken=85089178b9ac3181'. StackTrace: at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly() at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name) at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run() at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm) at System.AppDomain.Deserialize(Byte[] blob) at System.AppDomain.UnmarshalObject(Byte[] blob)
3) We do sometimes get an IO examine error on the slave server(s) a bit after a publish..
An unhandled exception occurred and the process was terminated. Application ID: /LM/W3SVC/2/ROOT Process ID: 5580 Exception: System.IO.FileNotFoundException Message: Could not find file 'E:\App_Data\TEMP\ExamineIndexes\CGMVPSPWEB02\External\Index_46_1.del'. StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at Lucene.Net.Store.SimpleFSDirectory.OpenInput(String name, Int32 bufferSize) at Lucene.Net.Util.BitVector..ctor(Directory d, String name) at Lucene.Net.Index.SegmentReader.LoadDeletedDocs() at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, SegmentInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDivisor) at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores, Int32 readBufferSize, Int32 termsIndexDivisor) at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores) at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos) at Lucene.Net.Index.IndexWriter.ApplyDeletes() at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes) at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes) at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary
2 commitUserData, Boolean internal_Renamed) at Lucene.Net.Index.IndexWriter.Commit(IDictionary2 commitUserData) at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1710 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.TimerQueueTimer.CallCallback() at System.Threading.TimerQueueTimer.Fire() at System.Threading.TimerQueue.FireNextTimers()4) I have noticed an "APP_DATA" permissions issue on local where it hasn't inherited properly. I'm hoping this is related to it but not sure it will help much
We've had another couple of instances of the issue through and on the last I tried rebuilding the indexes but it did not refresh still.
What we have noticed is that there is some user connections through the load balancer at times. These aren't directly the ones that experienced the problems though or even dead around them and are in a minority. A thought was maybe causing issues with locked index files as that seemed to be the error reported again
We have similar problems. We have 3 servers and umbraco backooffice website on server1.
Only websites are exposed through loadbalancer and backoffice from local domain.
They all share their own code base except app_data, can this create havoc on our website?
Will the above We are experiencing similar issues as above:
1
2
and then the application would restart.
is working on a reply...