Umbraco load balanced slave server DISTRIBUTED CACHE NOT UPDATED
Hi community,
Wonder if anyone can give me a steer on this error: Scenario is two web servers in a load balanced environment, with one master CMS server separate and outside of the LB environment.
There is no shared file system, and master/slave servers are specified explicitly using the example given here:
Our master CMS server is updating the umbracoCacheInstruction table correctly. However. both load balanced web servers are reporting this error:
2017-08-26 12:29:23,186 [P6940/D6/T53] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 7192, instruction count: 1000). Instruction is being skipped/ignored
The thread has been aborted, because the request has timed out.
System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken)
at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Umbraco.Core.AsyncLock.Lock()
at umbraco.SafeXmlReaderWriter.Get(IScopeProviderInternal scopeProvider, AsyncLock xmlLock, XmlDocument xml, Action1 refresh, Action2 apply, Boolean writer)
at umbraco.content.GetSafeXmlWriter()
at umbraco.content.UpdateDocumentCache(Document d)
at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id)
at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable1 instructions, HashSet1 processed)
at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection1 instructionBatch, CacheInstructionDto dto, HashSet1 processed, Int32& lastId)
From the error log, it looks like the semaphore expired while trying to get access to the cache file - but as the cachefile is on the local disk and has all the correct read/write permissions, I'm confused.
How did you fix this problem please without a cold boot?
The only fix that works is the following as far as I see;
- Stop the Front-End App Service
- Delete everything under AppData folder
- Delete umbracoCachedInstruction table data
- Start the FE App Service and re-create the AppData items
automatically
We're got one "admin" server with 2 front end servers, we're on 7.6.4 and seeing plenty of the errors. Admin server publishes content, but front end machines do not seem to get updated unless "forced" but the usual ways of deleting umbraco.config, recycle app pool and that's not a fix at all.
Server's are only set to robocopy media from Admin to the front end servers and that's it, nothing else is copied.
Umbraco load balanced slave server DISTRIBUTED CACHE NOT UPDATED
Hi community,
Wonder if anyone can give me a steer on this error: Scenario is two web servers in a load balanced environment, with one master CMS server separate and outside of the LB environment.
There is no shared file system, and master/slave servers are specified explicitly using the example given here:
Our master CMS server is updating the umbracoCacheInstruction table correctly. However. both load balanced web servers are reporting this error:
From the error log, it looks like the semaphore expired while trying to get access to the cache file - but as the cachefile is on the local disk and has all the correct read/write permissions, I'm confused.
This is on Umbraco 7.6.3
Can anyone help?
Many thanks
Simon
Found the issue - was due to the fact the slaves couldn't "find" themselves.
Once that was sorted everything works beautifully!
How did you fix this problem please without a cold boot?
The only fix that works is the following as far as I see; - Stop the Front-End App Service - Delete everything under AppData folder - Delete umbracoCachedInstruction table data - Start the FE App Service and re-create the AppData items automatically
@Simon
Any chance you could share what's your fix?
We're got one "admin" server with 2 front end servers, we're on 7.6.4 and seeing plenty of the errors. Admin server publishes content, but front end machines do not seem to get updated unless "forced" but the usual ways of deleting umbraco.config, recycle app pool and that's not a fix at all.
Server's are only set to robocopy media from Admin to the front end servers and that's it, nothing else is copied.
Hi Simon, how did you solve this issue?
is working on a reply...