You will not need an additional server (that's mostly for Azure WebApps only) as long as you can route Umbraco traffic to a single server bypassing the load balancer by using a specific subdomain, e.g. admin.mywebsite.com. You are likely doing this already?
As you are already setup for traditional load balancing, once you've upgraded to v7.3, all you should need to do is disable the distributed calls in umbracoSettings.config and now you are using the flexible load balancing and everything should work as it did before.
Thanks for this post Jeavon. Just upgraded a project from Umbraco 7.2.8 to 7.5.6. So now I would also like to start using flexible load balancing instead of traditional. Time to disable the distributed calls in umbracoSettings.config :-).
We switched from traditional load balancing to flexible load balancing, but somehow the Examine indexes on the slave servers seem outdated. When I check the log of the slave servers I also see this error a lot:
2017-01-11 20:00:10,076 [14] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 50, instruction count: 2). Instruction is being skipped/ignored
System.Threading.ThreadAbortException: Thread was being aborted.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
at Umbraco.Core.Models.PropertyGroupCollection.DeepClone()
at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
at Umbraco.Core.Models.ContentTypeCompositionBase.<DeepClone>b__29(IContentTypeComposition x)
at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.<GetAll>b__13(TEntity x)
at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.Get(TId id, Func`2 getFromRepo)
at Umbraco.Core.Persistence.Repositories.RepositoryBase`2.Get(TId id)
at Umbraco.Core.Persistence.Repositories.ContentRepository.ProcessQuery(Sql sql, Boolean withCache)
at Umbraco.Core.Persistence.Repositories.VersionableRepositoryBase`2.GetPagedResultsByQuery[TDto,TContentBase](IQuery`1 query, Int64 pageIndex, Int32 pageSize, Int64& totalRecords, Tuple`2 nodeIdSelect, Func`2 processQuery, String orderBy, Direction orderDirection, Boolean orderBySystemField, Func`1 defaultFilter)
at Umbraco.Core.Persistence.Repositories.ContentRepository.GetPagedResultsByQuery(IQuery`1 query, Int64 pageIndex, Int32 pageSize, Int64& totalRecords, String orderBy, Direction orderDirection, Boolean orderBySystemField, IQuery`1 filter)
at Umbraco.Core.Services.ContentService.GetPagedDescendants(Int32 id, Int64 pageIndex, Int32 pageSize, Int64& totalChildren, String orderBy, Direction orderDirection, Boolean orderBySystemField, IQuery`1 filter)
at UmbracoExamine.UmbracoContentIndexer.PerformIndexAll(String type)
at UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild()
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)
Does anyone know what this means and how we can fix it?
I already tried truncating the instruction table, then cold boot the slaves. I thought that worked, but later the errors appeared again. I'm on 7.5.6 so I thought that the DISTRIBUTED CACHE IS NOT UPDATED was fixed in there. Anything else I can do?
well it is 'fixed', but if there's other unknown problems, then i guess those are still unknown.
do you have any custom DatabaseServerMessengerOptions configured or anything else I should know about? ... perhaps how to replicate, what leads to this problem, any other details you wish to share?
There is a custom web api which calls a cache refresher on all servers. This cache refresher rebuilds the Examine indexes on all servers like this:
public void RefreshAll()
{
var cacheManager = new ServiceLocator().GetCacheManager();
cacheManager.ExternalRuntimeCache.Clear();
cacheManager.RuntimeCache.Clear();
cacheManager.ClearOutputCache();
ExamineManager.Instance.IndexProviderCollection
.Where(p => p.Name == "ArticleIndexer" || p.Name == "TrainingIndexer")
.ToList()
.ForEach(index => index.RebuildIndex());
}
This needs to be done because the content in Examine is not just Umbraco content, but also from a external 3rd party XML file. So the save and publish PageCacheRefresher isn't enough.
We get the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message when we call the web api.
There are no custom DatabaseServerMessengerOptions configured.
Is it possible that the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message isn't the real error? Maybe because we do some funky things with Examine some goes wrong with that?
Maybe the Examine indexes in Azure blob storage could also help here ;-).
You will have issues with this I guarantee especially if this is called often.
The problem is that it's taking too long to process the instructions, a single instruction shouldn't do something like rebuild all indexes. The fix we put in is to ensure that only a certain amount of instructions are processed at once, the instructions we normally process are things like updating in memory cache or updating an entry in the xml cache file or re-index a single node.
If you want to hack a solution together you could run the logic on a background thread but this will just be masking another problem because you shouldn't be rebuilding indexes unless they become corrupt.
After looking a bit better at the code with another developer it seems that the external data isn't stored in Examine. I thought this was the case because some content only appeared after updating the external XML and rebuilding the indexes, but apparently the content showed because some cache was cleared. It was late last night ;-).
Since the Examine indexes only store Umbraco content there is no reason to rebuild the indexes with the clear cache web api. I actually don't know why the previous developer added the reindex code, but simply removing the reindex code should be enough.
Will give another update when we deployed the changes.
that is precisely why that is written as an anti-pattern, i see this far too often, some developers for whatever (most likely old legacy reasons) think that they need to rebuild indexes all the time, we've seen this on every request in some cases (wat!? :)
glad you got it sorted, site should run a lot quicker now!
Flexible load balancing vs traditional load balancing
Are there any articles available comparing these 2 methods and explaining the advantages of flexible load balancing from a performance point of view?
We currently load balance the traditional way:
2 x IIS, 1 x DB . The admin site is on one of the IIS servers.
If we move to flexible load balancing we will need an additional IIS server as the master.
I need therefore to justify this additional cost to my client and wonder if it is worth the change.
Thanks
Dan
Hi Dan,
You will not need an additional server (that's mostly for Azure WebApps only) as long as you can route Umbraco traffic to a single server bypassing the load balancer by using a specific subdomain, e.g. admin.mywebsite.com. You are likely doing this already?
The documentation you need is Option 2 here
As you are already setup for traditional load balancing, once you've upgraded to v7.3, all you should need to do is disable the distributed calls in umbracoSettings.config and now you are using the flexible load balancing and everything should work as it did before.
Jeavon
Thanks for this post Jeavon. Just upgraded a project from Umbraco 7.2.8 to 7.5.6. So now I would also like to start using flexible load balancing instead of traditional. Time to disable the distributed calls in umbracoSettings.config :-).
Jeroen
We switched from traditional load balancing to flexible load balancing, but somehow the Examine indexes on the slave servers seem outdated. When I check the log of the slave servers I also see this error a lot:
Does anyone know what this means and how we can fix it?
Jeroen
I fixed it by reading all the comments from these issues:
http://issues.umbraco.org/issue/U4-7428
http://issues.umbraco.org/issue/U4-7673
Jeroen
Spoke too soon. The same error is back again.
Try truncating your instruction table, then cold boot your slaves: http://issues.umbraco.org/issue/U4-7673#comment=67-32174 you might have instruction blobs left over in there from previous versions that are massive.
Hi Shannon,
I already tried truncating the instruction table, then cold boot the slaves. I thought that worked, but later the errors appeared again. I'm on 7.5.6 so I thought that the DISTRIBUTED CACHE IS NOT UPDATED was fixed in there. Anything else I can do?
Jeroen
well it is 'fixed', but if there's other unknown problems, then i guess those are still unknown.
do you have any custom
DatabaseServerMessengerOptionsconfigured or anything else I should know about? ... perhaps how to replicate, what leads to this problem, any other details you wish to share?You've totally hijacked this thread btw ;)
Sorry I should provide more information. And yes this is going a bit off topic ;-).
This is the same inherited project as the Autofac issues when rebuilding Examine indexes topic.
There is a custom web api which calls a cache refresher on all servers. This cache refresher rebuilds the Examine indexes on all servers like this:
This needs to be done because the content in Examine is not just Umbraco content, but also from a external 3rd party XML file. So the save and publish PageCacheRefresher isn't enough.
We get the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message when we call the web api.
There are no custom DatabaseServerMessengerOptions configured.
Is it possible that the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message isn't the real error? Maybe because we do some funky things with Examine some goes wrong with that?
Maybe the Examine indexes in Azure blob storage could also help here ;-).
Jeroen
This sounds like a very bad idea. Rebuilding indexes is just bad: https://our.umbraco.org/Documentation/Reference/Common-Pitfalls/#rebuilding-indexes
You will have issues with this I guarantee especially if this is called often.
The problem is that it's taking too long to process the instructions, a single instruction shouldn't do something like rebuild all indexes. The fix we put in is to ensure that only a certain amount of instructions are processed at once, the instructions we normally process are things like updating in memory cache or updating an entry in the xml cache file or re-index a single node.
If you want to hack a solution together you could run the logic on a background thread but this will just be masking another problem because you shouldn't be rebuilding indexes unless they become corrupt.
I think you'll need to re-think this external xml file and how it integrates with content and indexes.
Hi Shannon,
Thank you for answering this. I agree that it's a bad idea. It's an inherited project and we're adding improvements on the go.
I think I'll go with the hack solution for now because we have this problem on a live environment so we need to fix it asap.
We already have some ideas on how to improve the external XML integration, but that's something for the long run.
Jeroen
Ok, good luck with that ... don't say i didn't warn you ;)
After looking a bit better at the code with another developer it seems that the external data isn't stored in Examine. I thought this was the case because some content only appeared after updating the external XML and rebuilding the indexes, but apparently the content showed because some cache was cleared. It was late last night ;-).
Since the Examine indexes only store Umbraco content there is no reason to rebuild the indexes with the clear cache web api. I actually don't know why the previous developer added the reindex code, but simply removing the reindex code should be enough.
Will give another update when we deployed the changes.
Jeroen
Changes have been deployed to the live environment. Since we no longer reindex the Examine indexes as a instruction everything works like it should.
Thanks Shannon for helping.
Jeroen
that is precisely why that is written as an anti-pattern, i see this far too often, some developers for whatever (most likely old legacy reasons) think that they need to rebuild indexes all the time, we've seen this on every request in some cases (wat!? :)
glad you got it sorted, site should run a lot quicker now!
is working on a reply...