Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Dan Evans 599 posts 949 karma points
    Nov 18, 2015 @ 11:18
    Dan Evans
    0

    Flexible load balancing vs traditional load balancing

    Are there any articles available comparing these 2 methods and explaining the advantages of flexible load balancing from a performance point of view?

    We currently load balance the traditional way:

    2 x IIS, 1 x DB . The admin site is on one of the IIS servers.

    If we move to flexible load balancing we will need an additional IIS server as the master.

    I need therefore to justify this additional cost to my client and wonder if it is worth the change.

    Thanks

    Dan

  • Jeavon Leopold 2999 posts 13138 karma points MVP 6x admin c-trib
    Nov 18, 2015 @ 11:41
    Jeavon Leopold
    2

    Hi Dan,

    You will not need an additional server (that's mostly for Azure WebApps only) as long as you can route Umbraco traffic to a single server bypassing the load balancer by using a specific subdomain, e.g. admin.mywebsite.com. You are likely doing this already?

    The documentation you need is Option 2 here

    As you are already setup for traditional load balancing, once you've upgraded to v7.3, all you should need to do is disable the distributed calls in umbracoSettings.config and now you are using the flexible load balancing and everything should work as it did before.

    Jeavon

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 09, 2017 @ 13:09
    Jeroen Breuer
    1

    Thanks for this post Jeavon. Just upgraded a project from Umbraco 7.2.8 to 7.5.6. So now I would also like to start using flexible load balancing instead of traditional. Time to disable the distributed calls in umbracoSettings.config :-).

    Jeroen

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 11, 2017 @ 19:29
    Jeroen Breuer
    0

    We switched from traditional load balancing to flexible load balancing, but somehow the Examine indexes on the slave servers seem outdated. When I check the log of the slave servers I also see this error a lot:

    2017-01-11 20:00:10,076 [14] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 50, instruction count: 2). Instruction is being skipped/ignored
    System.Threading.ThreadAbortException: Thread was being aborted.
       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
       at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
       at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
       at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
       at Umbraco.Core.Models.PropertyGroupCollection.DeepClone()
       at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
       at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
       at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
       at Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(IDeepCloneable input, IDeepCloneable output)
       at Umbraco.Core.Models.EntityBase.Entity.DeepClone()
       at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
       at Umbraco.Core.Models.ContentTypeCompositionBase.<DeepClone>b__29(IContentTypeComposition x)
       at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
       at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
       at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
       at Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()
       at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.<GetAll>b__13(TEntity x)
       at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
       at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
       at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.Get(TId id, Func`2 getFromRepo)
       at Umbraco.Core.Persistence.Repositories.RepositoryBase`2.Get(TId id)
       at Umbraco.Core.Persistence.Repositories.ContentRepository.ProcessQuery(Sql sql, Boolean withCache)
       at Umbraco.Core.Persistence.Repositories.VersionableRepositoryBase`2.GetPagedResultsByQuery[TDto,TContentBase](IQuery`1 query, Int64 pageIndex, Int32 pageSize, Int64& totalRecords, Tuple`2 nodeIdSelect, Func`2 processQuery, String orderBy, Direction orderDirection, Boolean orderBySystemField, Func`1 defaultFilter)
       at Umbraco.Core.Persistence.Repositories.ContentRepository.GetPagedResultsByQuery(IQuery`1 query, Int64 pageIndex, Int32 pageSize, Int64& totalRecords, String orderBy, Direction orderDirection, Boolean orderBySystemField, IQuery`1 filter)
       at Umbraco.Core.Services.ContentService.GetPagedDescendants(Int32 id, Int64 pageIndex, Int32 pageSize, Int64& totalChildren, String orderBy, Direction orderDirection, Boolean orderBySystemField, IQuery`1 filter)
       at UmbracoExamine.UmbracoContentIndexer.PerformIndexAll(String type)
       at UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild()
       at System.Collections.Generic.List`1.ForEach(Action`1 action)
       at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
       at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)
    

    Does anyone know what this means and how we can fix it?

    Jeroen

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 11, 2017 @ 20:25
    Jeroen Breuer
    0

    I fixed it by reading all the comments from these issues:

    http://issues.umbraco.org/issue/U4-7428

    http://issues.umbraco.org/issue/U4-7673

    Jeroen

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 11, 2017 @ 20:51
    Jeroen Breuer
    0

    Spoke too soon. The same error is back again.

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 00:16
    Shannon Deminick
    0

    Try truncating your instruction table, then cold boot your slaves: http://issues.umbraco.org/issue/U4-7673#comment=67-32174 you might have instruction blobs left over in there from previous versions that are massive.

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 12, 2017 @ 06:21
    Jeroen Breuer
    0

    Hi Shannon,

    I already tried truncating the instruction table, then cold boot the slaves. I thought that worked, but later the errors appeared again. I'm on 7.5.6 so I thought that the DISTRIBUTED CACHE IS NOT UPDATED was fixed in there. Anything else I can do?

    Jeroen

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 06:30
    Shannon Deminick
    0

    well it is 'fixed', but if there's other unknown problems, then i guess those are still unknown.

    do you have any custom DatabaseServerMessengerOptions configured or anything else I should know about? ... perhaps how to replicate, what leads to this problem, any other details you wish to share?

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 06:31
    Shannon Deminick
    1

    You've totally hijacked this thread btw ;)

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 12, 2017 @ 06:51
    Jeroen Breuer
    0

    Sorry I should provide more information. And yes this is going a bit off topic ;-).

    This is the same inherited project as the Autofac issues when rebuilding Examine indexes topic.

    There is a custom web api which calls a cache refresher on all servers. This cache refresher rebuilds the Examine indexes on all servers like this:

    public void RefreshAll()
    {
        var cacheManager = new ServiceLocator().GetCacheManager();
        cacheManager.ExternalRuntimeCache.Clear();
        cacheManager.RuntimeCache.Clear();
        cacheManager.ClearOutputCache();
    
        ExamineManager.Instance.IndexProviderCollection
            .Where(p => p.Name == "ArticleIndexer" || p.Name == "TrainingIndexer")
            .ToList()
            .ForEach(index => index.RebuildIndex());
    }
    

    This needs to be done because the content in Examine is not just Umbraco content, but also from a external 3rd party XML file. So the save and publish PageCacheRefresher isn't enough.

    We get the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message when we call the web api.

    There are no custom DatabaseServerMessengerOptions configured.

    Is it possible that the "DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions" message isn't the real error? Maybe because we do some funky things with Examine some goes wrong with that?

    Maybe the Examine indexes in Azure blob storage could also help here ;-).

    Jeroen

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 07:11
    Shannon Deminick
    1

    This sounds like a very bad idea. Rebuilding indexes is just bad: https://our.umbraco.org/Documentation/Reference/Common-Pitfalls/#rebuilding-indexes

    You will have issues with this I guarantee especially if this is called often.

    The problem is that it's taking too long to process the instructions, a single instruction shouldn't do something like rebuild all indexes. The fix we put in is to ensure that only a certain amount of instructions are processed at once, the instructions we normally process are things like updating in memory cache or updating an entry in the xml cache file or re-index a single node.

    If you want to hack a solution together you could run the logic on a background thread but this will just be masking another problem because you shouldn't be rebuilding indexes unless they become corrupt.

    ThreadPool.QueueUserWorkItem(o => DoTheRebuilding());
    

    I think you'll need to re-think this external xml file and how it integrates with content and indexes.

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 12, 2017 @ 07:17
    Jeroen Breuer
    0

    Hi Shannon,

    Thank you for answering this. I agree that it's a bad idea. It's an inherited project and we're adding improvements on the go.

    I think I'll go with the hack solution for now because we have this problem on a live environment so we need to fix it asap.

    We already have some ideas on how to improve the external XML integration, but that's something for the long run.

    Jeroen

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 07:29
    Shannon Deminick
    0

    Ok, good luck with that ... don't say i didn't warn you ;)

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 12, 2017 @ 09:52
    Jeroen Breuer
    0

    After looking a bit better at the code with another developer it seems that the external data isn't stored in Examine. I thought this was the case because some content only appeared after updating the external XML and rebuilding the indexes, but apparently the content showed because some cache was cleared. It was late last night ;-).

    Since the Examine indexes only store Umbraco content there is no reason to rebuild the indexes with the clear cache web api. I actually don't know why the previous developer added the reindex code, but simply removing the reindex code should be enough.

    Will give another update when we deployed the changes.

    Jeroen

  • Jeroen Breuer 4860 posts 12126 karma points MVP 3x admin c-trib
    Jan 12, 2017 @ 11:17
    Jeroen Breuer
    0

    Changes have been deployed to the live environment. Since we no longer reindex the Examine indexes as a instruction everything works like it should.

    Thanks Shannon for helping.

    Jeroen

  • Shannon Deminick 1486 posts 4986 karma points hq
    Jan 12, 2017 @ 13:34
    Shannon Deminick
    1

    that is precisely why that is written as an anti-pattern, i see this far too often, some developers for whatever (most likely old legacy reasons) think that they need to rebuild indexes all the time, we've seen this on every request in some cases (wat!? :)

    glad you got it sorted, site should run a lot quicker now!

Please Sign in or register to post replies

Write your reply to:

Draft