Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Chris 7 posts 25 karma points
    Aug 03, 2020 @ 12:16
    Chris
    0

    Distributed cache - failed to execute on Master/admin server

    Hi,

    I'm getting a lot of errors (some days thousands) in the log files that say the following:

    2020-08-03 21:41:08,301 [P4692/D2/T33] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 948354, instruction count: 5). Instruction is being skipped/ignored
    System.NullReferenceException: Object reference not set to an instance of an object.
       at TeaCommerce.Umbraco.Application.Caching.TeaCommerceCacheRefresherBase`3.Refresh(String jsonPayload)
       at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
       at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)
     2020-08-03 21:41:08,317 [P4692/D2/T33] INFO  Umbraco.Core.Sync.DatabaseServerMessenger - The current batch of instructions was not processed, app is shutting down
    

    The ID of the instruction (e.g. 948354) appears once per instruction and once the errors start it repeats the error for each subsequent instruction.

    The instructions in the UmbracoCacheInstruction look like this:

    [{"RefreshType":4,"RefresherId":"a3b1db6a-48f2-4c2c-9114-8ce7922e376e","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":null,"JsonIdCount":1,"JsonPayload":"{\"instanceId\":\"659152E170234CE892BC4C80851F06A6\",\"storeId\":1,\"id\":\"bd46acb7-26e5-42f2-8fcc-0fe9b5646099\",\"action\":1}"}]
    

    Checking the JsonPayload id I can see it refers to a Teacommerce order that exists in the database.

    This is occurring on the admin server, which is set as the Master, and not on the public web servers.

    Also the LastSynced ID Does stay up to date in the temp directory on this server.

    Can you help?

    Thanks, Chris.

  • Matt Brailsford 2768 posts 14049 karma points MVP 7x c-trib
    Aug 03, 2020 @ 12:21
    Matt Brailsford
    0

    Hi Chris,

    What version of Tea Commerce are you running on?

    Matt

  • Chris 7 posts 25 karma points
    Aug 03, 2020 @ 12:30
    Chris
    0

    Hi Matt,

    Teacommerce: 3.4.2 Umbraco: 7.15.1

    Thanks, Chris.

  • Matt Brailsford 2768 posts 14049 karma points MVP 7x c-trib
    Aug 03, 2020 @ 12:57
    Matt Brailsford
    0

    Hmm,

    So the code for the cache refreshers is in our open source UI code and this is the refresh function https://github.com/TeaCommerce/Tea-Commerce-for-Umbraco/blob/master/Source/TeaCommerce.Umbraco.Application/Caching/TeaCommerceCacheRefresherBase.cs#L19

    Really, the only thing I could think could be null here could be the result of the CacheService.GetCacheValue call, which maybe we should also perform a null check in the if statement.

    Would you feel comfortable compiling the TC4Umb front end project to test this? or if I supplied you a DLL for you to test? Just I'm not sure if I could replicate this locally as I'm not sure under what scenario the cache could be null.

    Matt

  • Chris 7 posts 25 karma points
    Aug 03, 2020 @ 13:12
    Chris
    0

    I've been looking through the code and I'm not sure what could be causing this..

    TeaCommerceCacheRefresherPayload<TId> refresherPayload = JsonConvert.DeserializeObject<TeaCommerceCacheRefresherPayload<TId>>(jsonPayload);
          if (!(refresherPayload.InstanceId != Constants.InstanceId))
            return;
          ConcurrentDictionary<TId, TEntity> cacheValue = this.CacheService.GetCacheValue<ConcurrentDictionary<TId, TEntity>>(string.Format(this.CacheKeyFormat, (object) refresherPayload.StoreId, (object) refresherPayload.Id));
          if (cacheValue.ContainsKey(refresherPayload.Id))
          {
            TEntity entity;
            cacheValue.TryRemove(refresherPayload.Id, out entity);
          }
          base.Refresh(jsonPayload);
    

    It looks like the options are that one of these is null: this.CacheService, or cacheValue

    I am wondering whether GetCachedList has not been called in Orders and so the cacheValue is not set?

    I will test by loading the admin and searching orders then see if the errors are still appearing in the log.

    Back in 5..

  • Chris 7 posts 25 karma points
    Aug 03, 2020 @ 13:24
    Chris
    0

    The errors have stopped!

    I reckon this should do the trick:

    if (cacheValue != null && cacheValue.ContainsKey(refresherPayload.Id))
          {
            TEntity entity;
            cacheValue.TryRemove(refresherPayload.Id, out entity);
          }
    

    It does explain why I have been seeing it occur in batches - every order instruction that comes through for a while and then nothing for a time.

    I think if you want to recreate locally, it should be possible by starting the site without viewing a page.

    Then run this on the DB:

      INSERT INTO umbracoCacheInstruction (
      [utcStamp]
          ,[jsonInstruction]
          ,[originated]
          ,[instructionCount]
      ) VALUES
      (
      '2020-08-03 10:26:45.410'
      ,'[{"RefreshType":4,"RefresherId":"a3b1db6a-48f2-4c2c-9114-8ceeeeee3eee","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":null,"JsonIdCount":1,"JsonPayload":"{\"instanceId\":\"659152E170234CE892BC4C80851F06A6\",\"storeId\":1,\"id\":\"bd46acb7-26e5-42f2-8fcc-0fe9b5646099\",\"action\":1}"}]'
      ,'MyServer CF86DDE4D02D4060AE20477357090C6E'
      ,1);
    

    Do you think this makes sense?

  • Matt Brailsford 2768 posts 14049 karma points MVP 7x c-trib
    Aug 03, 2020 @ 13:31
    Matt Brailsford
    0

    Hi Chris,

    Yes, that make total sense now. As your master server isn't generating orders and you probably haven't been into the TC orders list section then the master server has never created it's local cache store and so when the refresher assumes the cache exists, it errors.

    So yup, looks like we need to add a null check around the returned cache value. 👍

    Nice detective work. I'll see what I can do to get an update out some time this week.

    Matt

  • Chris 7 posts 25 karma points
    Aug 03, 2020 @ 13:33
    Chris
    0

    Fantastic - if you could send me an email when ready that would be ace.

    Thanks, Chris.

  • Matt Brailsford 2768 posts 14049 karma points MVP 7x c-trib
    Aug 07, 2020 @ 08:32
    Matt Brailsford
    0

    Morning Chris,

    Just to let you know 3.4.3 is out now with this fix included.

    Many thanks for reporting this.

    Matt

Please Sign in or register to post replies

Write your reply to:

Draft