distributed cache failed to execute on masteradmin server - Tea Commerce support

Chris 8 posts 26 karma points

Aug 03, 2020 @ 12:16

Distributed cache - failed to execute on Master/admin server

Hi,

I'm getting a lot of errors (some days thousands) in the log files that say the following:

2020-08-03 21:41:08,301 [P4692/D2/T33] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 948354, instruction count: 5). Instruction is being skipped/ignored
System.NullReferenceException: Object reference not set to an instance of an object.
   at TeaCommerce.Umbraco.Application.Caching.TeaCommerceCacheRefresherBase`3.Refresh(String jsonPayload)
   at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
   at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)
 2020-08-03 21:41:08,317 [P4692/D2/T33] INFO  Umbraco.Core.Sync.DatabaseServerMessenger - The current batch of instructions was not processed, app is shutting down

The ID of the instruction (e.g. 948354) appears once per instruction and once the errors start it repeats the error for each subsequent instruction.

The instructions in the UmbracoCacheInstruction look like this:

[{"RefreshType":4,"RefresherId":"a3b1db6a-48f2-4c2c-9114-8ce7922e376e","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":null,"JsonIdCount":1,"JsonPayload":"{\"instanceId\":\"659152E170234CE892BC4C80851F06A6\",\"storeId\":1,\"id\":\"bd46acb7-26e5-42f2-8fcc-0fe9b5646099\",\"action\":1}"}]

Checking the JsonPayload id I can see it refers to a Teacommerce order that exists in the database.

This is occurring on the admin server, which is set as the Master, and not on the public web servers.

Also the LastSynced ID Does stay up to date in the temp directory on this server.

Can you help?

Thanks, Chris.

Copy Link

Matt Brailsford 4125 posts 22223 karma points MVP 9x c-trib

Aug 03, 2020 @ 12:21

Hi Chris,

What version of Tea Commerce are you running on?

Matt

Copy Link

Chris 8 posts 26 karma points

Aug 03, 2020 @ 12:30

Hi Matt,

Teacommerce: 3.4.2 Umbraco: 7.15.1

Thanks, Chris.

Copy Link

Matt Brailsford 4125 posts 22223 karma points MVP 9x c-trib

Aug 03, 2020 @ 12:57

Hmm,

So the code for the cache refreshers is in our open source UI code and this is the refresh function https://github.com/TeaCommerce/Tea-Commerce-for-Umbraco/blob/master/Source/TeaCommerce.Umbraco.Application/Caching/TeaCommerceCacheRefresherBase.cs#L19

Really, the only thing I could think could be null here could be the result of the CacheService.GetCacheValue call, which maybe we should also perform a null check in the if statement.

Would you feel comfortable compiling the TC4Umb front end project to test this? or if I supplied you a DLL for you to test? Just I'm not sure if I could replicate this locally as I'm not sure under what scenario the cache could be null.

Matt

Copy Link

Chris 8 posts 26 karma points

Aug 03, 2020 @ 13:12

I've been looking through the code and I'm not sure what could be causing this..

TeaCommerceCacheRefresherPayload<TId> refresherPayload = JsonConvert.DeserializeObject<TeaCommerceCacheRefresherPayload<TId>>(jsonPayload);
      if (!(refresherPayload.InstanceId != Constants.InstanceId))
        return;
      ConcurrentDictionary<TId, TEntity> cacheValue = this.CacheService.GetCacheValue<ConcurrentDictionary<TId, TEntity>>(string.Format(this.CacheKeyFormat, (object) refresherPayload.StoreId, (object) refresherPayload.Id));
      if (cacheValue.ContainsKey(refresherPayload.Id))
      {
        TEntity entity;
        cacheValue.TryRemove(refresherPayload.Id, out entity);
      }
      base.Refresh(jsonPayload);

It looks like the options are that one of these is null: this.CacheService, or cacheValue

I am wondering whether GetCachedList has not been called in Orders and so the cacheValue is not set?

I will test by loading the admin and searching orders then see if the errors are still appearing in the log.

Back in 5..

Copy Link

Chris 8 posts 26 karma points

Aug 03, 2020 @ 13:24

The errors have stopped!

I reckon this should do the trick:

if (cacheValue != null && cacheValue.ContainsKey(refresherPayload.Id))
      {
        TEntity entity;
        cacheValue.TryRemove(refresherPayload.Id, out entity);
      }

It does explain why I have been seeing it occur in batches - every order instruction that comes through for a while and then nothing for a time.

I think if you want to recreate locally, it should be possible by starting the site without viewing a page.

Then run this on the DB:

  INSERT INTO umbracoCacheInstruction (
  [utcStamp]
      ,[jsonInstruction]
      ,[originated]
      ,[instructionCount]
  ) VALUES
  (
  '2020-08-03 10:26:45.410'
  ,'[{"RefreshType":4,"RefresherId":"a3b1db6a-48f2-4c2c-9114-8ceeeeee3eee","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":null,"JsonIdCount":1,"JsonPayload":"{\"instanceId\":\"659152E170234CE892BC4C80851F06A6\",\"storeId\":1,\"id\":\"bd46acb7-26e5-42f2-8fcc-0fe9b5646099\",\"action\":1}"}]'
  ,'MyServer CF86DDE4D02D4060AE20477357090C6E'
  ,1);

Do you think this makes sense?

Copy Link

Matt Brailsford 4125 posts 22223 karma points MVP 9x c-trib

Aug 03, 2020 @ 13:31

Hi Chris,

Yes, that make total sense now. As your master server isn't generating orders and you probably haven't been into the TC orders list section then the master server has never created it's local cache store and so when the refresher assumes the cache exists, it errors.

So yup, looks like we need to add a null check around the returned cache value. 👍

Nice detective work. I'll see what I can do to get an update out some time this week.

Matt

Copy Link

Chris 8 posts 26 karma points

Aug 03, 2020 @ 13:33

Fantastic - if you could send me an email when ready that would be ace.

Thanks, Chris.

Copy Link

Matt Brailsford 4125 posts 22223 karma points MVP 9x c-trib

Aug 07, 2020 @ 08:32

Morning Chris,

Just to let you know 3.4.3 is out now with this fix included.

Many thanks for reporting this.

Matt

Copy Link

is working on a reply...

Flag this post as spam?

Distributed cache - failed to execute on Master/admin server