I've raised a bug in GitHub, but thought I'd check here to see if anyone has any thoughts.

Bug summary

3 times in the last 11 days we've observed an issue on a production website where Umbraco is reporting a large number of "Missing cmsContentNu published content for node {NodeId}, consider rebuilding." warnings, followed by a large number of "Umbraco has no content" warnings in the logs.

Specifics

We're running Umbraco in Azure with 2 x (fixed - not auto-scaling) front end instances. When this issue occurs, it only ever affects one of the instances - I can verify this by deleting the ARRAffinity cookie until I hit the other instance, which is always fine. This leads me to believe the issue is not with the database table, but the locally stored NuCache files on the affected instance, otherwise I'd expect it to affect all instances.

I've noticed a pattern in the logs in that, when this issue occurs, it's the first time that particular machine has been used in a while (based on the machine name in the log files). I've made the following observations:

UmbracoTraceLog.RD501AC5770A50.20210913

• log file starts with umbraco booting on 13/09/2021
• this machine hasnt been used since 22/04/2021
• reports a number of "Missing cmsContentNu published content for node {NodeId}, consider rebuilding" warnings
• followed by a number of "Umbraco has no content" warnings

UmbracoTraceLog.RD501AC56ED7C2.20210910

• log file starts with umbraco booting on 10/09/2021
• this machine hasnt been used since 17/08/2021
• reports a number of "Missing cmsContentNu published content for node {NodeId}, consider rebuilding" warnings
• followed by a number of "Umbraco has no content" warnings

UmbracoTraceLog.RD0003FF8A886F.20210902

• log file starts with umbraco booting on 02/09/2021
• this machine hasn't been used since 05/10/2020
• reports a number of "Missing cmsContentNu published content for node {NodeId}, consider rebuilding" warnings
• followed by a number of "Umbraco has no content" warnings

The issue can be resolved by either performing a reload of the memory cache (which further suggests to me that the issue is with the local files rather than the database) or performing an App Service restart. However, it then re-occurs in a few days.

Based on the pattern above, my question would be whether it's possible that the issue is in some way related to running on a machine that hasn't been used for a while - potentially some stale temp data for example. However, as I understand, the local temporary storage does not persist between app pool recycles, so it wouldn't really make sense that this would be the case.

Is there any reason you can think of that this issue would keep occurring and only ever on 1/3 instances of the running application?

When the issue occurred this morning, I set WEBSITE_DISABLE_SCM_SEPARATION to true, so next time it occurs I'll be able to compare the local temporary storage on the two instances and should be able to see if there's any obvious issue i.e. files missing/different size on the affected server. Would be good in the meantime however to understand if there's any explanation for why this may be happening.

Steps to reproduce

I've not yet been able to find a way of reproducing this - I'm hoping you may be able to lead me in the direction of some steps that would cause such behaviour to occur!