Umbraco 7.3 Azure Outage (Cache lost - Welcome to Umbraco)
Hi all,
Just wondering if you could help me with an issue we faced yesterday on a public facing website.
For background:
The site is hosted on an Azure app service.
The site uses a V12 Azure database
We're running Umbraco 7.3
Yesterday our front-end started displaying the 'Welcome to Umbraco' screen - you have no content (not the installer screen), following investigation we saw the following line in the logs:
2015-10-20 16:36:07,502 [P2900/D2/T1] WARN Umbraco.Core.Sync.DatabaseServerMessenger - No last synced Id found, this generally means this is a new server/install. The server will rebuild its caches and indexes and then adjust it's last synced id to the latest found in the database and will start maintaining cache updates based on that id
I had a look at the code that generates this line and traced it back to the APP_DATA/TEMP/DistCache directory and can see the machine name changed at the exact same time.
We fixed this by logging into the admin and republishing the entire node tree.
I apologise if this is more of an Azure question than an Umbraco question as it seems like it's based on Azure load balancing behaviour? Is there any guidance why this happened and how we can configure Umbraco to prevent this happening in future?
Absolutely and if you have auto scaling, you will have multiple instances with different names that may or may not change. This shouldn't matter with the new iServer messenger stuff, but its is new in 7.3 so you may be the first to of spotted a bug?
Let me know if I can help, would be interested to get to the bottom of this as we are about to go to production in a similar environment.
OK - So I've checked out the video and it's really cleared up how Umbraco handles load balancing.
I've double checked and there wasn't any scaling out, just being 'migrated' automatically by Azure and it looks like for some reason Umbraco tripped up.
Following the machine change the logs are just filled with:
2015-10-20 16:42:12,933 [P2900/D2/T24] WARN Umbraco.Web.UmbracoModule - Umbraco has no content
Until we published all children from the root node excluding unpublished content (a cache refresh didn't work).
I checked the umbracoCacheInstruction table which confirms a change in the originating machine but nothing around the time of the Azure machine switch, we have our previous cache instructions, a gap of a couple of hours (as the machine switch happened outside of business hours) and then the instructions on the new machine triggered by our republish.
I need to learn a little more about the Azure platform's app service as this may have been an unplanned event or could be something that's previously been happening and 7.2.2 just handled it differently.
It's interesting that it had this undesirable outcome of assuming there was nothing there and presenting the empty Umbraco instance afterwards.
Hey Dan, is this still affecting you in 7.3.4? We've been in Azure on a single web app too and it'd be good to know if this has been resolved.
The machine name is never guaranteed in Azure Web Apps because it's PaaS; it'll get shuffled without warning, which is also why you can never have a dedicated IP address. However, you shouldn't need to use a traditional load balanced config within an Azure Web App because the file system is shared across all instances in a single Azure Web App.
@Tristan we ran on 7.3.0 until last week and never had that problem again, we've been running on 7.3.4 for a week and had no issues either.
I can't say for sure if it's an event that can happen on Azure that trips Umbraco which we've only happened to experience once or it's since been fixed.
We're occasionally noticing published items being lost from the cache 'Oops this item is published but not visible because it's not in the cache' or something along those lines in Umbraco 7.3.0 and still in 7.3.4 but having difficulty reproducing.
Umbraco 7.3 Azure Outage (Cache lost - Welcome to Umbraco)
Hi all,
Just wondering if you could help me with an issue we faced yesterday on a public facing website.
For background:
Yesterday our front-end started displaying the 'Welcome to Umbraco' screen - you have no content (not the installer screen), following investigation we saw the following line in the logs:
I had a look at the code that generates this line and traced it back to the APP_DATA/TEMP/DistCache directory and can see the machine name changed at the exact same time.
We fixed this by logging into the admin and republishing the entire node tree.
I apologise if this is more of an Azure question than an Umbraco question as it seems like it's based on Azure load balancing behaviour? Is there any guidance why this happened and how we can configure Umbraco to prevent this happening in future?
Hi
Looks like you just came across an issue with the new iServer Messenger changes in 7.3.
Basically, the cache is now invalidated in load balanced environments via meta saved in the database.
Not sure why a machine name change would cause issues but take a look at this presentation;
https://vimeo.com/132815038
the second half goes into details about Azure and cache invalidation.
It may help you troubleshoot this issue.
Let me know how you get on, would be interested to know what the exact cause was.
Thanks Mark, I appreciate the help - I'll take a look at the video and post back my findings.
By machine name changed it may not be that the machine name was changed - but that Azure moved me to a different machine perhaps?
Absolutely and if you have auto scaling, you will have multiple instances with different names that may or may not change. This shouldn't matter with the new iServer messenger stuff, but its is new in 7.3 so you may be the first to of spotted a bug?
Let me know if I can help, would be interested to get to the bottom of this as we are about to go to production in a similar environment.
OK - So I've checked out the video and it's really cleared up how Umbraco handles load balancing.
I've double checked and there wasn't any scaling out, just being 'migrated' automatically by Azure and it looks like for some reason Umbraco tripped up.
Following the machine change the logs are just filled with:
Until we published all children from the root node excluding unpublished content (a cache refresh didn't work).
I checked the umbracoCacheInstruction table which confirms a change in the originating machine but nothing around the time of the Azure machine switch, we have our previous cache instructions, a gap of a couple of hours (as the machine switch happened outside of business hours) and then the instructions on the new machine triggered by our republish.
I need to learn a little more about the Azure platform's app service as this may have been an unplanned event or could be something that's previously been happening and 7.2.2 just handled it differently.
It's interesting that it had this undesirable outcome of assuming there was nothing there and presenting the empty Umbraco instance afterwards.
Starting to think this is worthy of the issue tracker! I've raised it: http://issues.umbraco.org/issue/U4-7290
Hey Dan, is this still affecting you in 7.3.4? We've been in Azure on a single web app too and it'd be good to know if this has been resolved.
The machine name is never guaranteed in Azure Web Apps because it's PaaS; it'll get shuffled without warning, which is also why you can never have a dedicated IP address. However, you shouldn't need to use a traditional load balanced config within an Azure Web App because the file system is shared across all instances in a single Azure Web App.
Just placing an obligatory "Me too" here.
I got hit by this issue at work the other day. This was with version 7.3.1
I haven't seen anything related to this issue in any of the changelogs for more recent releases. I've added a vote to the raised issue.
@Tristan we ran on 7.3.0 until last week and never had that problem again, we've been running on 7.3.4 for a week and had no issues either.
I can't say for sure if it's an event that can happen on Azure that trips Umbraco which we've only happened to experience once or it's since been fixed.
We're occasionally noticing published items being lost from the cache 'Oops this item is published but not visible because it's not in the cache' or something along those lines in Umbraco 7.3.0 and still in 7.3.4 but having difficulty reproducing.
We've experienced this also though site is not running on Azure, and we could not see any server configuration change at the time.
Raised an issue for it here
is working on a reply...