Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Dan Roddis 57 posts 241 karma points
    Oct 21, 2015 @ 08:29
    Dan Roddis
    0

    Umbraco 7.3 Azure Outage (Cache lost - Welcome to Umbraco)

    Hi all,

    Just wondering if you could help me with an issue we faced yesterday on a public facing website.

    For background:

    • The site is hosted on an Azure app service.
    • The site uses a V12 Azure database
    • We're running Umbraco 7.3

    Yesterday our front-end started displaying the 'Welcome to Umbraco' screen - you have no content (not the installer screen), following investigation we saw the following line in the logs:

     2015-10-20 16:36:07,502 [P2900/D2/T1] WARN  Umbraco.Core.Sync.DatabaseServerMessenger - No last synced Id found, this generally means this is a new server/install. The server will rebuild its caches and indexes and then adjust it's last synced id to the latest found in the database and will start maintaining cache updates based on that id

    I had a look at the code that generates this line and traced it back to the APP_DATA/TEMP/DistCache directory and can see the machine name changed at the exact same time.

    We fixed this by logging into the admin and republishing the entire node tree.

    I apologise if this is more of an Azure question than an Umbraco question as it seems like it's based on Azure load balancing behaviour? Is there any guidance why this happened and how we can configure Umbraco to prevent this happening in future?

  • Mark 91 posts -18 karma points
    Oct 22, 2015 @ 10:24
    Mark
    1

    Hi

    Looks like you just came across an issue with the new iServer Messenger changes in 7.3.

    Basically, the cache is now invalidated in load balanced environments via meta saved in the database.

    Not sure why a machine name change would cause issues but take a look at this presentation;

    https://vimeo.com/132815038

    the second half goes into details about Azure and cache invalidation.

    It may help you troubleshoot this issue.

    Let me know how you get on, would be interested to know what the exact cause was.

  • Dan Roddis 57 posts 241 karma points
    Oct 22, 2015 @ 10:58
    Dan Roddis
    0

    Thanks Mark, I appreciate the help - I'll take a look at the video and post back my findings.

    By machine name changed it may not be that the machine name was changed - but that Azure moved me to a different machine perhaps?

  • Mark 91 posts -18 karma points
    Oct 22, 2015 @ 11:08
    Mark
    0

    Absolutely and if you have auto scaling, you will have multiple instances with different names that may or may not change. This shouldn't matter with the new iServer messenger stuff, but its is new in 7.3 so you may be the first to of spotted a bug?

    Let me know if I can help, would be interested to get to the bottom of this as we are about to go to production in a similar environment.

  • Dan Roddis 57 posts 241 karma points
    Oct 22, 2015 @ 11:53
    Dan Roddis
    0

    OK - So I've checked out the video and it's really cleared up how Umbraco handles load balancing.

    I've double checked and there wasn't any scaling out, just being 'migrated' automatically by Azure and it looks like for some reason Umbraco tripped up.

    Following the machine change the logs are just filled with:

     2015-10-20 16:42:12,933 [P2900/D2/T24] WARN  Umbraco.Web.UmbracoModule - Umbraco has no content
    

    Until we published all children from the root node excluding unpublished content (a cache refresh didn't work).

    I checked the umbracoCacheInstruction table which confirms a change in the originating machine but nothing around the time of the Azure machine switch, we have our previous cache instructions, a gap of a couple of hours (as the machine switch happened outside of business hours) and then the instructions on the new machine triggered by our republish.

    I need to learn a little more about the Azure platform's app service as this may have been an unplanned event or could be something that's previously been happening and 7.2.2 just handled it differently.

    It's interesting that it had this undesirable outcome of assuming there was nothing there and presenting the empty Umbraco instance afterwards.

    Starting to think this is worthy of the issue tracker! I've raised it: http://issues.umbraco.org/issue/U4-7290

  • Tristan Stahnke 4 posts 25 karma points
    Dec 22, 2015 @ 02:31
    Tristan Stahnke
    0

    Hey Dan, is this still affecting you in 7.3.4? We've been in Azure on a single web app too and it'd be good to know if this has been resolved.

    The machine name is never guaranteed in Azure Web Apps because it's PaaS; it'll get shuffled without warning, which is also why you can never have a dedicated IP address. However, you shouldn't need to use a traditional load balanced config within an Azure Web App because the file system is shared across all instances in a single Azure Web App.

  • James Jackson-South 489 posts 1747 karma points c-trib
    Dec 22, 2015 @ 06:25
    James Jackson-South
    0

    Just placing an obligatory "Me too" here.

    I got hit by this issue at work the other day. This was with version 7.3.1

    I haven't seen anything related to this issue in any of the changelogs for more recent releases. I've added a vote to the raised issue.

  • Dan Roddis 57 posts 241 karma points
    Dec 22, 2015 @ 09:15
    Dan Roddis
    0

    @Tristan we ran on 7.3.0 until last week and never had that problem again, we've been running on 7.3.4 for a week and had no issues either.

    I can't say for sure if it's an event that can happen on Azure that trips Umbraco which we've only happened to experience once or it's since been fixed.

    We're occasionally noticing published items being lost from the cache 'Oops this item is published but not visible because it's not in the cache' or something along those lines in Umbraco 7.3.0 and still in 7.3.4 but having difficulty reproducing.

  • Gavin Faux 10 posts 75 karma points c-trib
    Dec 22, 2015 @ 15:00
    Gavin Faux
    0

    We've experienced this also though site is not running on Azure, and we could not see any server configuration change at the time.

    Raised an issue for it here

Please Sign in or register to post replies

Write your reply to:

Draft