Umbraco upgrade wizard in flexible load balanced environment
Hey gang,
Umbraco 7.3.8 here. 3 web servers, 1 responsible for administrating content and 2 consumer facing behind a load balancer. If you try accessing /umbraco on the consumer facing sites, it'll redirect the user to the management server. Quick example:
Now, when we push a new version of Umbraco to the 3 web servers (let's say 7.3.8 being upgraded from 7.3.7), the Umbraco upgrade wizard runs on the management server and then updates the config just on that server. As you know, Umbraco looks at the version in the web.config and compares it to the version in the Umbraco library DLLs to determine whether to run the upgrade wizard.
The problem is the other 2 web servers have no idea that Umbraco was upgraded, so their config files still say 7.3.7...so when visiting the site it'll keep punting the user over to the management server to upgrade Umbraco.
Now, the not so happy way to handle this is to update the web.configs version from 7.3.7 to 7.3.8 manually. But is there a better workflow for this that people are doing? If we add additional web servers, this upgrade process will get tedious.
There's a few things at work here, lets talk first without load balancing:
Firstly we need to have a way to configure the splash screen for when the installer kicks in so end users don't see it if one is configured and instead displays any page you like.
Second is we then need to provide a way for the adminstrator to run the installer as per normal - this should be fine to detect because we know they will be logged in.
This same principal will need to be used when we implement Package Migrations - since when a migration is detected it will also run a version of the installer.
Some advice for now would be to:
Be sure to check in your web.config into version control - I'm sure you already have done this then be sure that when you upgrade you deploy your changes to your servers. No matter what, when you upgrade you will have to deploy the changes/dlls/etc... to the other servers so you should also be deploying the web.config update.
On you master server, even if you deploy the web.config update with the new version in it, it will still execute the installer because the version in the umbraco versions table isn't consistent. Once the installer finishes, all of your servers will serve pages since they share the same db.
Until we have the ability in the core to put a holding page up instead of the installer screen you can acheive this yourselve by either using an HttpModule or OWIN middleware to detect if ApplicationContext.Current.IsConfigured, if it is not then the installer will need to run, you could check this on BeginRequest, return a holding page on your slave servers and end the request
Your question on twitter was about scripting the installation process, there's an important point to know up front: We will not be able to script the upgrade/installer without a web context, the website will need to be running.
But as for 'scripting' the upgrade with a REST API, this is what Umbraco as a Service does for the patch release upgrades. If you want to see how that works, perform an upgrade on a local site and analyze the REST requests made by the upgrader - you too can use this endpoint if you want to script it. There are limitations to this process because if user input is required for any of the steps you will need to know how to map those values to the values passed in the installer REST api - but for patch and probably almost all minor releases this won't be necessary. We also don't consider changes to this REST endpoint breaking so you'd want to be aware of that.
That said, i don't see that scripting the installer to run on your slave servers is going to solve many issues, a holding page will still be required if you don't want your end users to see the installer screen and you need to deploy all changes including web.config updates to all servers anyways.
Thanks for the in depth reply. I have a couple of questions, but real quick:
We do have the web.config checked into version control...and we have XML transforms when deploying to the appropriate environments (dev, QA, staging, production).
So questions:
So you're saying if we set the version ahead of time (say: 7.3.8) before the site is updated, there's actually a DB check that determines whether to run the upgrader? I always thought it was a comparison of the DLL and what was in the config. This is fascinating.
Would the consumer facing sites cache the upgrade check? That is, if our master and slave sites all are set to 7.3.8, and we run the upgrader on the master and the slave sites are getting requests...once the master is done the upgrade process....will the slave sites automatically start serving content?
Now, re: holding page, how would we determine if it's a slave server? We're doing flexible load balancing, so curious if there are any Umbraco hooks we can take advantage of?
Thanks for all the information about scripting it. I figured we needed a web/Umbraco context. But it is interesting to see how UaaS does its thing.
That upgrade check has been in there since 7.3 iirc. This was done for this particular reason (more or less) because people check in their changes to their web.config and then deploy to another environment. Prior to this change, the installer would not execute because it thought it was on the correct version and then terrible things would happen.
Now we store this version information in the database. So we do two checks, one against the web.config and another against the db. You can test this by looking in your umbracoMigration table and removing the version you are currently on while leaving your web.config at the correct version - you will have to bump your web.config to see the affect because this check only happens on startup.
So that might answer your other question - since we only check on startup, the slave sites will not automatically just start serving requests once the db record is in there. That is something that we should certainly allow for when we allow for a custom holding page. This page should check on each request if it's ready and then start serving pages. Currently the installer screen will not do this.
How do you know if it's a slave?
You should absolutely know because you should only have 1 master server, all others are slaves. How you set that up is up to you. Maybe you access the master server from a different DNS address, in which case you can check if you are on that DNS address and determine that it is the master (don't show the holding page), otherwise show the holding page. There's probably several ways you can achieve this. You can also force servers to be readonly slaves and in which case you would already have this logic built in: https://our.umbraco.org/Documentation/Getting-Started/Setup/Server-Setup/Load-Balancing/flexible-advanced
Umbraco upgrade wizard in flexible load balanced environment
Hey gang,
Umbraco 7.3.8 here. 3 web servers, 1 responsible for administrating content and 2 consumer facing behind a load balancer. If you try accessing /umbraco on the consumer facing sites, it'll redirect the user to the management server. Quick example:
http://manage.website.com - not load balanced http://www.website.com - 2 web servers, load balanced
So if trying to access http://www.website.com/umbraco, it'll redirect the user to http://manage.website.com/umbraco.
Now, when we push a new version of Umbraco to the 3 web servers (let's say 7.3.8 being upgraded from 7.3.7), the Umbraco upgrade wizard runs on the management server and then updates the config just on that server. As you know, Umbraco looks at the version in the web.config and compares it to the version in the Umbraco library DLLs to determine whether to run the upgrade wizard.
The problem is the other 2 web servers have no idea that Umbraco was upgraded, so their config files still say 7.3.7...so when visiting the site it'll keep punting the user over to the management server to upgrade Umbraco.
Now, the not so happy way to handle this is to update the web.configs version from 7.3.7 to 7.3.8 manually. But is there a better workflow for this that people are doing? If we add additional web servers, this upgrade process will get tedious.
There's a few things at work here, lets talk first without load balancing:
Firstly we need to have a way to configure the splash screen for when the installer kicks in so end users don't see it if one is configured and instead displays any page you like.
Second is we then need to provide a way for the adminstrator to run the installer as per normal - this should be fine to detect because we know they will be logged in.
This same principal will need to be used when we implement Package Migrations - since when a migration is detected it will also run a version of the installer.
Some advice for now would be to:
Your question on twitter was about scripting the installation process, there's an important point to know up front: We will not be able to script the upgrade/installer without a web context, the website will need to be running.
But as for 'scripting' the upgrade with a REST API, this is what Umbraco as a Service does for the patch release upgrades. If you want to see how that works, perform an upgrade on a local site and analyze the REST requests made by the upgrader - you too can use this endpoint if you want to script it. There are limitations to this process because if user input is required for any of the steps you will need to know how to map those values to the values passed in the installer REST api - but for patch and probably almost all minor releases this won't be necessary. We also don't consider changes to this REST endpoint breaking so you'd want to be aware of that.
That said, i don't see that scripting the installer to run on your slave servers is going to solve many issues, a holding page will still be required if you don't want your end users to see the installer screen and you need to deploy all changes including web.config updates to all servers anyways.
Hi Shannon,
Thanks for the in depth reply. I have a couple of questions, but real quick:
We do have the web.config checked into version control...and we have XML transforms when deploying to the appropriate environments (dev, QA, staging, production).
So questions: So you're saying if we set the version ahead of time (say: 7.3.8) before the site is updated, there's actually a DB check that determines whether to run the upgrader? I always thought it was a comparison of the DLL and what was in the config. This is fascinating.
Would the consumer facing sites cache the upgrade check? That is, if our master and slave sites all are set to 7.3.8, and we run the upgrader on the master and the slave sites are getting requests...once the master is done the upgrade process....will the slave sites automatically start serving content?
Now, re: holding page, how would we determine if it's a slave server? We're doing flexible load balancing, so curious if there are any Umbraco hooks we can take advantage of?
Thanks for all the information about scripting it. I figured we needed a web/Umbraco context. But it is interesting to see how UaaS does its thing.
-Steve
That upgrade check has been in there since 7.3 iirc. This was done for this particular reason (more or less) because people check in their changes to their web.config and then deploy to another environment. Prior to this change, the installer would not execute because it thought it was on the correct version and then terrible things would happen.
Now we store this version information in the database. So we do two checks, one against the web.config and another against the db. You can test this by looking in your umbracoMigration table and removing the version you are currently on while leaving your web.config at the correct version - you will have to bump your web.config to see the affect because this check only happens on startup.
So that might answer your other question - since we only check on startup, the slave sites will not automatically just start serving requests once the db record is in there. That is something that we should certainly allow for when we allow for a custom holding page. This page should check on each request if it's ready and then start serving pages. Currently the installer screen will not do this.
You should absolutely know because you should only have 1 master server, all others are slaves. How you set that up is up to you. Maybe you access the master server from a different DNS address, in which case you can check if you are on that DNS address and determine that it is the master (don't show the holding page), otherwise show the holding page. There's probably several ways you can achieve this. You can also force servers to be readonly slaves and in which case you would already have this logic built in: https://our.umbraco.org/Documentation/Getting-Started/Setup/Server-Setup/Load-Balancing/flexible-advanced
Hope that helps
This is all great info, Shannon. Thank you.
Are there plans to bake this holding page into 7.5.x?
7.5 is feature frozen it would need to wait until 7.6
is working on a reply...