We have a website with a large amount of content. Publishing can take a while, sometimes up to a couple of minutes. During publishing, the front-end of the website is un-responsive, presumably due to a lock being applied to the in memory content cache by the Umbraco publishing mechanism which prevents front-end threads from completing.
We've got a couple of ideas about how to make the scenario more palatable, one being that we split our environment up into a load balanaced environment. Or rather:
1 x server for content entry
1 x server for content delivery (i.e. front-end)
My question is, will this solve our problem? Presumably the distributed call to publish the content on the content delivery server will result in the same locks and therefore a hanging front end whilst the publish completes?
It might be a good idea to do load balancing. There is some nice documentation about that here - I have not tried doing it myself, but this documentation looks pretty thourough.
Generally speaking the front-end will not be blocked while the back end is publishing. There are however a few locks in place in the umbraco.content class which controls this and depending on the various settings that you may have configured and under certain circumstances like the app restarting then the front-end might be blocked while the xml cache is updated. The umbraco.content class needs to be rewritten as it's a bit of a mess and this work is in the pipeline.
I've ran some tests and the front-end doesn't block under normal publishing circumstances. Do you have any custom/non-standard configuration set that modifies how the xml cache works (i.e. there are various settings in umbracoSettings.config) ?
When you get this blocking on the front-end, are you actively debugging or running the site normally? Is this caused by a normal publish or a bulk publish?
Ok thanks. Can you replicate this on your dev/staging environment?
View a front-end page
In a different tab go to publish that same page
In the front-end page tab can you refresh it while publishing is occuring?
I know that is a rudimentary check but in all my tests locally with the standard settings (which it looks like you have) I can consistently refresh/reload front-end pages while they are publishing. I've even tested in the codebase to put Thread.Sleep inside the locking mechanisms during publishing and I can still refresh the front-end pages.
The difficulty is determining if/when the potential for blocking occurs in the code of umbraco.content for a front-end request. As far as I can tell so far this might only occur during an app startup.
Yes, that's the exact test which causes the front end thread to lock.
Could it be a database lock instead? As mentioned, Contour is used quite heavily on all of the pages on the site, and our client has mentioned the same issues when "saving" a document (which obviously won't update the xml cache).
This would also explain why the loading of the 301 redirects in the url tracker causes front end threads to run (which doesn't touch the content cache I presume, but just loads stuff from the database)?
For reference our Views / macros use the legacy api: umbraco.NodeFactory.Node
If you can replicate this locally/staging then that is terrific because this is going to end up being a trial and error check.
My advise would be to see if this might be a Contour thing - if it is you should be able to find a page with Contour on it that you can replicate this scenario, and then remove the rendering of Contour from the page rendering and see if the problem goes away.
The other thing to try is to create a new basic doc type with a basic template that doesn't have any extra functionality and see if you can replicate with that. If it works properly, then you can start adding complexity until you figure out when it starts blocking.
Thanks for taking the time to chat about this at CG. Since speaking to you I've ran some further tests, and even templates including no macros fail. i.e. the template includes just the text hello world. I've recorded a screenr here:
The site includes a lot of content - 1000's of nodes
There are a couple of "automation" events that fire on publish to invalidate the cache, but nothing too strenuous
It's possible to get the back-end to fail every time by right clicking on the home page node, doing a publish (and all sub pages), then refreshing the back-end brwoser tab. At this point, all front end threads will fail
Thanks for taking the time to make the screenr, this does help show the issue you are having.
So we know a few things now:
The locking isn't particularly related to macros
it's related to something in the request pipeline
something is locked during the request which is caused by something being locked during publish
So the next questions are:
Do you have any event handlers in your code (attached to saving/publishing of content items) ?
If you don't have any in your code, what packages do you have installed?
Do you have exceptions in your umbraco log during this time?
We need to determine if this is something specific to your install or in the Umbraco core. One way to test this would easily be to create a vanilla umbraco site, create a site structure with thousands of nodes and perform this test again - no event handlers, no plugins.
We're running some tests based on the above. In the meantime, is there anything in the reqeuest pipeline that looks at the database to determine the template to use for a given node? i.e. when receiving the current url, looking up a node then looking up which template to render?
We only have a template ID stored in the cache file, so for any first request where we have not looked up and cached the template with that particular ID we need to go get it's definition from the database.
So on first request, if the template ID is "2", we'll go look that template up in the db and cache it. On the next request, we have already cached the Template with id of "2" so another db lookup doesn't occur.
Hope that makes sense? This same sort of thing will happen with domain lookups, so in some cases a front-end request (generally the first requests) may make a db call.
Stephen has been running this in production for over a month and is working great. You could always test a custom Umbraco build with that PR applied and see if that fixes your issue.
Front-end locked whilst publishing, will Load Balancing help
We have a website with a large amount of content. Publishing can take a while, sometimes up to a couple of minutes. During publishing, the front-end of the website is un-responsive, presumably due to a lock being applied to the in memory content cache by the Umbraco publishing mechanism which prevents front-end threads from completing.
We've got a couple of ideas about how to make the scenario more palatable, one being that we split our environment up into a load balanaced environment. Or rather:
1 x server for content entry
1 x server for content delivery (i.e. front-end)
My question is, will this solve our problem? Presumably the distributed call to publish the content on the content delivery server will result in the same locks and therefore a hanging front end whilst the publish completes?
Hi Janusz
How many nodes are there?
It might be a good idea to do load balancing. There is some nice documentation about that here - I have not tried doing it myself, but this documentation looks pretty thourough.
I guess you might also benefit from setting up database indexes. We had a rather large site, which was pretty slow in the backoffice content and media sections. When we ran the db index stuff that Mikkel writes about here http://our.umbraco.org/forum/developers/extending-umbraco/42361-This-is-a-bit-dirtybut-I-need-to-mess-with-the-databasecomment#153824 it solved our issues on that solution. Make sure to read the other pages as well.
Hope this helps.
/Jan
Thanks Jan
I'm sure Load Balancing will help with the "load". However, if the front-end is still locked whilst publishing then the problem still occurs.
I've already applied the database indexes, but publishing can still be slow.
Generally speaking the front-end will not be blocked while the back end is publishing. There are however a few locks in place in the umbraco.content class which controls this and depending on the various settings that you may have configured and under certain circumstances like the app restarting then the front-end might be blocked while the xml cache is updated. The umbraco.content class needs to be rewritten as it's a bit of a mess and this work is in the pipeline.
I've ran some tests and the front-end doesn't block under normal publishing circumstances. Do you have any custom/non-standard configuration set that modifies how the xml cache works (i.e. there are various settings in umbracoSettings.config) ?
When you get this blocking on the front-end, are you actively debugging or running the site normally? Is this caused by a normal publish or a bulk publish?
Any other info to replicate would be appreciated.
Hi Shannon
Are these the settings you are looking for (these are the values set in our config file), or are there any others you'd like to see?
<!-- Enable / disable xml content cache -->
<XmlCacheEnabled>True</XmlCacheEnabled>
<!-- Update disk cache every time content has changed -->
<ContinouslyUpdateXmlDiskCache>False</ContinouslyUpdateXmlDiskCache>
<!-- Update in-memory cache if xml file is changed -->
<XmlContentCheckForDiskChanges>True</XmlContentCheckForDiskChanges>
In regards to the rest of your questions:
When you get this blocking on the front-end, are you actively debugging or running the site normally? : Runing the site normally on IIS &
Is this caused by a normal publish or a bulk publish?: Any back-end publish, bulk or single.
Other maybe related info:
- The site has about 200 concurrent users at any one time
- Most of the pages being hit include Contour forms (i.e. lots of requests coming in for pages with contour forms on them)
- Heavy use of ImageGen
Ok thanks. Can you replicate this on your dev/staging environment?
I know that is a rudimentary check but in all my tests locally with the standard settings (which it looks like you have) I can consistently refresh/reload front-end pages while they are publishing. I've even tested in the codebase to put Thread.Sleep inside the locking mechanisms during publishing and I can still refresh the front-end pages.
The difficulty is determining if/when the potential for blocking occurs in the code of umbraco.content for a front-end request. As far as I can tell so far this might only occur during an app startup.
Yes, that's the exact test which causes the front end thread to lock.
Could it be a database lock instead? As mentioned, Contour is used quite
heavily on all of the pages on the site, and our client has mentioned the
same issues when "saving" a document (which obviously won't update the xml
cache).
This would also explain why the loading of the 301 redirects in the url
tracker causes front end threads to run (which doesn't touch the content
cache I presume, but just loads stuff from the database)?
For reference our Views / macros use the legacy
api: umbraco.NodeFactory.Node
If you can replicate this locally/staging then that is terrific because this is going to end up being a trial and error check.
My advise would be to see if this might be a Contour thing - if it is you should be able to find a page with Contour on it that you can replicate this scenario, and then remove the rendering of Contour from the page rendering and see if the problem goes away.
The other thing to try is to create a new basic doc type with a basic template that doesn't have any extra functionality and see if you can replicate with that. If it works properly, then you can start adding complexity until you figure out when it starts blocking.
Hi Shannon
Thanks for taking the time to chat about this at CG. Since speaking to you I've ran some further tests, and even templates including no macros fail. i.e. the template includes just the text hello world. I've recorded a screenr here:
http://screenr.com/QJAN
For reference:
Thanks for taking the time to make the screenr, this does help show the issue you are having.
So we know a few things now:
So the next questions are:
We need to determine if this is something specific to your install or in the Umbraco core. One way to test this would easily be to create a vanilla umbraco site, create a site structure with thousands of nodes and perform this test again - no event handlers, no plugins.
Hi Shannon
We're running some tests based on the above. In the meantime, is there anything in the reqeuest pipeline that looks at the database to determine the template to use for a given node? i.e. when receiving the current url, looking up a node then looking up which template to render?
Hi,
We only have a template ID stored in the cache file, so for any first request where we have not looked up and cached the template with that particular ID we need to go get it's definition from the database.
So on first request, if the template ID is "2", we'll go look that template up in the db and cache it. On the next request, we have already cached the Template with id of "2" so another db lookup doesn't occur.
Hope that makes sense? This same sort of thing will happen with domain lookups, so in some cases a front-end request (generally the first requests) may make a db call.
Hi Shannon
We've not been able to isolate the issue yet, but it seems very similar to the experience mentioned here:
http://issues.umbraco.org/issue/U4-4931
What's your thoughts? Are there any fixes to this yet? We use Examine quite a bit.
There's a few 'experiences' mentioned in that issue, the Examine one is the least worry in that issue discussion.
If you think it's the Examine lock issue it will not hurt to try to downgrade your Examine version to: https://www.nuget.org/packages/Examine/0.1.52.2941
The comment trail for Examine downgrade starts around here: http://issues.umbraco.org/issue/U4-4931#comment=67-14693
However this will only be the case if you are running 6.2, otherwise you'll already be running the older Examine version.
It may be that your issue is related to the application cache locking issue, the PR is still under review here:
https://github.com/umbraco/Umbraco-CMS/pull/418
Stephen has been running this in production for over a month and is working great. You could always test a custom Umbraco build with that PR applied and see if that fixes your issue.
is working on a reply...