We have a large Umbraco 7.5.4 website (over 160,000 nodes) and we seem to have intermittent issues with the website going slow and hanging, this can happen maybe once or twice a week.
We have a database analyser running and it appears that the following SQL statement is causing the wait (see bottom of post).
Does anyone know what calls this SQL statement and if anything can be done to resolve this and stop the website hanging?
Just as a side note, we have "Umbraco.ModelsBuilder.Enable" set to "false" in the web.config to resolve another hanging issue with this version of Umbraco.
We also have "ContinouslyUpdateXmlDiskCache" set to "false" as we used to experience issues with the umbraco.config file rebuilding and becoming corrupt as it was such as large file.
The SQL statement that appears to be causing the wait is as follows:
(@0 nvarchar(40))
/* BEGIN ACTIVE SECTION (inserted by DPA) */
SELECT umbracoNode.id,
umbracoNode.parentID,
umbracoNode.sortOrder,
cmsContentXml.xml,
umbracoNode.level
FROM umbracoNode
INNER JOIN cmsContentXml
ON cmsContentXml.nodeId = umbracoNode.id
AND umbracoNode.nodeObjectType = @0
WHERE umbracoNode.id in
(
SELECT cmsDocument.nodeId
FROM cmsDocument
WHERE cmsDocument.published = 1
)
ORDER BY umbracoNode.level,
umbracoNode.parentID,
umbracoNode.sortOrder
/* END ACTIVE SECTION (inserted by DPA) */
Did anyone else have any solutions or suggestions for this? This has started to become a problem again.
Considering moving the website away from Umbraco or at least moving some of the node data (i.e. article on the website) to a different database and reading from there not using Umbraco for that part. Although this in itself would be a huge undertaking considering how many articles there are.
It seems as though Umbraco cannot cope with a site that has such as large amount of data, as previously mentioned all the pages are needed and they grow on a weekly basis of at least 100 new pages.
Are newer versions of Umbraco more efficient in how they handle the data/indexing etc and can cope with very large sites better, we are willing to consider upgrading to a newer version or rewriting the site in Umbraco 8 if this could solve the problem but again this would be a huge piece of work at a great cost to the customer and we don't want to attempt it if we are going to face exactly the same issues.
What version are you on now? We are on 7.14 and it seems better than earlier versions of 7. Our startup time is pretty slow, but once its up and running the performance seems to be pretty good now.
Thank you for the reply, we are still on version 7.5.4, we did attempt an upgrade about a year ago but ran into various issues, it is a very large complex site with lots of customisation's and of course the many nodes that get added to it on a weekly basis makes upgrading very difficult and if we can't guarantee that it will resolve the problem then it could be a big waste of time and money that the customer will not be very happy about paying for.
In general the actual performance of the site the majority of the time is really good, pages load in 1 or 2 seconds. A lot of the pages utilise the Lucene index to load content.
However every now and then it just hangs completely, CPU goes through the roof for the application pool and using a SQL database analyser tool I can see that the SQL query from my original post above is taking up a lot of time.
I was hoping someone might know what part of Umbraco calls this query, is it when the Examine index rebuilds completely (which shouldn't be happening as far as I'm aware), is it when a page gets opened, is there some other process that happens in the background that calls this query, is this query or the database refactored in later versions of Umbraco that make it no longer a problem in later version, or is this just a red herring and there is something else that could be causing the problem?
Umbraco site performance affected by xml cache (look on App_data\umbraco.config), big website generate big file and try use and load it.
In umbraco 8 it works much better :) If you wan't or can't move to Umbraco 8 you can try move pages to archive by unpuslihing old pages and retrive page content using custom DB and examine indexed (in any case page exists in index)
We do not use the umbraco.config file as we disabled this long ago because of problems with the size of the file getting too large and taking too long to regenerate.
If upgrading to Umbraco 8 is going to solve the problem then I would be happy to put this forward to the customer but that would require a big rewrite of the website, it still uses webforms so if we were going move to Umbraco 8 which from what I understand you cannot do an in place upgrade then we would most likely rewrite in MVC at the same time, it would essentially be rewriting the site from scratch but migrating over the content.
I am looking into solutions regarding archiving unpublished content and reading it through a different way, also possibly using Azure Search instead of Examine.
I would still like to know what calls the query mentioned above though if anyone does know the answer to that?
Yakov, from my understanding I thought that the XML cache that is saved to file is only used to load the XML cache into memory when the website starts up rather than calling the database, therefore making startup of the website faster. However we had problems with this becoming corrupt and needing to be regenerated therefore making the startup of the website extremely slow at times. Is that not the case?
From my understanding the umbraco.config is just a copy of the "cmsContentXml" table which is the cache of all the articles, by disabling the creation of of the umbraco.config file the website still uses a cache but when the website starts it loads the cache into memory from the "cmsContentXml" table rather than from the umbraco.config file, this means that the startup of the site is slower but everything else functions the same.
The website still uses a cache, all we have done is disabled the disk cache (i.e. the umbraco.config file) with the following setting:
I could potentially give it a try, do you think it is Examine that calls the query mentioned above, or is it something else like the XML cache that is loaded into memory? Is the XML cache maybe getting lost from memory then the website is calling that query to regenerate it?
The site works 99% of the time and is very quick, page loads of 1 or 2 seconds, it uses Examine index to load most of the data, it is just every now and then it hangs (once or twice a week maybe) for example today it has done it once, yesterday was fine but happened the day before. There is no correlation between when this occurs, it is completely random.
When it does hang a SQL profile tool we use shows the query mentioned in my original post is causing a long wait time, what part of Umbraco calls this query?
Kill the application pool process in Task Manager is usually the first step, this sometimes resolves it but it does sometimes come back again shortly after, and then like magic it will resolve itself and be okay for possibly days before happening again.
Do you have the rebuild examine indexes on startup turned on?
When you have the issue, have you looked in the developer section under example to see the status of the indexes? I seem to remember having issues with the indexes getting corrupted here and there for a while
We do not have "RebuildOnAppStart" set which I believe means it defaults to true.
No I haven't looked at the status of the indexes when this happens, I will try and do this next time.
I am going to start looking today at moving to Azure Search from Examine to see if this resolves the issue and the added benefit will be it will take some of the load off the website and web server.
The issue happened again this morning at just after 9:00, I checked the status of the indexes and they were not reporting any errors, CPU of app pool was hitting almost 100%.
I had to kill the app pool using task manager for the website and then the site seems to be running okay since (for the time being at least!).
Checked the SQL analyser and once again the same SQL query had spiked and was causing a long wait, screenshots below:
Large Umbraco website intermittently slow
We have a large Umbraco 7.5.4 website (over 160,000 nodes) and we seem to have intermittent issues with the website going slow and hanging, this can happen maybe once or twice a week.
We have a database analyser running and it appears that the following SQL statement is causing the wait (see bottom of post).
Does anyone know what calls this SQL statement and if anything can be done to resolve this and stop the website hanging?
Just as a side note, we have "Umbraco.ModelsBuilder.Enable" set to "false" in the web.config to resolve another hanging issue with this version of Umbraco.
We also have "ContinouslyUpdateXmlDiskCache" set to "false" as we used to experience issues with the umbraco.config file rebuilding and becoming corrupt as it was such as large file.
The SQL statement that appears to be causing the wait is as follows:
Helle trfletch ☺️
My large site was also hanging / very slow. The CPU was in max our solution was to move all redirects to the IIS. We was using Urlrewrite.
Are you using azure?
/TA
Delete some nodes? :-)
We have intermittent speed issues as well on 7.14.0; hosted in azure.
@TheArtist All our redirects are already using IIS Url Rewrite.
We are running on an Azure virtual server with the database running on a separate Azure virtual server.
@Paul Write Deleting nodes is not an option, all the content that is on the site is needed.
Did anyone else have any solutions or suggestions for this? This has started to become a problem again.
Considering moving the website away from Umbraco or at least moving some of the node data (i.e. article on the website) to a different database and reading from there not using Umbraco for that part. Although this in itself would be a huge undertaking considering how many articles there are.
It seems as though Umbraco cannot cope with a site that has such as large amount of data, as previously mentioned all the pages are needed and they grow on a weekly basis of at least 100 new pages.
Are newer versions of Umbraco more efficient in how they handle the data/indexing etc and can cope with very large sites better, we are willing to consider upgrading to a newer version or rewriting the site in Umbraco 8 if this could solve the problem but again this would be a huge piece of work at a great cost to the customer and we don't want to attempt it if we are going to face exactly the same issues.
What version are you on now? We are on 7.14 and it seems better than earlier versions of 7. Our startup time is pretty slow, but once its up and running the performance seems to be pretty good now.
Hi John,
Thank you for the reply, we are still on version 7.5.4, we did attempt an upgrade about a year ago but ran into various issues, it is a very large complex site with lots of customisation's and of course the many nodes that get added to it on a weekly basis makes upgrading very difficult and if we can't guarantee that it will resolve the problem then it could be a big waste of time and money that the customer will not be very happy about paying for.
In general the actual performance of the site the majority of the time is really good, pages load in 1 or 2 seconds. A lot of the pages utilise the Lucene index to load content.
However every now and then it just hangs completely, CPU goes through the roof for the application pool and using a SQL database analyser tool I can see that the SQL query from my original post above is taking up a lot of time.
I was hoping someone might know what part of Umbraco calls this query, is it when the Examine index rebuilds completely (which shouldn't be happening as far as I'm aware), is it when a page gets opened, is there some other process that happens in the background that calls this query, is this query or the database refactored in later versions of Umbraco that make it no longer a problem in later version, or is this just a red herring and there is something else that could be causing the problem?
Umbraco site performance affected by xml cache (look on App_data\umbraco.config), big website generate big file and try use and load it. In umbraco 8 it works much better :) If you wan't or can't move to Umbraco 8 you can try move pages to archive by unpuslihing old pages and retrive page content using custom DB and examine indexed (in any case page exists in index)
We do not use the umbraco.config file as we disabled this long ago because of problems with the size of the file getting too large and taking too long to regenerate.
If upgrading to Umbraco 8 is going to solve the problem then I would be happy to put this forward to the customer but that would require a big rewrite of the website, it still uses webforms so if we were going move to Umbraco 8 which from what I understand you cannot do an in place upgrade then we would most likely rewrite in MVC at the same time, it would essentially be rewriting the site from scratch but migrating over the content.
I am looking into solutions regarding archiving unpublished content and reading it through a different way, also possibly using Azure Search instead of Examine.
I would still like to know what calls the query mentioned above though if anyone does know the answer to that?
I found my Umbraco 8 sample -https://umb8-content.dyfort.com/category-1/
you can demo site using 479671 documents without using arhive, just umbraco api. pool size is 6G and site works, but it very simple usage
disabling xml cache is not recommened what I know.
I see in past article about it, I'll try find it
Main idea:
Yakov, from my understanding I thought that the XML cache that is saved to file is only used to load the XML cache into memory when the website starts up rather than calling the database, therefore making startup of the website faster. However we had problems with this becoming corrupt and needing to be regenerated therefore making the startup of the website extremely slow at times. Is that not the case?
Calling DB is very expencive becouse it need parse all fields from properties and versions.
Cache - In my vision it saves xml to disk and in memory, when you have a lot numbers of published node it make cache very big.
From my understanding the umbraco.config is just a copy of the "cmsContentXml" table which is the cache of all the articles, by disabling the creation of of the umbraco.config file the website still uses a cache but when the website starts it loads the cache into memory from the "cmsContentXml" table rather than from the umbraco.config file, this means that the startup of the site is slower but everything else functions the same.
The website still uses a cache, all we have done is disabled the disk cache (i.e. the umbraco.config file) with the following setting:
When you app is running - what it is application pool size?
Are you referring to the RAM being used by the application pool, this varies quite a lot, currently it is around 3GB
What version of Examine and Lucene do you have included in the project?
John, the Examine.dll version is 0.1.70.0, the Lucene.Net.dll version is 2.9.4.1
Mine are 0.1.90 and 2.9.4.1
I wonder if upgrading to the new examine would help with your issues...
I could potentially give it a try, do you think it is Examine that calls the query mentioned above, or is it something else like the XML cache that is loaded into memory? Is the XML cache maybe getting lost from memory then the website is calling that query to regenerate it?
As I understand it Examine is a wrapper around lucene.
We also sit between 2.5 and 3gb usually
In my vision 3G show what all data loads in memory.
I my prevouse experience sites based on Umbraco 7 with more than 50K pages can be see performance issues, becouse a big cache.
Also I see that sites with a lot numbers or archtype fields affected by performce issues.
What miniprofile saying for you ? My be you have not optimal queries?
The site works 99% of the time and is very quick, page loads of 1 or 2 seconds, it uses Examine index to load most of the data, it is just every now and then it hangs (once or twice a week maybe) for example today it has done it once, yesterday was fine but happened the day before. There is no correlation between when this occurs, it is completely random.
When it does hang a SQL profile tool we use shows the query mentioned in my original post is causing a long wait time, what part of Umbraco calls this query?
What do you do to clear the issue?
Kill the application pool process in Task Manager is usually the first step, this sometimes resolves it but it does sometimes come back again shortly after, and then like magic it will resolve itself and be okay for possibly days before happening again.
Do you have the rebuild examine indexes on startup turned on?
When you have the issue, have you looked in the developer section under example to see the status of the indexes? I seem to remember having issues with the indexes getting corrupted here and there for a while
We do not have "RebuildOnAppStart" set which I believe means it defaults to true.
No I haven't looked at the status of the indexes when this happens, I will try and do this next time.
I am going to start looking today at moving to Azure Search from Examine to see if this resolves the issue and the added benefit will be it will take some of the load off the website and web server.
The issue happened again this morning at just after 9:00, I checked the status of the indexes and they were not reporting any errors, CPU of app pool was hitting almost 100%.
I had to kill the app pool using task manager for the website and then the site seems to be running okay since (for the time being at least!).
Checked the SQL analyser and once again the same SQL query had spiked and was causing a long wait, screenshots below:
is working on a reply...