Low performance when several admin users are logged in (Umbraco 7.2.4)
Hello!
We have WEB-farm for Umbraco website consisting of four balanced frontend servers for internet access by users and one backend server for staff access to Umbraco admin panel. All servers are running on Windows 2012 R2 Server + IIS8.5. Database is running on a separate server with MS SQL Server 2014.
Our website usually has about 100 visitors simultaneously and we are observing a weird thing. From 10 am to 7 pm when staff (about 5 to 10 users simultaneously) is working in admin panel our website works terribly slow and CPU usage by IIS Worker process on all frontend servers rises up to 100% almost all the time. But after 7 pm when all users disconnect from Umbraco admin panel our web site with the same number of visitors starts to work fine.
Load balancing is configured according to documentation on load-balanced scheme for Umbraco.
Admin staff doesn’t perform any “heavy” actions in admin panel like publishing content ten times a minute. Actually, they only collect information about users’ feedbacks and create new pages and media for future use without immediate publishing.
It looks like Umbraco admin panel generates additional load on frontend servers when actually we expect it to work on separated server only. What could be the problem? Does some obscure influence take place somewhere between admin backend and frontend?
I think the problem can be with updating xml for you site.
When admin is publishing node, Umbraco changes xml cache on the disk, this is a pretty expensive operation for performance.
Look at these config keys:
XmlCacheEnabled: Turn Xml caching of content on/off. Umbraco Makes heavy use of caching content in memory to avoid database calls. This makes umbraco faster and more efficient. You should not in any way turn this off, unless you have a very good reason to do so. It will make your website very slow.
ContinouslyUpdateXmlDiskCache: Update the XmlCache whenever content is published
Thank you for your answer.
We tried to disable XmlCache but after that another problem arouse, i. e. all our front-end servers started to query database much more often that made it overloaded and slow.
We investigated the issue using database data collection and profiling tools and found out that database spends a lot of time on waiting for unlock of umbracoUser table because there a lot of SELECT and UPDATE queries are executed: the former is to check backoffice-user access rights, the latter is to update user last login time.
After most of backoffice-users have logged out the system starts to work perfectly. I guess, the problem is in UserMembershopProvider in concurrent SELECTS and UPDATES of the same table. SELECTS are waiting for UPDATES to complete, thus some functionality in Umbraco admin panel is constantly waiting for the user access rights check to finish against the database.
The problem is keen. Are there any ideas of how to get rid of such delays?
Thank you, Alex. Of course, we created several indexes that we found to be useful having analyzed database queries but it didn't help to avoid waiting delays when SELECT queries are waiting for UPDATE queries to complete on umbracoUser table.
By now according to database operation analysis, exactly these delays seem to be the source of slow database working.
As for cleaning tool, we'll look at it. Thank you!
Low performance when several admin users are logged in (Umbraco 7.2.4)
Hello! We have WEB-farm for Umbraco website consisting of four balanced frontend servers for internet access by users and one backend server for staff access to Umbraco admin panel. All servers are running on Windows 2012 R2 Server + IIS8.5. Database is running on a separate server with MS SQL Server 2014.
Our website usually has about 100 visitors simultaneously and we are observing a weird thing. From 10 am to 7 pm when staff (about 5 to 10 users simultaneously) is working in admin panel our website works terribly slow and CPU usage by IIS Worker process on all frontend servers rises up to 100% almost all the time. But after 7 pm when all users disconnect from Umbraco admin panel our web site with the same number of visitors starts to work fine.
Load balancing is configured according to documentation on load-balanced scheme for Umbraco. Admin staff doesn’t perform any “heavy” actions in admin panel like publishing content ten times a minute. Actually, they only collect information about users’ feedbacks and create new pages and media for future use without immediate publishing.
It looks like Umbraco admin panel generates additional load on frontend servers when actually we expect it to work on separated server only. What could be the problem? Does some obscure influence take place somewhere between admin backend and frontend?
Hi AK,
I think the problem can be with updating xml for you site. When admin is publishing node, Umbraco changes xml cache on the disk, this is a pretty expensive operation for performance.
Look at these config keys:
XmlCacheEnabled: Turn Xml caching of content on/off. Umbraco Makes heavy use of caching content in memory to avoid database calls. This makes umbraco faster and more efficient. You should not in any way turn this off, unless you have a very good reason to do so. It will make your website very slow.
ContinouslyUpdateXmlDiskCache: Update the XmlCache whenever content is published
Maybe if you disable these functionality, problem will be solved. Read more : https://our.umbraco.org/wiki/reference/files-and-folders/files-in-the-config-folder/umbracosettingsconfig
Thanks,
Oleksandr
Hi, Alex!
Thank you for your answer. We tried to disable XmlCache but after that another problem arouse, i. e. all our front-end servers started to query database much more often that made it overloaded and slow.
We investigated the issue using database data collection and profiling tools and found out that database spends a lot of time on waiting for unlock of umbracoUser table because there a lot of SELECT and UPDATE queries are executed: the former is to check backoffice-user access rights, the latter is to update user last login time.
After most of backoffice-users have logged out the system starts to work perfectly. I guess, the problem is in UserMembershopProvider in concurrent SELECTS and UPDATES of the same table. SELECTS are waiting for UPDATES to complete, thus some functionality in Umbraco admin panel is constantly waiting for the user access rights check to finish against the database.
The problem is keen. Are there any ideas of how to get rid of such delays?
Hi AK,
Maybe you can make some fixes on DB level ? Maybe indexes ?
Also this package is helpful - https://our.umbraco.org/projects/website-utilities/unversion
It cleans DB.
Thanks,
Alex
Thank you, Alex. Of course, we created several indexes that we found to be useful having analyzed database queries but it didn't help to avoid waiting delays when SELECT queries are waiting for UPDATE queries to complete on umbracoUser table.
By now according to database operation analysis, exactly these delays seem to be the source of slow database working.
As for cleaning tool, we'll look at it. Thank you!
You are welcome, interesting topic, write results )
is working on a reply...