Major performance issues with large Umbraco sites with Members
Hi All
Not sure how best to tackle this one but here goes. We've build a humongous site in Umbraco 4.9 for a client and are experiencing major performance issues.
Site wise, the scope of the project is for around 220 clubs websites around the UK. We currently have around 80 domains in the one install of Umbraco all with centralised management for members.
Some numbers on the tables we have at the moment to give you an idea of size:
cmsContentCount = 45572 rows
cmsContentXmlCount = 36960 rows
cmsContentTypeCount = 107 rows
cmsDocumentCount = 87472 rows
cmsDocumentTypeCount = 625 rows
cmsMemberCount = 1997 rows
cmsMemberTypeCount = 108 rows
cmsMember2MemberGroupCount = 1443 rows
cmsPreviewXmlCount = 15537 rows
cmsPropertyDataCount = 2201702 rows
cmsPropertyTypeCount = 517 rows
cmsStylesheetCount = 52 rows
umbracoDomainsCount = 83 rows
Umbraco.config file is around 79meg
The following is what we are experiencing:
Importing members - typically we are importing members from a spreadsheet for each club, can be anything from 400 - 1500 members, around 20 properties per member. CPU usage is averaging around 80% whilst this import is going on both with our custom import module and also similar with CMSImport. Network traffic to the DB is also massive, so much so that the server has actually crashed occasionally due to the high loads - is there a better way for us to be managing members in Umbraco i.e. split out the members tables completely and would this help performance?
Publishing is slow - there is one event handler in place but even if that is disabled, its still slow taking many seconds to publish one node. Multiple node publishing is painful and occasionally times out. When we do a full republish using http;//YOURDOMAIN/Umbraco/dialogs/republish.aspx?xml=true which we often need to do for some reason, if this process completes it can take in excess of 1hr 15 mins and obviously gets longer each time as more data is added to the sites.
CSS tree is very slow to initially load. Once a file is selected, large files are very hard to navigate around (Editor very slow and unresponsive)
Server wise the infrastructure is run upon Amazon EC2 (managed by our client), IIS on one server, SQL Server on another box with fast links between the two.
What we have done so far to help improve things:
Added indexes to some of the SQL tables - this has helped although there are still a lot of seeks and scans in the tables which should not be happening.
Removed properties from the document types and trying to extract more to other custom sections on the site i.e. members move out of the clubs homepage into a dedicated section
Refactored our code to the nth degree to ensure it's running as efficiently as possible
Modified core to support SQL Connection Timeout property
I can reproduce a lot of the issues locally on my dev and staging environments but no where near to the same extent as on the live boxes (i.e. I rarely get SQL Timeouts for instance).
Any ideas on what we can do to improve performance and how we can cope with the large amounts of data being bounded around here.
Creating and updating members is particularly slow. If we create a member for instance, a hidden exception seems to be thrown but the member is still created! This will obviously be impacting performance. I've not yet debugged this but it's there and real!
You won't find these issues in smaller sites but when you get to larger sites like in this instance, especially with many members on the system, Umbraco really does start to creak.
From the investigations here, hopefully we can develop some best practices for improving the speed and performance of large Umbraco installations which of course will mean even faster smaller sites.
1. When working with members, do you use the built-ind methods and properties on the Member object from the Umbraco API? 2. Are you using Razor or XSLT for outputting data? 3. When working with the Umbraco API in general from C#: Are you using Node over Document whereever this is possible? 4. Which version are you running?
I think you could gain some good performance when performing reads on the members by using some of the Member extension methods from uComponents uQuery. What I'm using for getting members is the uQuery.GetMembersByXPath(string xpath) method. It's very fast :-)
Or your could move some read-logic to Examine for even faster performance.
Bonus question: do you, by any chance, use a lot of extension methods (custom ones) in your XSLT?
Thanks, will look into those - for the member directory I wrote i'm actually using Lee Messengers uMember library which is faster but pulls back too much data - got a new query to optimise that.
The members performance is mostly around importing data into Umbraco in the first place, that is where we are hitting major issues.
There are a few extension methods but not overly used.
A few years back, I came to the same conclusion you have come to now: The Umbraco Member API is painfully slow.
Sadly, AFAIK, that's just how things are. If you want to get decent performance, you have to go another route. My suggestion would definitely be to look at using a custom table for members and member types and then import member data to these tables using either raw sql or a simple data access component like Simple.Data. I would then use DEWD to setup ways to query the members in any way you see fit; DEWD is really an awesome little tool for jobs like this.
Again, publishing in Umbraco is slow. There is a lot of stuff going on at each publish and as you found out, it scales poorly. IIRC Umbraco has an upper limit of around 50k nodes before performance starts to take a serious hit. AFAIK, the reason why it starts to take so long is because of the xml cache being constantly updated, which is very slow when the physical Umbraco.config xml file gets large enough. Someone please correct me if I’m wrong.
Unfortunately, I don’t think there’s anything to do about it at the moment. Work is being done on a new caching system for Umbraco which will drop the xml cache completely while still supporting xslt. If/when this cache is fully implemented the upper limit for nodes should increase at least tenfold while maintaining great performance. Of course, this doesn’t help you now, and it’s just to let you know, that the future of Umbraco is indeed shaping up to better handle large Umbraco installations.
That’s all for me, hopefully you found some of it just a tad helpful :)
Using Lucene.Net, can a solution be built for this issue, as I've read it's very fast for parsing nodes? I've not used it myself, bu it could be worth considering.
Major performance issues with large Umbraco sites with Members
Hi All
Not sure how best to tackle this one but here goes. We've build a humongous site in Umbraco 4.9 for a client and are experiencing major performance issues.
Site wise, the scope of the project is for around 220 clubs websites around the UK. We currently have around 80 domains in the one install of Umbraco all with centralised management for members.
Some numbers on the tables we have at the moment to give you an idea of size:
The following is what we are experiencing:
Server wise the infrastructure is run upon Amazon EC2 (managed by our client), IIS on one server, SQL Server on another box with fast links between the two.
What we have done so far to help improve things:
I can reproduce a lot of the issues locally on my dev and staging environments but no where near to the same extent as on the live boxes (i.e. I rarely get SQL Timeouts for instance).
Any ideas on what we can do to improve performance and how we can cope with the large amounts of data being bounded around here.
Creating and updating members is particularly slow. If we create a member for instance, a hidden exception seems to be thrown but the member is still created! This will obviously be impacting performance. I've not yet debugged this but it's there and real!
You won't find these issues in smaller sites but when you get to larger sites like in this instance, especially with many members on the system, Umbraco really does start to creak.
From the investigations here, hopefully we can develop some best practices for improving the speed and performance of large Umbraco installations which of course will mean even faster smaller sites.
Simon
Hi Simon,
There's a few things I'm curious about here:
1. When working with members, do you use the built-ind methods and properties on the Member object from the Umbraco API?
2. Are you using Razor or XSLT for outputting data?
3. When working with the Umbraco API in general from C#: Are you using Node over Document whereever this is possible?
4. Which version are you running?
All the best,
Bo
HI Bo
To answer your questions:
Hi Simon,
I think you could gain some good performance when performing reads on the members by using some of the Member extension methods from uComponents uQuery. What I'm using for getting members is the uQuery.GetMembersByXPath(string xpath) method. It's very fast :-)
Or your could move some read-logic to Examine for even faster performance.
Bonus question: do you, by any chance, use a lot of extension methods (custom ones) in your XSLT?
All the best,
Bo
Hi
Thanks, will look into those - for the member directory I wrote i'm actually using Lee Messengers uMember library which is faster but pulls back too much data - got a new query to optimise that.
The members performance is mostly around importing data into Umbraco in the first place, that is where we are hitting major issues.
There are a few extension methods but not overly used.
Simon
Hey Simon
A few years back, I came to the same conclusion you have come to now:
The Umbraco Member API is painfully slow.
Sadly, AFAIK, that's just how things are. If you want to get decent performance, you have to go another route. My suggestion would definitely be to look at using a custom table for members and member types and then import member data to these tables using either raw sql or a simple data access component like Simple.Data. I would then use DEWD to setup ways to query the members in any way you see fit; DEWD is really an awesome little tool for jobs like this.
Again, publishing in Umbraco is slow. There is a lot of stuff going on at each publish and as you found out, it scales poorly. IIRC Umbraco has an upper limit of around 50k nodes before performance starts to take a serious hit. AFAIK, the reason why it starts to take so long is because of the xml cache being constantly updated, which is very slow when the physical Umbraco.config xml file gets large enough. Someone please correct me if I’m wrong.
Unfortunately, I don’t think there’s anything to do about it at the moment. Work is being done on a new caching system for Umbraco which will drop the xml cache completely while still supporting xslt. If/when this cache is fully implemented the upper limit for nodes should increase at least tenfold while maintaining great performance. Of course, this doesn’t help you now, and it’s just to let you know, that the future of Umbraco is indeed shaping up to better handle large Umbraco installations.
That’s all for me, hopefully you found some of it just a tad helpful :)
Regards
Mads
Using Lucene.Net, can a solution be built for this issue, as I've read it's very fast for parsing nodes? I've not used it myself, bu it could be worth considering.
Is this issue just for the members section?
is working on a reply...