I have a Fasthosts dedicated server on which I run several websites in Umbraco. The machine has 2GB RAM running Windows 2003 web edition. On the machine is SQL Server 2005 Express, along with Filezilla FTP server and several sites running umbraco 4.1.
None of these sites are huge in terms of nodes, nor in terms of pageviews. They vary in size from 10 pages with 60 page views per day to 100 pages with about 300 page views per day. Each site has it's own application pool, primarily so that I can track resources used per site.
The issue is that these few sites are chewing up a large amount of memory on the server. 200mb is going to SQL Server, with 70-150mb going to the Umbraco instances. At this rate I'll only be able to pack on around another 5 sites before I run out of RAM.
I've had this situation in the past on a previous server where the number of umbraco sites caused me to run out of RAM, in this case the machine slowed sigificantly and started swapping to disk a lot. As a short term solution I started recycling app-pools every 30mins which kept the amount of memory down, but that seems like a daft idea as I loose the whole cache when I do that. I then setup this new server, but as I add new sites I reaching the same problem as previous.
What ideas / tips/ best practice guidelines do people have for running multi-tennanted Umbraco servers? Should sites share app pools, and what are the pros/cons? Should I setup recycling of app pools?
Am I missing the point, because this is expected behaviour?
I've just tried using some hints and tips I found elsewhere, including switching on the caching of macro output. But that seems to have made no discernable difference to the memory footprint of each webapp.
Hi Paul, we are experiencing the same behaviour except we have larger sites (nodes) and more pageviews. We have 5 sites running umbraco v4, 4 smaller sites < 1000 nodes and 1 large site (12.000 nodes). In total we have around 1.000.000 pageviews a month. SQL server is consuming around 1 to 1.5 GB for these 5 sites, and the w3wp.exe process around 300mb for the smaller sites and around 500mb for the large site.
We have tuned the macro caching. The server is a win 2003 server with 8gb.
I'd like to hear some ideas / tips/ best practices too!
Like any technology, you can implement sites that are lean and those that are memory hogs, and that's true of umbraco sites as well.
As a general rule of thumb, better performance is (often) coincident with a leaner site that requires fewer overall resources.
Do think about caching your macro's output and setting good cache times (even 5 seconds makes a performance difference on busy sites) and you can often cache macro output for much much longer periods. Be sure to disable the 'by page' option if the macro produces the same output no matter what page is being displayed as this will save some memory.
You can find out which macros are the slowest and either cache them extensively or re-work the to speed them up by adding ?umbDebugShowTrace=true to the querystring of the url. If you've got the debug info enabled in the web.config file you'll get a lot of extra output. Umbraco macros are usually very fast... a few hundredths of a second) but caching can give a 10 to 100-fold improvement.
Next, consider SQL server. It is a memory hog and will take all it can get unless you configure it otherwise. This is worth doing. Bear in mind that umbraco makes lots of database calls when you're working with the umbraco administration interface but few calls for a live site that your visitors see. Thus, you can restrain sql server from using all your resources, at the expense of potentially slowing the admin experience slightly. There's some trial and error involved in finding the right setting for this and you can google for advice on finding that balance.
You might also share application pools. That will save some memory and resources but may not be an option you wish to look into. There are good reasons for wanting separate pools, and decent reasons you might choose to put multiple sites in a single pool, especially the smaller sites.
Also be sure to disable all debugging and similar options in the web.config, which is important for live sites once you've optimized them.
I'm sure there are other things you can look into, but this is a good start.
Let us know what you find, this is a very interesting discussion!
Macro caching doesn't mean a whole lot here as the footprint of a cached macro is very, very little. To understand memory usage in a .NET application it's important to remember that it'll use as much memory as you allow it until it reaches the maximum with only minor cleaning along the way. This *is* intended behaviour in .NET and not a memory leak.
The biggest memory hog in Umbraco is the internal content cache as each published node is represented as an in-memory cached XmlNode in a .NET XmlDocument. At the same time this document is cloned (in memory) whenever you publish. So say you have 12.000 nodes each containing 10kb of data. That makes a total xmldocument be 120mb of raw xml data. When this is represented as a .NET XmlDocument it'll probably grow 50% in size due to meta data so 180mb is likely. Then it'll be cloned upon publishing meaning that the published xml alone is 360mb.
Number of visits/pageviews doesn't really affect the memory usage of Umbraco. If you need to host sevaral sites on the same box you could try to experiment with the memory size of each app pool and see how low it can go. A site with <100 pages should need more than 20-30mb RAM I guess (without any test backing that statement up I must emphasize!). One last thing to be aware of is that on 32bit Windows a process cannot consume more than 2gb of memory, so unless you run 64bit Windows, your application pool will never be able to utilize more than this.
Hope this helps! Would be excellent to do some matrix with tests and stats as well as looking at the core to see if we could optimize memory usage further.
So just to put 2 and 2 together. The cached copy of the published nodes are stored on disk as data\umbraco.config. This file is loaded in memory assume 50% overhead for metadata. A publish will double the memory consumed by cloning that.
So one of my sites has a 28k data\umbraco.config file. So it need minimum 84k for cached content. But it's consuming 61mb.
So whats the other 60mb doing? Obviously there's going to be a fixed cost in terms of memory to run Umbraco, and a variable cost. In this case that variable cost is associated with the number of nodes. So with a low number of nodes the fixed cost is more significant.
Looking at the app pool, I can set a maximum memory before the app pool recycles. Now each of this small sites take about 4 - 5 hours to consume ~120mb of RAM. Which means that if I set a Maximum Used Memory (in Memory Recycling on the App Pool) of 20mb then they'll recycle every hour.
Is there another setting that I'm missing which sets a maximum memory without recycling?
BTW - I've had the idea of a white paper on performance testing multi-tenanted servers using Umbraco for a while. It was on the task list for my Intern and I this summer, but customers sadly got in the way.
I totally agree with douglas SQL server is the memory hog.
Limit maximum memory used by SQL Server 2005
By default SQL server express 2005 sets the SQL server maximum memory to approximately 2,000 GB
the memory counsumed by SQL server will continue to increase and decrease rapidly (SQL serve will hog memory from other services like IIS , filezila and mail server).
I prefer to set SQL server 2005 "maximum memory" to 1/4 of total server memory.
For your server with 2GB memory try setting the Maximum SQL server memory to 512MB.
To set SQL server 2005 maximum memory to 512 MB , you have 2 options:
1) In (SQL Server Managment Studio) Connect to Object Explorer -> Right click on server node and go to Server Properties -> Memory -> Change the Maximum server memory in (MB) to 512.
2) Run the following query after connecting to the server.
sp_configure 'max server memory', 512 RECONFIGURE GO
Sorry tareg, you might be right in prinicple, but in this case SqlServer isn't hogging memory. It's consuming less than the 1/4 of total RAM than you suggest setting it to.
It's definately the w3wp.exe processes running each of the Umbraco sites, that appear to be on the chubby side to me!
I had a similar problem on a shared hosting site. Changing the macros in my instance dramatically changed the memory usage, from 310k+ down to 110K. While this might not have helped you, other topic lurkers should give Doug's excellent advice a shot.
Squeezing more sites onto a dedicated server
I have a Fasthosts dedicated server on which I run several websites in Umbraco. The machine has 2GB RAM running Windows 2003 web edition. On the machine is SQL Server 2005 Express, along with Filezilla FTP server and several sites running umbraco 4.1.
None of these sites are huge in terms of nodes, nor in terms of pageviews. They vary in size from 10 pages with 60 page views per day to 100 pages with about 300 page views per day. Each site has it's own application pool, primarily so that I can track resources used per site.
The issue is that these few sites are chewing up a large amount of memory on the server. 200mb is going to SQL Server, with 70-150mb going to the Umbraco instances. At this rate I'll only be able to pack on around another 5 sites before I run out of RAM.
I've had this situation in the past on a previous server where the number of umbraco sites caused me to run out of RAM, in this case the machine slowed sigificantly and started swapping to disk a lot. As a short term solution I started recycling app-pools every 30mins which kept the amount of memory down, but that seems like a daft idea as I loose the whole cache when I do that. I then setup this new server, but as I add new sites I reaching the same problem as previous.
What ideas / tips/ best practice guidelines do people have for running multi-tennanted Umbraco servers? Should sites share app pools, and what are the pros/cons? Should I setup recycling of app pools?
Am I missing the point, because this is expected behaviour?
I've just tried using some hints and tips I found elsewhere, including switching on the caching of macro output. But that seems to have made no discernable difference to the memory footprint of each webapp.
Hi Paul, we are experiencing the same behaviour except we have larger sites (nodes) and more pageviews. We have 5 sites running umbraco v4, 4 smaller sites < 1000 nodes and 1 large site (12.000 nodes). In total we have around 1.000.000 pageviews a month. SQL server is consuming around 1 to 1.5 GB for these 5 sites, and the w3wp.exe process around 300mb for the smaller sites and around 500mb for the large site.
We have tuned the macro caching. The server is a win 2003 server with 8gb.
I'd like to hear some ideas / tips/ best practices too!
Like any technology, you can implement sites that are lean and those that are memory hogs, and that's true of umbraco sites as well.
As a general rule of thumb, better performance is (often) coincident with a leaner site that requires fewer overall resources.
Do think about caching your macro's output and setting good cache times (even 5 seconds makes a performance difference on busy sites) and you can often cache macro output for much much longer periods. Be sure to disable the 'by page' option if the macro produces the same output no matter what page is being displayed as this will save some memory.
You can find out which macros are the slowest and either cache them extensively or re-work the to speed them up by adding ?umbDebugShowTrace=true to the querystring of the url. If you've got the debug info enabled in the web.config file you'll get a lot of extra output. Umbraco macros are usually very fast... a few hundredths of a second) but caching can give a 10 to 100-fold improvement.
Next, consider SQL server. It is a memory hog and will take all it can get unless you configure it otherwise. This is worth doing. Bear in mind that umbraco makes lots of database calls when you're working with the umbraco administration interface but few calls for a live site that your visitors see. Thus, you can restrain sql server from using all your resources, at the expense of potentially slowing the admin experience slightly. There's some trial and error involved in finding the right setting for this and you can google for advice on finding that balance.
You might also share application pools. That will save some memory and resources but may not be an option you wish to look into. There are good reasons for wanting separate pools, and decent reasons you might choose to put multiple sites in a single pool, especially the smaller sites.
Also be sure to disable all debugging and similar options in the web.config, which is important for live sites once you've optimized them.
I'm sure there are other things you can look into, but this is a good start.
Let us know what you find, this is a very interesting discussion!
cheers,
doug.
Macro caching doesn't mean a whole lot here as the footprint of a cached macro is very, very little. To understand memory usage in a .NET application it's important to remember that it'll use as much memory as you allow it until it reaches the maximum with only minor cleaning along the way. This *is* intended behaviour in .NET and not a memory leak.
The biggest memory hog in Umbraco is the internal content cache as each published node is represented as an in-memory cached XmlNode in a .NET XmlDocument. At the same time this document is cloned (in memory) whenever you publish. So say you have 12.000 nodes each containing 10kb of data. That makes a total xmldocument be 120mb of raw xml data. When this is represented as a .NET XmlDocument it'll probably grow 50% in size due to meta data so 180mb is likely. Then it'll be cloned upon publishing meaning that the published xml alone is 360mb.
Number of visits/pageviews doesn't really affect the memory usage of Umbraco. If you need to host sevaral sites on the same box you could try to experiment with the memory size of each app pool and see how low it can go. A site with <100 pages should need more than 20-30mb RAM I guess (without any test backing that statement up I must emphasize!). One last thing to be aware of is that on 32bit Windows a process cannot consume more than 2gb of memory, so unless you run 64bit Windows, your application pool will never be able to utilize more than this.
Hope this helps! Would be excellent to do some matrix with tests and stats as well as looking at the core to see if we could optimize memory usage further.
Cheers,
Niels...
So just to put 2 and 2 together. The cached copy of the published nodes are stored on disk as data\umbraco.config. This file is loaded in memory assume 50% overhead for metadata. A publish will double the memory consumed by cloning that.
So one of my sites has a 28k data\umbraco.config file. So it need minimum 84k for cached content. But it's consuming 61mb.
So whats the other 60mb doing? Obviously there's going to be a fixed cost in terms of memory to run Umbraco, and a variable cost. In this case that variable cost is associated with the number of nodes. So with a low number of nodes the fixed cost is more significant.
Looking at the app pool, I can set a maximum memory before the app pool recycles. Now each of this small sites take about 4 - 5 hours to consume ~120mb of RAM. Which means that if I set a Maximum Used Memory (in Memory Recycling on the App Pool) of 20mb then they'll recycle every hour.
Is there another setting that I'm missing which sets a maximum memory without recycling?
BTW - I've had the idea of a white paper on performance testing multi-tenanted servers using Umbraco for a while. It was on the task list for my Intern and I this summer, but customers sadly got in the way.
I totally agree with douglas SQL server is the memory hog.
Limit maximum memory used by SQL Server 2005
By default SQL server express 2005 sets the SQL server maximum memory to approximately 2,000 GB
the memory counsumed by SQL server will continue to increase and decrease rapidly (SQL serve will hog memory from other services like IIS , filezila and mail server).
I prefer to set SQL server 2005 "maximum memory" to 1/4 of total server memory.
For your server with 2GB memory try setting the Maximum SQL server memory to 512MB.
To set SQL server 2005 maximum memory to 512 MB , you have 2 options:
1) In (SQL Server Managment Studio) Connect to Object Explorer -> Right click on server node and go to Server Properties -> Memory -> Change the Maximum server memory in (MB) to 512.
2) Run the following query after connecting to the server.
sp_configure 'max server memory', 512
RECONFIGURE
GO
http://support.microsoft.com/kb/920093
http://support.microsoft.com/kb/321363
Sorry tareg, you might be right in prinicple, but in this case SqlServer isn't hogging memory. It's consuming less than the 1/4 of total RAM than you suggest setting it to.
It's definately the w3wp.exe processes running each of the Umbraco sites, that appear to be on the chubby side to me!
Out of interest Paul what did you do to sort this out? Just use the memory setting on the app pool?
I had a similar problem on a shared hosting site. Changing the macros in my instance dramatically changed the memory usage, from 310k+ down to 110K. While this might not have helped you, other topic lurkers should give Doug's excellent advice a shot.
is working on a reply...