we have seen prolonged spikes of 100% CPU on an Umbraco CMS (latest version, we recently updated to 7.15.5, we have built an ecommerce with Ucommerce version 8) installation on Azure.
We contacted Microsoft support to help us in analyzing what was thrashing the CPU, the following is what we found (it's from a memory dump analyzed with WinDbg) :
0:000> !runaway2
Showing top 10 threads
Thread ID User Time 40 | 0 days 1:49:18.578
1 | 0 days 0:00:15.531
44 | 0 days 0:00:10.203
37 | 0 days 0:00:07.343
52 | 0 days 0:00:06.250
49 | 0 days 0:00:05.609
48 | 0 days 0:00:04.484
50 | 0 days 0:00:04.437
38 | 0 days 0:00:04.000
36 | 0 days 0:00:03.171
0:000> !AspxPagesExt
Address Completed Timeout Time (secs) ThreadId ReturnCode Verb Url 1fc454e8 no 110 6974 40 200 POST /umbraco/WebServices/NodeSorter.asmx/UpdateSortOrder?app=content
20b32b3c no 110 6246 38 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
20bb5338 no 110 6003 37 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
20f18380 no 110 3639 41 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
245cc6a8 no 110 3407 42 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
24c28200 no 110 3017 43 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
0:000> !mex.ClrStack2 -t 0x1a34
DbgId ThreadId Apartment Kind CLR GC Mode GC Suspending?
40 1a34 MTA Worker v4.7.3610.00 Cooperative no
Did anyone else ever found something similar on your website?
If not I would be glad if you could give a couple of suggestion to either better troubleshoot the issue or prevent it from happening is there is some common knoledge of this kind of issue.
We've just encountered a similar problem in 7.15.5
We are getting a passing bot overnight, which was handled fine before our upgrade from 7.14 but now causes 100% CPU which doesn't recover for hours after the traffic has passed.
I don't have a good profile dump to analyse but it seems to be similar to what you've found.
Strangely it seems fine during the day, under normal load. It's when the site is spidered, and a lot of blog listing pages are retrieved, that it seems to go crazy.
Umbraco V7 sometimes uses 100% CPU
Hi there,
we have seen prolonged spikes of 100% CPU on an Umbraco CMS (latest version, we recently updated to 7.15.5, we have built an ecommerce with Ucommerce version 8) installation on Azure.
We contacted Microsoft support to help us in analyzing what was thrashing the CPU, the following is what we found (it's from a memory dump analyzed with WinDbg) :
0:000> !runaway2
Showing top 10 threads
Thread ID User Time
40 | 0 days 1:49:18.578
1 | 0 days 0:00:15.531
44 | 0 days 0:00:10.203
37 | 0 days 0:00:07.343
52 | 0 days 0:00:06.250
49 | 0 days 0:00:05.609
48 | 0 days 0:00:04.484
50 | 0 days 0:00:04.437
38 | 0 days 0:00:04.000
36 | 0 days 0:00:03.171
0:000> !AspxPagesExt
Address Completed Timeout Time (secs) ThreadId ReturnCode Verb Url
1fc454e8 no 110 6974 40 200 POST /umbraco/WebServices/NodeSorter.asmx/UpdateSortOrder?app=content
20b32b3c no 110 6246 38 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
20bb5338 no 110 6003 37 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
20f18380 no 110 3639 41 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
245cc6a8 no 110 3407 42 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave 24c28200 no 110 3017 43 200 POST /umbraco/backoffice/UmbracoApi/Content/PostSave
0:000> !mex.ClrStack2 -t 0x1a34
DbgId ThreadId Apartment Kind CLR GC Mode GC Suspending?
40 1a34 MTA Worker v4.7.3610.00 Cooperative no
SP IP Function
0095e404 00000000 HelperMethodFrame
0095e474 71aa488f DomainNeutralILStubClass.ILSTUBPInvoke(IntPtr, IntPtr, System.String, Int32, System.String, Int32, Int32, System.String, Int32)
0095e47c 00000000 InlinedCallFrame
0095e4dc 71a38e19 System.Globalization.CompareInfo.IsPrefix(System.String, System.String, System.Globalization.CompareOptions)
0095e514 71a27b01 System.String.StartsWith(System.String, System.StringComparison)
0095e528 71a38e44 System.String.StartsWith(System.String)
0095e534 1ada1a53 Umbraco.Core.Cache.HttpRuntimeCacheProvider+<>c.
0:040> !dso OS Thread Id: 0x1a34 (40)
ESP/REG Object Name
0095DF18 3162e938 System.Linq.Enumerable+WhereEnumerableIterator1[[System.Collections.DictionaryEntry, mscorlib]]
0095DFE8 1b97a0f0 System.String umbrtmche-
0095DFEC 3bafc274 System.String NHibernate- Cache:CatalogFoundation:UCommerce.EntitiesV2.ProductProperty#343761@343761
0095DFF4 1372227c System.String it-IT
0095DFFC 1372227c System.String it-IT
0095E000 3bafc274 System.String NHibernate-Cache:CatalogFoundation:UCommerce.EntitiesV2.ProductProperty#343761@343761
0095E004 1b97a0f0 System.String umbrtmche-
0095E078 40168000 Lucene.Net.Store.SimpleFSDirectory+SimpleFSIndexInput+Descriptor 0095E0A4 3162e938 System.Linq.Enumerable+WhereEnumerableIterator1[[System.Collections.DictionaryEntry, mscorlib]]
0095E0A8 3162e95c System.Linq.Enumerable+ <>c_DisplayClass601[[System.Collections.DictionaryEntry, mscorlib]] 0095E0AC 070745cc Umbraco.Core.Cache.HttpRuntimeCacheProvider
The high CPU usage seems caused by the GetDictionaryEntries.
I've also checked the code on the DictionaryCacheProviderBase https://github.com/umbraco/Umbraco-CMS/blob/v7/dev/src/Umbraco.Core/Cache/DictionaryCacheProviderBase.cs
and https://github.com/umbraco/Umbraco-CMS/blob/v7/dev/src/Umbraco.Core/Cache/HttpRuntimeCacheProvider.cs
It doesn't seem to me like the dictionary is incorrectly used (ReaderWriterLockSlim should take care of race conditions).
Did anyone else ever found something similar on your website? If not I would be glad if you could give a couple of suggestion to either better troubleshoot the issue or prevent it from happening is there is some common knoledge of this kind of issue.
Thanks in advance,
Matteo
I don't have a solution I'm afraid, but I'm seeing the call umbraco/Webservices/nodesorter.asmx/UpdateSortOrder?app=content returning a 500.
I'm in the process of updating a production site to implement the following settings to see if it makes any difference:
https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/azure-web-apps-v7
Also updating the dataNvarchar column in cmsPropertyData to be nvarchar(max) https://our.umbraco.com/forum/umbraco-7/using-umbraco-7/68283-sort-pages-not-working
Also updating the naming of the nodesorter file and reference to it per William Charlton's solution here https://our.umbraco.com/forum/umbraco-7/using-umbraco-7/46822-Unable-to-Re-Sort-Content-in-Umbraco-700#comment-225738
I haven't implemented these yet so cannot say if they will help.
If you or anyone can help with this matter, it would be greatly appreciated.
We've just encountered a similar problem in 7.15.5
We are getting a passing bot overnight, which was handled fine before our upgrade from 7.14 but now causes 100% CPU which doesn't recover for hours after the traffic has passed.
I don't have a good profile dump to analyse but it seems to be similar to what you've found.
Strangely it seems fine during the day, under normal load. It's when the site is spidered, and a lot of blog listing pages are retrieved, that it seems to go crazy.
is working on a reply...