We where facing issue with umbraco 7.1.8 deployed on azure.

we analyze the dumps and this is what I found.

The following threads are waiting to enter a .NET Lock and are basically stuck here.

( 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 55 56 57 58 59 60 61 62 63 64 65 66 67 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 )

85.44% of threads blocked (264 threads)

If I have a look at the call stack of one of these threads “40”, I see the following.

0:040> kL

Child-SP RetAddr Call Site

00 00000079f980cdb8 00007ffe79dd13ed ntdll!ZwWaitForMultipleObjects+0xa

01 00000079f980cdc0 00007ffe72dc8976 KERNELBASE!WaitForMultipleObjectsEx+0xe1

02 00000079f980d0a0 00007ffe72dc877a clr!WaitForMultipleObjectsExSOTOLERANT+0x62

03 00000079f980d100 00007ffe72dc8591 clr!Thread::DoAppropriateWaitWorker+0x1d0

04 00000079f980d200 00007ffe72dc883d clr!Thread::DoAppropriateWait+0x7d

05 00000079f980d280 00007ffe72de3ed6 clr!CLREventBase::WaitEx+0xc0

06 00000079f980d310 00007ffe72de3dea clr!AwareLock::EnterEpilogHelper+0xc6

07 00000079f980d3d0 00007ffe72de4521 clr!AwareLock::EnterEpilog+0x62

08 00000079f980d430 00007ffe72de4293 clr!AwareLock::Contention+0x1e3

09 00000079f980d4f0 00007ffe147076b1 clr!JITutil_MonContention+0xaf

0a 00000079f980d680 00007ffe1470a605 umbraco!umbraco.content.CheckDiskCacheForUpdate()+0xa1

0b 00000079f980d710 00007ffe1470ab40 umbraco!umbraco.content.CheckXmlContentPopulation()+0x75

0c 00000079f980d780 00007ffe1470abf6 umbraco!umbraco.content.get_XmlContentInternal()+0x10

0d 00000079f980d7b0 00007ffe1470b451 umbraco!umbraco.content.get_XmlContent()+0x96

0e 00000079f980d7f0 00007ffe1470b5c2 umbraco!Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedContentCache.HasContent(Umbraco.Web.UmbracoContext, Boolean)+0x21

0f 00000079f980d830 00007ffe1470aedc umbraco!Umbraco.Web.PublishedCache.ContextualPublishedCache`1[[System.__Canon, mscorlib]].HasContent(Boolean)+0x22

10 00000079f980d860 00007ffe1470a38f umbraco!Umbraco.Web.UmbracoModule.EnsureUmbracoRoutablePage(Umbraco.Web.UmbracoContext, System.Web.HttpContextBase)+0x9c

* WARNING: Unable to verify checksum for System.Web.ni.dll

11 00000079f980d8c0 00007ffe6d2fcd1c umbraco!Umbraco.Web.UmbracoModule.ProcessRequest(System.Web.HttpContextBase)+0xff

12 00000079f980d940 00007ffe6d2dd495 SystemWebni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x5c

13 00000079f980d9a0 00007ffe6d2fabda SystemWebni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xa5

14 00000079f980da40 00007ffe6d2dd6a3 SystemWebni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x92a

15 00000079f980db90 00007ffe6d2d75de SystemWebni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x83

16 00000079f980dbe0 00007ffe6d2e0561 SystemWebni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x28e

17 00000079f980dc80 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x5c1

18 00000079f980de90 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

19 00000079f980dee0 00007ffe72d0b42e SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

1a 00000079f980df70 00007ffe6b2d1d6e clr!UMThunkStub+0x6e

1b 00000079f980e000 00007ffe6b2d1cfb webengine4!W3MGDHANDLER::ProcessNotification+0x78

1c 00000079f980e030 00007ffe6b2d1bfe webengine4!W3MGDHANDLER::DoWork+0x34f

1d 00000079f980e090 00007ffe6b2d27f3 webengine4!RequestDoWork+0x34e

1e 00000079f980e130 00007ffe6b3e8cfa webengine4!CMgdEngHttpModule::OnResolveRequestCache+0x1f

1f 00000079f980e170 00007ffe6b3e2908 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x1ae

20 00000079f980e1b0 00007ffe6b3e19a6 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x198

21 00000079f980e2c0 00007ffe6b3e2e3a iiscore!NOTIFICATION_CONTEXT::CallModules+0x36

22 00000079f980e320 00007ffe6b3e1e27 iiscore!NOTIFICATION_MAIN::DoWork+0x1cb

23 00000079f980e630 00007ffe6b2d2734 iiscore!W3CONTEXTBASE::IndicateCompletion+0x97

24 00000079f980e6a0 00007ffe6d3895eb webengine4!MgdIndicateCompletion+0x60

25 00000079f980e6d0 00007ffe6d2e074f SystemWebni!DomainNeutralILStubClass.ILSTUBPInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+0x7b

26 00000079f980e7a0 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x7af

27 00000079f980e9b0 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

28 00000079f980ea00 00007ffe72d0b683 SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

29 00000079f980ea90 00007ffe72d703ba clr!UM2MThunk_WrapperHelper+0x43

2a 00000079f980ead0 00007ffe72d690ef clr!UM2MThunk_Wrapper+0x5a

2b 00000079f980eb20 00007ffe72d70335 clr!Thread::DoADCallBack+0x139

2c 00000079f980ecd0 00007ffe72d0b62d clr!UM2MDoADCallBack+0x91

2d 00000079f980ed50 00007ffe6b2d1d6e clr!UMThunkStub+0x26d

2e 00000079f980ede0 00007ffe6b2d2276 webengine4!W3MGDHANDLER::ProcessNotification+0x78

2f 00000079f980ee10 00007ffe72d68b42 webengine4!ProcessNotificationCallback+0x42

30 00000079f980ee40 00007ffe72d2f046 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x122

31 00000079f980eee0 00007ffe72d2ef3a clr!ThreadpoolMgr::ExecuteWorkRequest+0x46

32 00000079f980ef10 00007ffe72defcb6 clr!ThreadpoolMgr::WorkerThreadStart+0xf4

33 00000079f980efa0 00007ffe7ae113d2 clr!Thread::intermediateThreadProc+0x7d

34 00000079f980fde0 00007ffe7c925454 kernel32!BaseThreadInitThunk+0x22

35 00000079f980fe10 0000000000000000 ntdll!RtlUserThreadStart+0x34

The call stack of other threads is also similar.

Looking at the code of “umbraco.content.CheckDiskCacheForUpdate()”, looks like we are trying to enter into a lock and are basically stuck here.

Now we need to identify who is owning the lock such that all these threads are in waiting state.

0:040> !SyncBlk

Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner

57 00000079f31b5188 5 1 00000079f660f550 1920 43 00000078a6fb02d0 System.Object

1572 00000079f64e6da8 525 1 00000079f8633530 110c 39 00000078a6fb02e8 System.Object

We have 2 locks here. One is held by thread 39 and other one by thread 43. Let us have a look at these threads one by one.

Thread 39 has the following call stack.

0:039> kL

Child-SP RetAddr Call Site

00 00000079f83bcd88 00007ffe79dd13ed ntdll!ZwWaitForMultipleObjects+0xa

01 00000079f83bcd90 00007ffe72dc8976 KERNELBASE!WaitForMultipleObjectsEx+0xe1

02 00000079f83bd070 00007ffe72dc877a clr!WaitForMultipleObjectsExSOTOLERANT+0x62

03 00000079f83bd0d0 00007ffe72dc8591 clr!Thread::DoAppropriateWaitWorker+0x1d0

04 00000079f83bd1d0 00007ffe72dc883d clr!Thread::DoAppropriateWait+0x7d

05 00000079f83bd250 00007ffe72de3ed6 clr!CLREventBase::WaitEx+0xc0

06 00000079f83bd2e0 00007ffe72de3dea clr!AwareLock::EnterEpilogHelper+0xc6

07 00000079f83bd3a0 00007ffe72de4521 clr!AwareLock::EnterEpilog+0x62

08 00000079f83bd400 00007ffe72de4293 clr!AwareLock::Contention+0x1e3

09 00000079f83bd4c0 00007ffe14707735 clr!JITutil_MonContention+0xaf

0a 00000079f83bd650 00007ffe1470a605 umbraco!umbraco.content.CheckDiskCacheForUpdate()+0x125

0b 00000079f83bd6e0 00007ffe1470ab40 umbraco!umbraco.content.CheckXmlContentPopulation()+0x75

0c 00000079f83bd750 00007ffe1470abf6 umbraco!umbraco.content.get_XmlContentInternal()+0x10

0d 00000079f83bd780 00007ffe1470b451 umbraco!umbraco.content.get_XmlContent()+0x96

0e 00000079f83bd7c0 00007ffe1470b5c2 umbraco!Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedContentCache.HasContent(Umbraco.Web.UmbracoContext, Boolean)+0x21

0f 00000079f83bd800 00007ffe1470aedc umbraco!Umbraco.Web.PublishedCache.ContextualPublishedCache`1[[System.__Canon, mscorlib]].HasContent(Boolean)+0x22

10 00000079f83bd830 00007ffe1470a38f umbraco!Umbraco.Web.UmbracoModule.EnsureUmbracoRoutablePage(Umbraco.Web.UmbracoContext, System.Web.HttpContextBase)+0x9c

11 00000079f83bd890 00007ffe6d2fcd1c umbraco!Umbraco.Web.UmbracoModule.ProcessRequest(System.Web.HttpContextBase)+0xff

12 00000079f83bd910 00007ffe6d2dd495 SystemWebni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x5c

13 00000079f83bd970 00007ffe6d2fabda SystemWebni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xa5

14 00000079f83bda10 00007ffe6d2dd6a3 SystemWebni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x92a

15 00000079f83bdb60 00007ffe6d2d75de SystemWebni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x83

16 00000079f83bdbb0 00007ffe6d2e0561 SystemWebni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x28e

17 00000079f83bdc50 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x5c1

18 00000079f83bde60 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

19 00000079f83bdeb0 00007ffe72d0b42e SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

1a 00000079f83bdf40 00007ffe6b2d1d6e clr!UMThunkStub+0x6e

1b 00000079f83bdfd0 00007ffe6b2d1cfb webengine4!W3MGDHANDLER::ProcessNotification+0x78

1c 00000079f83be000 00007ffe6b2d1bfe webengine4!W3MGDHANDLER::DoWork+0x34f

1d 00000079f83be060 00007ffe6b2d27f3 webengine4!RequestDoWork+0x34e

1e 00000079f83be100 00007ffe6b3e8cfa webengine4!CMgdEngHttpModule::OnResolveRequestCache+0x1f

1f 00000079f83be140 00007ffe6b3e2908 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x1ae

20 00000079f83be180 00007ffe6b3e19a6 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x198

21 00000079f83be290 00007ffe6b3e2e3a iiscore!NOTIFICATION_CONTEXT::CallModules+0x36

22 00000079f83be2f0 00007ffe6b3e1e27 iiscore!NOTIFICATION_MAIN::DoWork+0x1cb

23 00000079f83be600 00007ffe6b2d2734 iiscore!W3CONTEXTBASE::IndicateCompletion+0x97

24 00000079f83be670 00007ffe6d3895eb webengine4!MgdIndicateCompletion+0x60

25 00000079f83be6a0 00007ffe6d2e074f SystemWebni!DomainNeutralILStubClass.ILSTUBPInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+0x7b

26 00000079f83be770 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x7af

27 00000079f83be980 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

28 00000079f83be9d0 00007ffe72d0b683 SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

29 00000079f83bea60 00007ffe72d703ba clr!UM2MThunk_WrapperHelper+0x43

2a 00000079f83beaa0 00007ffe72d690ef clr!UM2MThunk_Wrapper+0x5a

2b 00000079f83beaf0 00007ffe72d70335 clr!Thread::DoADCallBack+0x139

2c 00000079f83beca0 00007ffe72d0b62d clr!UM2MDoADCallBack+0x91

2d 00000079f83bed20 00007ffe6b2d1d6e clr!UMThunkStub+0x26d

2e 00000079f83bedb0 00007ffe6b2d2276 webengine4!W3MGDHANDLER::ProcessNotification+0x78

2f 00000079f83bede0 00007ffe72d68b42 webengine4!ProcessNotificationCallback+0x42

30 00000079f83bee10 00007ffe72d2f046 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x122

31 00000079f83beeb0 00007ffe72d2ef3a clr!ThreadpoolMgr::ExecuteWorkRequest+0x46

32 00000079f83beee0 00007ffe72defcb6 clr!ThreadpoolMgr::WorkerThreadStart+0xf4

33 00000079f83bef70 00007ffe7ae113d2 clr!Thread::intermediateThreadProc+0x7d

34 00000079f83bff30 00007ffe7c925454 kernel32!BaseThreadInitThunk+0x22

35 00000079f83bff60 0000000000000000 ntdll!RtlUserThreadStart+0x34

Thread 43 has the following call stack.

0:043> kL

Child-SP RetAddr Call Site

00 00000079f5c9c0f8 00007ffe79dd13ed ntdll!ZwWaitForMultipleObjects+0xa

01 00000079f5c9c100 00007ffe72dc8976 KERNELBASE!WaitForMultipleObjectsEx+0xe1

02 00000079f5c9c3e0 00007ffe72dc877a clr!WaitForMultipleObjectsExSOTOLERANT+0x62

03 00000079f5c9c440 00007ffe72dc8591 clr!Thread::DoAppropriateWaitWorker+0x1d0

04 00000079f5c9c540 00007ffe72dc883d clr!Thread::DoAppropriateWait+0x7d

05 00000079f5c9c5c0 00007ffe72de3ed6 clr!CLREventBase::WaitEx+0xc0

06 00000079f5c9c650 00007ffe72de3dea clr!AwareLock::EnterEpilogHelper+0xc6

07 00000079f5c9c710 00007ffe72de4521 clr!AwareLock::EnterEpilog+0x62

08 00000079f5c9c770 00007ffe72de4293 clr!AwareLock::Contention+0x1e3

09 00000079f5c9c830 00007ffe147076b1 clr!JITutil_MonContention+0xaf

0a 00000079f5c9c9c0 00007ffe1470a605 umbraco!umbraco.content.CheckDiskCacheForUpdate()+0xa1

0b 00000079f5c9ca50 00007ffe1470ab40 umbraco!umbraco.content.CheckXmlContentPopulation()+0x75

0c 00000079f5c9cac0 00007ffe16fdecc4 umbraco!umbraco.content.get_XmlContentInternal()+0x10

0d 00000079f5c9caf0 00007ffe16fde7bc umbraco!umbraco.content.UpdateDocumentCache(umbraco.cms.businesslogic.web.Document)+0x104

0e 00000079f5c9cb70 00007ffe16e8f617 umbraco!Umbraco.Web.Cache.PageCacheRefresher.Refresh(Umbraco.Core.Models.IContent)+0x8c

0f 00000079f5c9cbe0 00007ffe16e8f1b8 UmbracoCore!Umbraco.Core.Sync.DefaultServerMessenger.InvokeMethodOnRefresherInstance[[System._Canon, mscorlib]](umbraco.interfaces.ICacheRefresher, Umbraco.Core.Sync.MessageType, System.Func2<System.__Canon,System.Object>, System.Collections.Generic.IEnumerable1

10 00000079f5c9cc80 00007ffe72daf0b7 Umbraco_Core!Umbraco.Core.Sync.DefaultServerMessenger.PerformRefresh[[System.Canon, mscorlib]](System.Collections.Generic.IEnumerable1<Umbraco.Core.Sync.IServerAddress>, umbraco.interfaces.ICacheRefresher, System.Func2Canon[])+0xf8

11 00000079f5c9ccf0 00007ffe16e8f095 clr!InstantiatingMethodStubWorker+0xc7

12 00000079f5c9cdb0 00007ffe16fde70f umbraco!Umbraco.Web.Cache.DistributedCache.Refresh[[System.Canon, mscorlib]](System.Guid, System.Func`2Canon[])+0x175

13 00000079f5c9ce60 00007ffe13665f91 umbraco!Umbraco.Web.Cache.DistributedCacheExtensions.RefreshPageCache(Umbraco.Web.Cache.DistributedCache, Umbraco.Core.Models.IContent[])+0xbf

14 00000079f5c9ced0 00007ffe16fdc0e5 CLRStub[StubLinkStub]@7ffe13665f91

15 00000079f5c9cf90 00007ffe17c1d199 Umbraco_Core!Umbraco.Core.Services.ContentService.SaveAndPublishDo(Umbraco.Core.Models.IContent, Int32, Boolean)+0x485

16 00000079f5c9d0d0 00007ffe17c1d453 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

17 00000079f5c9d1a0 00007ffe17c1d453 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

18 00000079f5c9d270 00007ffe17c1cbc5 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

19 00000079f5c9d340 00007ffe17c1c427 Umbraco_Core!Umbraco.Core.Services.ContentService.Move(Umbraco.Core.Models.IContent, Int32, Int32)+0x155

1a 00000079f5c9d3e0 00007ffe1d0ddf0e umbraco!Umbraco.Web.Editors.ContentController.PostMove(Umbraco.Web.Models.ContentEditing.MoveOrCopy)+0x57

1b 00000079f5c9d430 00007ffe16197835 DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[])+0x5e

1c 00000079f5c9d470 00007ffe161977ff SystemWebHttp!System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>cDisplayClass13.c(System.Object, System.Object[])+0x15

1d 00000079f5c9d4b0 00007ffe161969d8 SystemWebHttp!System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[])+0xf

1e 00000079f5c9d4e0 00007ffe16196a89 SystemWebHttp!System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1

1f 00000079f5c9d530 00007ffe161969d8 SystemWebHttp!System.Web.Http.Controllers.ApiControllerActionInvoker+<>cDisplayClass3.0()+0x29

20 00000079f5c9d580 00007ffe161968b8 SystemWebHttp!System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1

21 00000079f5c9d5d0 00007ffe16196803 SystemWebHttp!System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken)+0x98

22 00000079f5c9d620 00007ffe161964d1 SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass3.2()+0x33

23 00000079f5c9d650 00007ffe161963eb SystemWebHttp!System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1

24 00000079f5c9d6e0 00007ffe161964d1 SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass14+<>cDisplayClass16.

25 00000079f5c9d710 00007ffe161963eb SystemWebHttp!System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1

26 00000079f5c9d7a0 00007ffe161964d1 SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass14+<>cDisplayClass16.

27 00000079f5c9d7d0 00007ffe161963eb SystemWebHttp!System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1

28 00000079f5c9d860 00007ffe16196305 SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass14+<>cDisplayClass16.

29 00000079f5c9d890 00007ffe1619621f SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass3.1()+0xc5

2a 00000079f5c9d8e0 00007ffe16196134 SystemWebHttp!System.Threading.Tasks.TaskHelpersExtensions+<>cDisplayClass32`1[[System.Canon, mscorlib]].

2b 00000079f5c9d910 00007ffe152dbfe4 SystemWebHttp!System.Threading.Tasks.TaskHelpersExtensions.ThenImpl[[System.Canon, mscorlib],[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, System.Web.Http]](System.Canon, System.Func2<System.__Canon,System.Threading.Tasks.Task1

2c 00000079f5c9d980 00007ffe161b542e SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClass3.0()+0x74

2d 00000079f5c9d9e0 00007ffe161b539b SystemWebHttp!System.Web.Http.Filters.AuthorizationFilterAttribute.System.Web.Http.Filters.IAuthorizationFilter.ExecuteAuthorizationFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1

2e 00000079f5c9da40 00007ffe161b542e SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClasse+<>cDisplayClass10.

2f 00000079f5c9da70 00007ffe161b539b SystemWebHttp!System.Web.Http.Filters.AuthorizationFilterAttribute.System.Web.Http.Filters.IAuthorizationFilter.ExecuteAuthorizationFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1

30 00000079f5c9dad0 00007ffe152d731f SystemWebHttp!System.Web.Http.ApiController+<>cDisplayClasse+<>cDisplayClass10.

31 00000079f5c9db00 00007ffe152d26c6 SystemWebHttp!System.Web.Http.ApiController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, System.Threading.CancellationToken)+0x1cf

* WARNING: Unable to verify checksum for System.Net.Http.ni.dll

32 00000079f5c9db70 00007ffe73cfd4af SystemWebHttp!System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+0x76

33 00000079f5c9dc00 00007ffe152d222d SystemNetHttp_ni!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+0x4f

34 00000079f5c9dc50 00007ffe152d131f SystemWebHttp!System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+0x15d

35 00000079f5c9dcb0 00007ffe73cfd4af SystemWebHttp!System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+0xff

36 00000079f5c9dd00 00007ffe152cfaef SystemNetHttp_ni!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+0x4f

37 00000079f5c9dd50 00007ffe6d3160f2 SystemWebHttp_WebHost!System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+0x11f

38 00000079f5c9ddd0 00007ffe6d2dd510 SystemWebni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x1e2

39 00000079f5c9deb0 00007ffe6d2fabda SystemWebni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x120

3a 00000079f5c9df50 00007ffe6d2dd6a3 SystemWebni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x92a

3b 00000079f5c9e0a0 00007ffe6d2d75de SystemWebni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x83

3c 00000079f5c9e0f0 00007ffe6d2e0561 SystemWebni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x28e

3d 00000079f5c9e190 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x5c1

3e 00000079f5c9e3a0 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

3f 00000079f5c9e3f0 00007ffe72d0b42e SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

40 00000079f5c9e480 00007ffe6b2d1d6e clr!UMThunkStub+0x6e

41 00000079f5c9e510 00007ffe6b2d1cfb webengine4!W3MGDHANDLER::ProcessNotification+0x78

42 00000079f5c9e540 00007ffe6b2d1bfe webengine4!W3MGDHANDLER::DoWork+0x34f

43 00000079f5c9e5a0 00007ffe6b2d27a9 webengine4!RequestDoWork+0x34e

44 00000079f5c9e640 00007ffe6b3e19fe webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+0x21

45 00000079f5c9e680 00007ffe6b3e2908 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x224

46 00000079f5c9e6c0 00007ffe6b3e19a6 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x198

47 00000079f5c9e7d0 00007ffe6b3e79c4 iiscore!NOTIFICATION_CONTEXT::CallModules+0x36

48 00000079f5c9e830 00007ffe6b3e1e27 iiscore!NOTIFICATION_MAIN::DoWork+0x4b4

49 00000079f5c9eb40 00007ffe6b2d2734 iiscore!W3CONTEXTBASE::IndicateCompletion+0x97

4a 00000079f5c9ebb0 00007ffe6d3895eb webengine4!MgdIndicateCompletion+0x60

4b 00000079f5c9ebe0 00007ffe6d2e074f SystemWebni!DomainNeutralILStubClass.ILSTUBPInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+0x7b

4c 00000079f5c9ecb0 00007ffe6d2dff92 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x7af

4d 00000079f5c9eec0 00007ffe6da35dc1 SystemWebni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x22

4e 00000079f5c9ef10 00007ffe72d0b683 SystemWebni!DomainNeutralILStubClass.ILSTUBReversePInvoke(Int64, Int64, Int64, Int32)+0x51

4f 00000079f5c9efa0 00007ffe72d703ba clr!UM2MThunk_WrapperHelper+0x43

50 00000079f5c9efe0 00007ffe72d690ef clr!UM2MThunk_Wrapper+0x5a

51 00000079f5c9f030 00007ffe72d70335 clr!Thread::DoADCallBack+0x139

52 00000079f5c9f1e0 00007ffe72d0b62d clr!UM2MDoADCallBack+0x91

53 00000079f5c9f260 00007ffe6b2d1d6e clr!UMThunkStub+0x26d

54 00000079f5c9f2f0 00007ffe6b2d2276 webengine4!W3MGDHANDLER::ProcessNotification+0x78

55 00000079f5c9f320 00007ffe72d68b42 webengine4!ProcessNotificationCallback+0x42

56 00000079f5c9f350 00007ffe72d2f046 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x122

57 00000079f5c9f3f0 00007ffe72d2ef3a clr!ThreadpoolMgr::ExecuteWorkRequest+0x46

58 00000079f5c9f420 00007ffe72defcb6 clr!ThreadpoolMgr::WorkerThreadStart+0xf4

59 00000079f5c9f4b0 00007ffe7ae113d2 clr!Thread::intermediateThreadProc+0x7d

5a 00000079f5c9fc70 00007ffe7c925454 kernel32!BaseThreadInitThunk+0x22

5b 00000079f5c9fca0 0000000000000000 ntdll!RtlUserThreadStart+0x34

Looks like both are stuck in the same method “umbraco.content.CheckDiskCacheForUpdate()”. Looking at the code of this method, I see the following.

private void CheckDiskCacheForUpdate()

{

if (!Umbraco.Core.Configuration.UmbracoConfig.getFor().UmbracoSettings().getContent().get_XmlCacheEnabled())

{

return;

}

bool boolean1 = false;

object object1;

try

{

System.Threading.Monitor.Enter((object1 = umbraco.content.TimestampSyncLock), ref boolean1); --------------------------------------------------------1

if (this._lastDiskCacheCheckTime > System.DateTime.UtcNow.AddSeconds(-1)) goto lab1;

if (this._lastDiskCacheCheckTime > System.DateTime.UtcNow.AddSeconds(-1))

{

}

else

{

  bool boolean2;

lab1:

  this._lastDiskCacheCheckTime = System.DateTime.UtcNow;

  boolean2 = false;

  object object2;

  try

  {

    System.Threading.Monitor.Enter((object2 = umbraco.content.XmlContentInternalSyncLock), ref boolean2);-----------------------------------------------2

    if (this.GetCacheFileUpdateTime() <= this._lastDiskCacheReadTime)

    {

      goto lab2;

    }

    this._xmlContent = null;

  }

  finally

  {

    if (boolean2)

    {

      System.Threading.Monitor.Exit(object2);

    }

  }

}

}

finally

{

if (boolean1)

{

  System.Threading.Monitor.Exit(object1);

}

}

lab2:

return;

}

To avoid confusion, Let me just summarize information that we are aware of till this moment.

We have many threads waiting for a lock to be released. However there are 2 locks present held by thread 39 and thread 43.

The call stack of thread 39 and 43 reveals that they are in turn waiting for some other lock to be released which needs to be figured out.

Let us continue from here now.

Digging further into the dumps, I figured out that thread 43 is stuck in position marked 1 where “object1” is waiting to enter into “TimestampSyncLock”.

Thread 39 has acquired lock for “object1” but is stuck in position marked 2 where “object2” is waiting to enter into “XmlContentInternalSyncLock” in the above code.

So basically thread 43 is also waiting for thread 39 to release its lock held under “object1”.

Let us revisit the call stack of thread 43 once again.

0:043> kL

Child-SP RetAddr Call Site

00 00000079f5c9c0f8 00007ffe79dd13ed ntdll!ZwWaitForMultipleObjects+0xa

01 00000079f5c9c100 00007ffe72dc8976 KERNELBASE!WaitForMultipleObjectsEx+0xe1

02 00000079f5c9c3e0 00007ffe72dc877a clr!WaitForMultipleObjectsExSOTOLERANT+0x62

03 00000079f5c9c440 00007ffe72dc8591 clr!Thread::DoAppropriateWaitWorker+0x1d0

04 00000079f5c9c540 00007ffe72dc883d clr!Thread::DoAppropriateWait+0x7d

05 00000079f5c9c5c0 00007ffe72de3ed6 clr!CLREventBase::WaitEx+0xc0

06 00000079f5c9c650 00007ffe72de3dea clr!AwareLock::EnterEpilogHelper+0xc6

07 00000079f5c9c710 00007ffe72de4521 clr!AwareLock::EnterEpilog+0x62

08 00000079f5c9c770 00007ffe72de4293 clr!AwareLock::Contention+0x1e3

09 00000079f5c9c830 00007ffe147076b1 clr!JITutil_MonContention+0xaf

0a 00000079f5c9c9c0 00007ffe1470a605 umbraco!umbraco.content.CheckDiskCacheForUpdate()+0xa1

0b 00000079f5c9ca50 00007ffe1470ab40 umbraco!umbraco.content.CheckXmlContentPopulation()+0x75

0c 00000079f5c9cac0 00007ffe16fdecc4 umbraco!umbraco.content.get_XmlContentInternal()+0x10

0d 00000079f5c9caf0 00007ffe16fde7bc umbraco!umbraco.content.UpdateDocumentCache(umbraco.cms.businesslogic.web.Document)+0x104

0e 00000079f5c9cb70 00007ffe16e8f617 umbraco!Umbraco.Web.Cache.PageCacheRefresher.Refresh(Umbraco.Core.Models.IContent)+0x8c

0f 00000079f5c9cbe0 00007ffe16e8f1b8 UmbracoCore!Umbraco.Core.Sync.DefaultServerMessenger.InvokeMethodOnRefresherInstance[[System._Canon, mscorlib]](umbraco.interfaces.ICacheRefresher, Umbraco.Core.Sync.MessageType, System.Func2<System.__Canon,System.Object>, System.Collections.Generic.IEnumerable1

10 00000079f5c9cc80 00007ffe72daf0b7 Umbraco_Core!Umbraco.Core.Sync.DefaultServerMessenger.PerformRefresh[[System.Canon, mscorlib]](System.Collections.Generic.IEnumerable1<Umbraco.Core.Sync.IServerAddress>, umbraco.interfaces.ICacheRefresher, System.Func2Canon[])+0xf8

11 00000079f5c9ccf0 00007ffe16e8f095 clr!InstantiatingMethodStubWorker+0xc7

12 00000079f5c9cdb0 00007ffe16fde70f umbraco!Umbraco.Web.Cache.DistributedCache.Refresh[[System.Canon, mscorlib]](System.Guid, System.Func`2Canon[])+0x175

13 00000079f5c9ce60 00007ffe13665f91 umbraco!Umbraco.Web.Cache.DistributedCacheExtensions.RefreshPageCache(Umbraco.Web.Cache.DistributedCache, Umbraco.Core.Models.IContent[])+0xbf

14 00000079f5c9ced0 00007ffe16fdc0e5 CLRStub[StubLinkStub]@7ffe13665f91

15 00000079f5c9cf90 00007ffe17c1d199 Umbraco_Core!Umbraco.Core.Services.ContentService.SaveAndPublishDo(Umbraco.Core.Models.IContent, Int32, Boolean)+0x485

16 00000079f5c9d0d0 00007ffe17c1d453 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

17 00000079f5c9d1a0 00007ffe17c1d453 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

18 00000079f5c9d270 00007ffe17c1cbc5 Umbraco_Core!Umbraco.Core.Services.ContentService.PerformMove(Umbraco.Core.Models.IContent, Int32, Int32, System.Collections.Generic.ICollection1<Umbraco.Core.Events.MoveEventInfo1

19 00000079f5c9d340 00007ffe17c1c427 Umbraco_Core!Umbraco.Core.Services.ContentService.Move(Umbraco.Core.Models.IContent, Int32, Int32)+0x155

1a 00000079f5c9d3e0 00007ffe1d0ddf0e umbraco!Umbraco.Web.Editors.ContentController.PostMove(Umbraco.Web.Models.ContentEditing.MoveOrCopy)+0x57

1b 00000079f5c9d430 00007ffe16197835 DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[])+0x5e

1c 00000079f5c9d470 00007ffe161977ff SystemWebHttp!System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>cDisplayClass13.c(System.Object, System.Object[])+0x15

1d 00000079f5c9d4b0 00007ffe161969d8 SystemWebHttp!System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[])+0xf

1e 00000079f5c9d4e0 00007ffe16196a89 SystemWebHttp!System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1

.

.

Looking at the code of “umbraco.content.UpdateDocumentCache”, I see the following.

public virtual void UpdateDocumentCache(umbraco.cms.businesslogic.web.Document d) {

umbraco.cms.businesslogic.DocumentCacheEventArgs args1 = new umbraco.cms.businesslogic.DocumentCacheEventArgs();

this.FireBeforeUpdateDocumentCache(d, args1);

if (!args1.Cancel)

{

bool boolean1 = false;

object object1;

try

{

  System.Threading.Monitor.Enter((object1 = umbraco.content.XmlContentInternalSyncLock), ref boolean1);

  System.Xml.XmlDocument document1 = Umbraco.Core.Configuration.UmbracoConfig.get_For().UmbracoSettings().get_Content().get_CloneXmlContent() ? umbraco.content.CloneXmlDoc(this.XmlContentInternal) : this.XmlContentInternal;

  document1 = umbraco.content.PublishNodeDo(d, document1, true);

  this.XmlContentInternal = document1;

  this.ClearContextCache();

}

finally

{

  if (boolean1)

  {

    System.Threading.Monitor.Exit(object1);

  }

}

string string1 = string.Format("{0}{1}_", "contentItem", d.get_Id());

Umbraco.Core.ApplicationContext.get_Current().get_ApplicationCache().ClearCacheByKeySearch(string1);

this.FireAfterUpdateDocumentCache(d, args1);

}

return;

}

So we have thread 43 which has acquired lock to “XmlContentInternalSyncLock” which thread 39 is waiting on.

We are running into a deadlock situation here with thread 39 waiting on thread 43 to release its lock and thread 43 waiting on thread 39 to release its lock.

Request Umbraco team to please look into the same and guide for resolving same.