Application pool recycle when i publish content !!
I am using version 7.3.6 at the moment and i got an issue where when i publish content my site goes down and starts up again. Everything gets persisted in the xml-file and so on ...
This problem occurs in the production environment and it could be something with load balancing.
This just started happening to us a couple times a day after a NuGet upgrade to 7.5.3 from 7.4.3. We host our own Windows 2008 R2 server, so I'm not finding a hotfix that addresses it. Did something different happen in 7.5.x that's causing these? Quick dump of the log file:
Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: ConfigurationChange
_shutDownMessage=Overwhelming Change Notification in
HostingEnvironment initiated shutdown
CONFIG change
Overwhelming Change Notification in D:\inetpub\devclean
CONFIG change
Change Notification for critical directories.
Overwhelming Change Notification in bin
Change Notification for critical directories.
Overwhelming Change Notification in App_LocalResources
CONFIG change
CONFIG change
CONFIG change
CONFIG change
Change Notification for critical directories.
Overwhelming Change Notification in App_WebReferences
CONFIG change
CONFIG change
Change Notification for critical directories.
Overwhelming Change Notification in App_LocalResources
Change Notification for critical directories.
Overwhelming Change Notification in App_Code
CONFIG change
Change Notification for critical directories.
Overwhelming Change Notification in App_LocalResources
CONFIG change
Change Notification for critical directories.
Overwhelming Change Notification in App_GlobalResources
Change Notification for critical directories.
Overwhelming Change Notification in App_Browsers
CONFIG change
CONFIG change
CONFIG change
CONFIG change
CONFIG change
I've removed the "fcnMode=single" to enable the default mode NotSet ("File change notification is not set, so the application creates an object that monitors each subdirectory. ") which seems to have removed the problem.
Ok, that's not an issue (was confusing myself about file size vs fileinfo size), It does seem very odd that if you change the fcnMode to NotSet (default) that this problem goes away. Not sure if you've read this which might help shed some light as to what the issue you are seeing is: http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/
Actually I'm also a bit baffled as to exactly why this solved our problem as I can't really figure out the underlying cause as we are a bit tied down as it's a third-party hosting company (which seems to run multiple application pools for the site).
But our first step during this incident was to create a copy of the website on a single instance environment (running Windows Server 2012 R2) where we couldn't replicate the same behavior and then went searching for an alternative solution other than moving the site to this new environment and stumbled upon this a possible connection to the fcnMode setting.
This problem only occurred after upgrading from 7.3.0 to 7.5.3 which we did not to long ago due to another problem with conflicting index locks for the internal member index that only occurred for this hosting provider.
We too are experiencing the issue of app pool recycles after content being published. 2012R2 fully patched... For the time being we have the fcnmode set to disabled.
Content being published has to wait until the application pool is recycled which is a bit of a pain.
Have you installed the optional update KB3052480? All information about this can be found in the following issue's description: http://issues.umbraco.org/issue/U4-6338
Yes, KB3052480 was installed. Thanks for the link, although I have read through that along with every other post I could find regarding the issue with no resolution at this point.
You should check if that patch is working on your server by setting up a website with the website restarter code that is available on that issue http://issues.umbraco.org/issueMobile/U4-6338
Make sure you don't have anti virus running.
Otherwise the only way to get to the bottom of this is to provide specific steps to reproduce the problem on a brand new server install
Setup the web app restarter site & no restarts are being detected. One thing I noticed though is that the site restart app only makes changes in the App_Data folder... When I examine our logs I see several entries regarding folders such as...
Ok thanks for reporting, at least you can rule that out. The listing of the folders is a 'red herring'. Are you running virtual directories, hosting off of UNC paths, or anything like that?
Within our site we do have a virtual directory setup for hosting some images, etc. The contents of which haven't been modified in a couple of months though.
It could be the issue. I have a ton of reading for you if you want to look at all the different places where file changes are done in .Net and what makes it work is that things work differently based on if you have Virtual Dirs, UNC paths, etc... In some cases it doesn't really matter if anything has change in those folders, the problem that you are seeing is due directly to the fact that the ASP.Net FileChangesMonitor's buffer has overflown.
So as this post has gotten very long, the first option you can try is: Change FCNMode to Single (which is what we ship with since this seems safer), or you can set it back to default and see if that makes any difference, or you can disable it - which will probably work but then you have other problems since the only way to restart your app is by changing the web.config
Otherwise, you can see if changing your Virtual Dir to not be a Virtual Dir helps. Both IIS and ASP.Net have file change watchers (for some ridiculous reason) and it turns out that even with FCN disabled, the IIS file change watchers still apparently watch files/folders on a Vdir or network share.
The other reason why buffers can overflow more easily is because you may have very long file paths, the shorter the full file paths are the less memory they consume for the watchers.
Make sure you have no anti-virus running, or backup software scanning any of your files since this could also trigger a ton of changes. The other thing that can trigger a ton of changes is ACL updates of the NTFS file system, since these changes also trigger the File watchers!
There's lots to read about this and lots of links I've collected (see the bottom): http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/ I'd love to know how this can be solved on all hosting setups so people don't have to worry about this but it seems that this issue has been going on for years.
Thanks again Shannon. So, I setup a test site on our production box with the virtual directory removed & set the fcnMode to single. The errors still occurred with the application restarts.
We have a media folder with over 15k folders/files in it,, I'm guessing that could potentially break the 64kb limit of using the fcn single mode... And if we were to go single mode generating a 4kb watcher for each one could cause some strain...
However, the strange thing is that when we were at version 6.2.6 this never happened and we're even using the exact same IIS site setup & content structure so it can't be any type of predefined configuration setting.
So this sounds like you are able to reproduce the issue on a different environment --> This is amazing news if you can because what I would absolutely love is to be able to reproduce this locally (VM or otherwise) so I can debug this and try to get to the bottom of this. If you have steps to reproduce and maybe even a solution that you can share, you can email me at shannon at umbraco dot com.
6.2.6 didn't have a lot of the features that 7.x has. One thing you can try is to disable Image Processor, I have a feeling that the number of folders it creates for it's cache in /App_Data/cache has a direct affect on FCN as well and it's something that I'm looking in to. This is something you could also play around with to see if that has any affect - perhaps you could also try storing the Image Processor cache outside of the web root, if you do that then no file monitors will be created for each of those folders.
Our instance is a recent upgrade (via Nuget) from 7.4.3 to 7.5.4. We do quite a lot of publishing during the day so disabling fcn may prove problematic.
For just content publishing (that is, not making changes to Views or Partials), leaving fcnMode disabled should work well (or at least it has in our recent NuGet upgrade to 7.5.4). We now only set that flag back to "Single" when we're working on Views (templates) or Partials, otherwise those changes don't show up until you touch web.config.
Since keeping fcnMode disabled seems the safest route, is there any way to edit views/templates/partials and have the reflect those changes without having to touch web.config or temporarily set that flag back to Single?
Application pool recycle when i publish content !!
I am using version 7.3.6 at the moment and i got an issue where when i publish content my site goes down and starts up again. Everything gets persisted in the xml-file and so on ...
This problem occurs in the production environment and it could be something with load balancing.
I tried to follow: https://our.umbraco.org/documentation/getting-started/setup/server-setup/load-balancing/flexible but it doesnt matter for me. Still got the problem above.
The log says:
2016-02-24 21:39:57,880 [P5704/D13/T35] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: ConfigurationChange
shutDownMessage=CONFIG change HostingEnvironment initiated shutdown Overwhelming Change Notification in Overwhelming Change Notification in XXXXXXXXXXXX CONFIG change Change Notification for critical directories. Overwhelming Change Notification in bin CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in AppWebReferences CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in AppGlobalResources Change Notification for critical directories. Overwhelming Change Notification in AppBrowsers
I had the same issue on one of our production servers running on Windows Server 2012 R2. It was related to this issue: https://support.microsoft.com/en-us/kb/3052480.
This just started happening to us a couple times a day after a NuGet upgrade to 7.5.3 from 7.4.3. We host our own Windows 2008 R2 server, so I'm not finding a hotfix that addresses it. Did something different happen in 7.5.x that's causing these? Quick dump of the log file:
Comment author was deleted
This may have a connection to the change to the fcnMode setting in the web.config file which was changed in:
https://github.com/umbraco/Umbraco-CMS/commit/5b9cbae860609eb4ea7d5bc9338003aa5351c2f5#diff-4891093f635b1f7e7df6e06a14702856
I've removed the "fcnMode=single" to enable the default mode NotSet ("File change notification is not set, so the application creates an object that monitors each subdirectory. ") which seems to have removed the problem.
The initial change was due to issue http://issues.umbraco.org/issue/U4-7712 .
How large is your umbraco.config file?
Comment author was deleted
It's about 360 KB
Ok, that's not an issue (was confusing myself about file size vs fileinfo size), It does seem very odd that if you change the fcnMode to NotSet (default) that this problem goes away. Not sure if you've read this which might help shed some light as to what the issue you are seeing is: http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/
Comment author was deleted
Actually I'm also a bit baffled as to exactly why this solved our problem as I can't really figure out the underlying cause as we are a bit tied down as it's a third-party hosting company (which seems to run multiple application pools for the site).
But our first step during this incident was to create a copy of the website on a single instance environment (running Windows Server 2012 R2) where we couldn't replicate the same behavior and then went searching for an alternative solution other than moving the site to this new environment and stumbled upon this a possible connection to the fcnMode setting.
This problem only occurred after upgrading from 7.3.0 to 7.5.3 which we did not to long ago due to another problem with conflicting index locks for the internal member index that only occurred for this hosting provider.
We too are experiencing the issue of app pool recycles after content being published. 2012R2 fully patched... For the time being we have the fcnmode set to disabled.
Content being published has to wait until the application pool is recycled which is a bit of a pain.
Any idea on how to resolve this?
Thanks.
Have you installed the optional update KB3052480? All information about this can be found in the following issue's description: http://issues.umbraco.org/issue/U4-6338
Yes, KB3052480 was installed. Thanks for the link, although I have read through that along with every other post I could find regarding the issue with no resolution at this point.
You should check if that patch is working on your server by setting up a website with the website restarter code that is available on that issue http://issues.umbraco.org/issueMobile/U4-6338
Make sure you don't have anti virus running.
Otherwise the only way to get to the bottom of this is to provide specific steps to reproduce the problem on a brand new server install
Thanks Shannon.
Setup the web app restarter site & no restarts are being detected. One thing I noticed though is that the site restart app only makes changes in the App_Data folder... When I examine our logs I see several entries regarding folders such as...
Overwhelming Change Notification in....
and no entries regarding the App_Data folder.
Ok thanks for reporting, at least you can rule that out. The listing of the folders is a 'red herring'. Are you running virtual directories, hosting off of UNC paths, or anything like that?
Within our site we do have a virtual directory setup for hosting some images, etc. The contents of which haven't been modified in a couple of months though.
It could be the issue. I have a ton of reading for you if you want to look at all the different places where file changes are done in .Net and what makes it work is that things work differently based on if you have Virtual Dirs, UNC paths, etc... In some cases it doesn't really matter if anything has change in those folders, the problem that you are seeing is due directly to the fact that the ASP.Net
FileChangesMonitor's buffer has overflown.So as this post has gotten very long, the first option you can try is: Change FCNMode to Single (which is what we ship with since this seems safer), or you can set it back to default and see if that makes any difference, or you can disable it - which will probably work but then you have other problems since the only way to restart your app is by changing the web.config
Otherwise, you can see if changing your Virtual Dir to not be a Virtual Dir helps. Both IIS and ASP.Net have file change watchers (for some ridiculous reason) and it turns out that even with FCN disabled, the IIS file change watchers still apparently watch files/folders on a Vdir or network share.
The other reason why buffers can overflow more easily is because you may have very long file paths, the shorter the full file paths are the less memory they consume for the watchers.
Make sure you have no anti-virus running, or backup software scanning any of your files since this could also trigger a ton of changes. The other thing that can trigger a ton of changes is ACL updates of the NTFS file system, since these changes also trigger the File watchers!
There's lots to read about this and lots of links I've collected (see the bottom): http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/ I'd love to know how this can be solved on all hosting setups so people don't have to worry about this but it seems that this issue has been going on for years.
If you really want to get your hands dirty here's the source of where this problem comes from: https://github.com/Microsoft/referencesource/blob/master/System.Web/FileChangesMonitor.cs
Thanks again Shannon. So, I setup a test site on our production box with the virtual directory removed & set the fcnMode to single. The errors still occurred with the application restarts.
We have a media folder with over 15k folders/files in it,, I'm guessing that could potentially break the 64kb limit of using the fcn single mode... And if we were to go single mode generating a 4kb watcher for each one could cause some strain...
However, the strange thing is that when we were at version 6.2.6 this never happened and we're even using the exact same IIS site setup & content structure so it can't be any type of predefined configuration setting.
Hi Paul!
So this sounds like you are able to reproduce the issue on a different environment --> This is amazing news if you can because what I would absolutely love is to be able to reproduce this locally (VM or otherwise) so I can debug this and try to get to the bottom of this. If you have steps to reproduce and maybe even a solution that you can share, you can email me at shannon at umbraco dot com.
6.2.6 didn't have a lot of the features that 7.x has. One thing you can try is to disable Image Processor, I have a feeling that the number of folders it creates for it's cache in /App_Data/cache has a direct affect on FCN as well and it's something that I'm looking in to. This is something you could also play around with to see if that has any affect - perhaps you could also try storing the Image Processor cache outside of the web root, if you do that then no file monitors will be created for each of those folders.
Hi Paul, any feedback on this comment? would be great if you could send me a project to replicate this locally on
Is there a resolution for this issue? We have started to see the same problem. We already have the Microsoft hot-fixes applied.
Not on our end. We currently have fcnMode disabled & have resorted to doing deployments during off-peak times to avoid any downtime.
Curious Steve, was your instance an upgrade or a fresh install?
This could go a long way to helping resolve the issue: https://github.com/JimBobSquarePants/ImageProcessor/issues/518
I also plan on putting together a Nuget package that you can install to show you all of the file monitors in an application
Our instance is a recent upgrade (via Nuget) from 7.4.3 to 7.5.4. We do quite a lot of publishing during the day so disabling fcn may prove problematic.
For just content publishing (that is, not making changes to Views or Partials), leaving fcnMode disabled should work well (or at least it has in our recent NuGet upgrade to 7.5.4). We now only set that flag back to "Single" when we're working on Views (templates) or Partials, otherwise those changes don't show up until you touch web.config.
Thanks Eric. We have just switched to disabled for now.
And thank you Shannon. We are using ImageProcessor with a huge amount of images so I am hoping that your proposed solution will solve the problem.
Since keeping fcnMode disabled seems the safest route, is there any way to edit views/templates/partials and have the reflect those changes without having to touch web.config or temporarily set that flag back to Single?
is working on a reply...