I got a site, Umbraco 4.7.0, that has crashed three times now with the error:
[InvalidOperationException: Hashtable insert
failed. Load factor too high. The most common cause is multiple threads writing
to the Hashtable simultaneously.]
The error has been seen, all three times, when our customer turn on his computer in the morning. Meaning after several hours of non-activity.
The only thing to do to resolve the issue is to restart the site on the server.
Searched Google to find a solution and 1 is that there is a Lock() missing somewhere in Umbraco and the other is a Microsoft bug. I found http://support.microsoft.com/kb/2576011 requested the ptch from Microsoft and installed it. The error went away for about 14 days, but happend again yesterday morning.
Has there been build anything custom of any kind that may be the cause of this? I have not seen this bug before and after a quick glance at umbraco.codeplex.com does not seemt that anyone has reported something similarily.
Also do you know if the proper permissions for network service and iusr has been set on the root folder containing the Umbraco installation? It seems odd that it happens on and off at different times...so maybe it's related to permissions being reset or something like that perhaps?
This was done yesterday afternoon and this morning I got a new report of the issue happening again.
As I understand our programmers, I am in no way a .Net'er myself, the error indicates that the issue is happening before any custom code would be reached. In other words this points to a missing lock somewhere.
Got some new info from the umbracoLog table. When this happens, this error is preceding it:
At /prices?da=1324626844952 (Referred by: http://www.xxx.com/da/nyheder/2011/december-2011/pelsen-overhaler-osten/): System.ArgumentException: Item has already been added. Key in dictionary: '/prices' Key being added: '/prices' at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add) at System.Collections.Hashtable.Add(Object key, Object value) at umbraco.requestHandler..ctor(XmlDocument umbracoContent, String url) at umbraco.UmbracoDefault.Page_PreInit(Object sender, EventArgs e) at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) at System.EventHandler.Invoke(Object sender, EventArgs e) at System.Web.UI.Page.OnPreInit(EventArgs e) at System.Web.UI.Page.PerformPreInit() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
This should make it possible for the devs to pinpoint the error quite precisely. Right?
I've just taken a quick look at the constructor of the requestHandler class, and can spot at least two places that might cause this.
The requestHandler class has a static HashTable (_processedRequests) where the Add method is called from the instances of the requestHandler. One of the Add calls happens after a call to Contains, but if several threads are in the same area, my guess is that Contains can return true, and then a "pico" second after that, Add will fail because another thread just added the same key to the HashTable.
There's another call to Add, without a prior call to Contains, but I'm not sure how that part of the code is reached.
Anyway, under the given circumstances I would probably have used locking to make sure this exception wouldn't happen.
Memory leaks? This looks a lot more like writing to a static hash table from several threads without making sure you have exclusive access. But yes, the source I've been looking at (my previous post), is 4.7.1.1.
Hi Just to confirm this appears to be fixed in Umbraco 4.7.2:
we just had this issue on one of our internal sites; Reloading the AppPool within IIS has given us tempory restbite, but It does look like this is an issue in 4.7.0 -
The source code for 4.7.2 does now contain locking sections around all hashtable activity, so we are upgrading to resolve the underlying problem.
Hashtable insert failed
Hi
I got a site, Umbraco 4.7.0, that has crashed three times now with the error:
The error has been seen, all three times, when our customer turn on his computer in the morning. Meaning after several hours of non-activity.
The only thing to do to resolve the issue is to restart the site on the server.
Searched Google to find a solution and 1 is that there is a Lock() missing somewhere in Umbraco and the other is a Microsoft bug. I found http://support.microsoft.com/kb/2576011 requested the ptch from Microsoft and installed it. The error went away for about 14 days, but happend again yesterday morning.
Anybody got any ideas on how to debug this thing?
Hi UmbracoNewb
Has there been build anything custom of any kind that may be the cause of this? I have not seen this bug before and after a quick glance at umbraco.codeplex.com does not seemt that anyone has reported something similarily.
Also do you know if the proper permissions for network service and iusr has been set on the root folder containing the Umbraco installation? It seems odd that it happens on and off at different times...so maybe it's related to permissions being reset or something like that perhaps?
/Jan
Will investigate right away!
I checked the permissions and they were already set correctly, but I did it again just to make sure. I followed this guide: http://our.umbraco.org/wiki/install-and-setup/set-umbraco-folder-permissions-from-command-line and set the permissions according to the Windows 2008 R2 and IIS7 methode.
This was done yesterday afternoon and this morning I got a new report of the issue happening again.
As I understand our programmers, I am in no way a .Net'er myself, the error indicates that the issue is happening before any custom code would be reached. In other words this points to a missing lock somewhere.
Got some new info from the umbracoLog table. When this happens, this error is preceding it:
At /prices?da=1324626844952 (Referred by: http://www.xxx.com/da/nyheder/2011/december-2011/pelsen-overhaler-osten/): System.ArgumentException: Item has already been added. Key in dictionary: '/prices' Key being added: '/prices' at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add) at System.Collections.Hashtable.Add(Object key, Object value) at umbraco.requestHandler..ctor(XmlDocument umbracoContent, String url) at umbraco.UmbracoDefault.Page_PreInit(Object sender, EventArgs e) at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) at System.EventHandler.Invoke(Object sender, EventArgs e) at System.Web.UI.Page.OnPreInit(EventArgs e) at System.Web.UI.Page.PerformPreInit() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
This should make it possible for the devs to pinpoint the error quite precisely. Right?
I've just taken a quick look at the constructor of the requestHandler class, and can spot at least two places that might cause this.
The requestHandler class has a static HashTable (_processedRequests) where the Add method is called from the instances of the requestHandler. One of the Add calls happens after a call to Contains, but if several threads are in the same area, my guess is that Contains can return true, and then a "pico" second after that, Add will fail because another thread just added the same key to the HashTable.
There's another call to Add, without a prior call to Contains, but I'm not sure how that part of the code is reached.
Anyway, under the given circumstances I would probably have used locking to make sure this exception wouldn't happen.
Any insight from somebody on the core team?
Does this still hapen in Umbraco 4.7.1.1? Some memory leaks have been fixed.
Jeroen
Memory leaks? This looks a lot more like writing to a static hash table from several threads without making sure you have exclusive access. But yes, the source I've been looking at (my previous post), is 4.7.1.1.
Hi Just to confirm this appears to be fixed in Umbraco 4.7.2:
we just had this issue on one of our internal sites; Reloading the AppPool within IIS has given us tempory restbite, but It does look like this is an issue in 4.7.0 -
The source code for 4.7.2 does now contain locking sections around all hashtable activity, so we are upgrading to resolve the underlying problem.
is working on a reply...