We've just deployed a new site (look for the #umbLaunch announcement later today!) on our usual load-balanced setup that has until now been fine. This has been the first site on 4.11.4 (and indeed, probably since the significant changes in 4.10+).
At app start, or sometimes a short time later, we're getting a YSOD, with the following exception logged;
Exception message: Exception has been thrown by the target of an invocation. at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck) at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache) at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache) at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly) at System.Activator.CreateInstance(Type type, Boolean nonPublic) at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes) at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes) at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture) at System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args) at System.Web.HttpRuntime.CreateNonPublicInstance(Type type) at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) at System.Web.HttpApplicationFactory.GetPipelineApplicationInstance(IntPtr appContext, HttpContext context) at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) Could not load file or assembly 'Umbraco.Core, Version=1.0.4780.19110, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Access is denied. at Umbraco.Web.UmbracoApplication..ctor() at ASP.global_asax..ctor()
Normally, I'd suspect something in my LB setup (shared file storage, with local App_Data folder for each server for Examine indexes, umbraco.config, etc), but I trust this normally works well, which leads me to suspect something is different about 4.11 in this respect. Recycling the worker process sometimes works, but we generally need to flush the 'Temporary ASP.NET Files' folder to get things running again.
We're currently running on a single-host with local files, which so far is OK (crosses fingers...)
I've also seen problems caused by log4net and NewRelic (http://issues.umbraco.org/issue/U4-1471) on this same site, in case that might be related.
Issues with Global.asax file? - I've had a few of these in site upgrades - but as it's a fresh install - might not be the case - delete any compiled versions or custom versions.
How have you setup shared files? We run a load balanced environment - but all files are local updates are replicated across (We have one designated 'admin' server) - the latest site we have in the load balanced environment is 4.8 though (seems old now)
Access Denied does seem to suggest an issue with file locking through...
Ian; agreed, this is a new install, so unlikely to be global.asax related. I should also have added that this ran fine on a single-server staging site, which adds weight to it being a load-balance issue.
I'd be inclined to agree it looks like permissions, but we're running the same setup as a dozen or so other Umbraco sites; two IIS boxes accessing a UNC share for the web files, using specific credentials to access the UNC share (which has read-only rights to all except the media folder). On each server, /App_Data/TEMP is a virtual directory mapped to a local folder, writeable by the Application Pool identity user.
It I didn't already have other sites running like this (various versions, from approx 4.8 onwards), I wouldn't suspect that something different is happening with 4.11... just seems too co-incidental to me...
I've spent some time on this this morning, and by enabling Assembly Binding Logging have been able to determine that the worker process is wrongly trying to load Umbraco.Core.dll as the IIS APPPOOL user, not the credentials configured to access the UNC path.
Yet, for every other assembly, it uses the correct credentials, and also does so for Umbraco.Core.dll on another site (running v4.9) configured the same way.
I've now been able to replicate this on a clean website, configured in IIS and with vanilla (and unconfigured) 4.11.4 uploaded;
Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\Windows\SysWOW64\inetsrv\w3wp.exe
--- A detailed error log follows.
=== Pre-bind state information ===
LOG: User = IIS APPPOOL\test4114.web13.dhaus.com
LOG: DisplayName = Umbraco.Core, Version=1.0.4780.19110, Culture=neutral, PublicKeyToken=null
(Fully-specified)
LOG: Appbase = file://file-1/data/websites/test4114.web13.dhaus.com/
LOG: Initial PrivatePath = \\file-1\data\websites\test4114.web13.dhaus.com\bin
Calling assembly : umbraco, Version=1.0.4780.19111, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: \\file-1\data\websites\test4114.web13.dhaus.com\web.config
LOG: Using host configuration file: C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet.config
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/Temporary ASP.NET Files/root/3b6b9c41/1b5bcdba/Umbraco.Core.DLL.
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/Temporary ASP.NET Files/root/3b6b9c41/1b5bcdba/Umbraco.Core/Umbraco.Core.DLL.
LOG: Attempting download of new URL file://file-1/data/websites/test4114.web13.dhaus.com/bin/Umbraco.Core.DLL.
ERR: A fatal error occurred when retrieving next codebase for download (hr = 0x80070005).
OK, deploying 4.9.1 into exactly the same environment works, so whilst I'm not convinved this a bug per-se, I've logged it at http://issues.umbraco.org/issue/U4-1806
YSOD in 4.11.4 load-balanced new install
We've just deployed a new site (look for the #umbLaunch announcement later today!) on our usual load-balanced setup that has until now been fine. This has been the first site on 4.11.4 (and indeed, probably since the significant changes in 4.10+).
At app start, or sometimes a short time later, we're getting a YSOD, with the following exception logged;
Normally, I'd suspect something in my LB setup (shared file storage, with local App_Data folder for each server for Examine indexes, umbraco.config, etc), but I trust this normally works well, which leads me to suspect something is different about 4.11 in this respect. Recycling the worker process sometimes works, but we generally need to flush the 'Temporary ASP.NET Files' folder to get things running again.
We're currently running on a single-host with local files, which so far is OK (crosses fingers...)
I've also seen problems caused by log4net and NewRelic (http://issues.umbraco.org/issue/U4-1471) on this same site, in case that might be related.
Any ideas where to begin tracking this down?
Phil
Issues with Global.asax file? - I've had a few of these in site upgrades - but as it's a fresh install - might not be the case - delete any compiled versions or custom versions.
How have you setup shared files? We run a load balanced environment - but all files are local updates are replicated across (We have one designated 'admin' server) - the latest site we have in the load balanced environment is 4.8 though (seems old now)
Access Denied does seem to suggest an issue with file locking through...
Ian; agreed, this is a new install, so unlikely to be global.asax related. I should also have added that this ran fine on a single-server staging site, which adds weight to it being a load-balance issue.
I'd be inclined to agree it looks like permissions, but we're running the same setup as a dozen or so other Umbraco sites; two IIS boxes accessing a UNC share for the web files, using specific credentials to access the UNC share (which has read-only rights to all except the media folder). On each server, /App_Data/TEMP is a virtual directory mapped to a local folder, writeable by the Application Pool identity user.
It I didn't already have other sites running like this (various versions, from approx 4.8 onwards), I wouldn't suspect that something different is happening with 4.11... just seems too co-incidental to me...
I've spent some time on this this morning, and by enabling Assembly Binding Logging have been able to determine that the worker process is wrongly trying to load Umbraco.Core.dll as the IIS APPPOOL user, not the credentials configured to access the UNC path.
Yet, for every other assembly, it uses the correct credentials, and also does so for Umbraco.Core.dll on another site (running v4.9) configured the same way.
Bizarre, and hair-tearingly confusing...
I've now been able to replicate this on a clean website, configured in IIS and with vanilla (and unconfigured) 4.11.4 uploaded;
OK, deploying 4.9.1 into exactly the same environment works, so whilst I'm not convinved this a bug per-se, I've logged it at http://issues.umbraco.org/issue/U4-1806
is working on a reply...