I have code that looks like this (it's in an event handler for TransformingIndexValues against the External Index:
using (var umbracoContextReference = umbracoContextFactory.EnsureUmbracoContext())
{
var documentStub = umbracoContextReference.UmbracoContext.Content.GetById(id);
Except, the line setting documentStub is throwing an ObjectDisposedException. What's even more unusual is that as soon as I've called EnsureUmbracoContext() and I examine umbracoContextReference.UmbracoContext, the disposed property on it is true.
If I let the exception throw and look in the logs in Umbraco it is saying that the NuCache snapshot that it is trying to access is disposed of.
To give more context, I have a HangFire task running in the background to import/update a large quantity of content. After which I need to trigger an Examine Index rebuild which all appears to work. Except when the index rebuild starts this code is hit and the exceptions are thrown.
Does anyone have any ideas as to why triggering the rebuild via a HangFire job would cause this to happen even when I'm "ensuring an Umbraco context?" What's also odd about it is that it doesn't fail for every document it's trying to index, and the number it fails for is inconsistent.
The problem is that most Umbraco operations require a "Scope" which is an Umbraco thing. Umbraco tries it's best to ensure something called a "Transient" scope so you don't have to worry about this too often. It's quite easy to ensure that a Scope is created and used during Http requests because we know when it starts and ends but for background tasks this is more complex.
A Scope is required for many things in Umbraco, from controlling database transactions to how service level events are handled to how nucache manages it's snapshots (and more).
For background tasks we use something called a "CallContext" and in theory this could/should be unique per threading operation but this cannot be guaranteed. Since this is a .Net thing it is out of our control. We try to create a Transient Scope based on the CallContext because there is no HttpContext and in some cases, depending on the type of background task, the type of thread (thread pool vs standalone), if async is involved, etc... the same CallContext may be used between threads. This will mean that the same Transient Scope is used between threads == bad news. (In .Net Core the notion of CallContext doesn't exist and has a much better approach).
This is precisely what is happening here, some other thread is using the same Scope and as such is using the same nucache Snapshot and when it thinks it's CallContext ends it disposes the Scope which disposes the Snapshot, but since that same Scope instance is being used on your other background thread, it tries to access a disposed object.
Generally, anytime you are doing this:
using (var umbracoContextReference = umbracoContextFactory.EnsureUmbracoContext())
It's because a) You are running on a non-request thread or b) You are unsure if you are running in a request thread or not and you don't really care.
In either of these cases you also aren't sure that there's an Umbraco Scope available so you should make one (where scopeProvider is IScopeProvider)
using (_scopeProvider.CreateScope(autoComplete: true))
{
// Do your ensure context, etc...
}
In this case, if there is already a non Transient Scope, no problem it will just create a nested one. If there is no non Transient Scope, no problem, it will create a new one. So in either case this is the safe way to perform some operation on a background thread.
I would suggest that pretty much anytime anyone is accessing the Services, UmbracoContext, etc... on a background thread that they manually create a scope. Since I think it is quite common to do such operations within TransformingIndexValues its probably worth updating the docs here https://our.umbraco.com/Documentation/Reference/Searching/Examine/examine-events to include creating a Scope within the example code for IndexProviderTransformingIndexValues ... if you have time it would be hugely appreciated if you could spend a minute sending a PR for that?
For further reading if you are interested :) There was actually a bug in the core relating to this same problem which is due for release in 8.7, you can see here for reference https://github.com/umbraco/Umbraco-CMS/pull/7994 The error is hard to reproduce because Users are cached but in long indexing operations and when there are a lot of users, if the users cache gets pruned and then during the index operation which is occurring on a background thread, it will query the users but because there is no explicit Scope, it uses the Transient Scope and in this case its CallContext we get a similar problem.
Thanks for the detailed discussion on scopes Shannon. In your example you use the parameter autoComplete: true but then mention that this scope being created might end up being a nested scope.
Documentation for CreateScope parameters explains that they are ignored for nested scopes. I in fact believe i ran into a bug related to just that. Am I correct then to assume that in this case a better example would be manually completing with scope.Complete() in case this scope being created ends up as a nested scope ?
They aren't 'ignored', if you don't complete a child scope the parent scope won't either IIRC. using autoComplete: true just means it will do a scope.Complete() automatically in it's Dispose method. Auto completing scope is an easy way to use Scopes if you are only reading and never expecting to not complete successfully with some logic. At the end of the day, it's just up to what you want to do and you shouldn't worry about whether your scope is nested or not, there can be situations where you can never assume that and there's no reason to worry about that either.
Triggering index rebuild via Hangfire causes ObjectDisposedException in NuCache
Hi fellow Dev's.
Got an odd one.
I have code that looks like this (it's in an event handler for TransformingIndexValues against the External Index:
Except, the line setting documentStub is throwing an ObjectDisposedException. What's even more unusual is that as soon as I've called
EnsureUmbracoContext()and I examineumbracoContextReference.UmbracoContext, the disposed property on it is true.If I let the exception throw and look in the logs in Umbraco it is saying that the NuCache snapshot that it is trying to access is disposed of.
To give more context, I have a HangFire task running in the background to import/update a large quantity of content. After which I need to trigger an Examine Index rebuild which all appears to work. Except when the index rebuild starts this code is hit and the exceptions are thrown.
Does anyone have any ideas as to why triggering the rebuild via a HangFire job would cause this to happen even when I'm "ensuring an Umbraco context?" What's also odd about it is that it doesn't fail for every document it's trying to index, and the number it fails for is inconsistent.
Thanks,
Nik
The problem is that most Umbraco operations require a "Scope" which is an Umbraco thing. Umbraco tries it's best to ensure something called a "Transient" scope so you don't have to worry about this too often. It's quite easy to ensure that a Scope is created and used during Http requests because we know when it starts and ends but for background tasks this is more complex.
A Scope is required for many things in Umbraco, from controlling database transactions to how service level events are handled to how nucache manages it's snapshots (and more).
For background tasks we use something called a "CallContext" and in theory this could/should be unique per threading operation but this cannot be guaranteed. Since this is a .Net thing it is out of our control. We try to create a Transient Scope based on the CallContext because there is no HttpContext and in some cases, depending on the type of background task, the type of thread (thread pool vs standalone), if async is involved, etc... the same CallContext may be used between threads. This will mean that the same Transient Scope is used between threads == bad news. (In .Net Core the notion of CallContext doesn't exist and has a much better approach).
This is precisely what is happening here, some other thread is using the same Scope and as such is using the same nucache Snapshot and when it thinks it's CallContext ends it disposes the Scope which disposes the Snapshot, but since that same Scope instance is being used on your other background thread, it tries to access a disposed object.
Generally, anytime you are doing this:
It's because a) You are running on a non-request thread or b) You are unsure if you are running in a request thread or not and you don't really care.
In either of these cases you also aren't sure that there's an Umbraco Scope available so you should make one (where scopeProvider is IScopeProvider)
In this case, if there is already a non Transient Scope, no problem it will just create a nested one. If there is no non Transient Scope, no problem, it will create a new one. So in either case this is the safe way to perform some operation on a background thread.
I would suggest that pretty much anytime anyone is accessing the Services, UmbracoContext, etc... on a background thread that they manually create a scope. Since I think it is quite common to do such operations within
TransformingIndexValuesits probably worth updating the docs here https://our.umbraco.com/Documentation/Reference/Searching/Examine/examine-events to include creating a Scope within the example code forIndexProviderTransformingIndexValues... if you have time it would be hugely appreciated if you could spend a minute sending a PR for that?For further reading if you are interested :) There was actually a bug in the core relating to this same problem which is due for release in 8.7, you can see here for reference https://github.com/umbraco/Umbraco-CMS/pull/7994 The error is hard to reproduce because Users are cached but in long indexing operations and when there are a lot of users, if the users cache gets pruned and then during the index operation which is occurring on a background thread, it will query the users but because there is no explicit Scope, it uses the Transient Scope and in this case its CallContext we get a similar problem.
Thanks for the detailed discussion on scopes Shannon. In your example you use the parameter autoComplete: true but then mention that this scope being created might end up being a nested scope.
Documentation for CreateScope parameters explains that they are ignored for nested scopes. I in fact believe i ran into a bug related to just that. Am I correct then to assume that in this case a better example would be manually completing with scope.Complete() in case this scope being created ends up as a nested scope ?
They aren't 'ignored', if you don't complete a child scope the parent scope won't either IIRC. using autoComplete: true just means it will do a scope.Complete() automatically in it's Dispose method. Auto completing scope is an easy way to use Scopes if you are only reading and never expecting to not complete successfully with some logic. At the end of the day, it's just up to what you want to do and you shouldn't worry about whether your scope is nested or not, there can be situations where you can never assume that and there's no reason to worry about that either.
is working on a reply...