We have a very strange issue on one of our TeaCommerce v2.3.3 / Umbraco v6 webshops.
Symptoms
Sometimes after an IIS ApplicationPool recycles the PaymentProviders are not correctly loaded and this breaks all functionality in Teacommerce.
We noticed this for the first when clients were not able to continue to the payment URL in the ordering process. When investigating, we also noticed the edit-orders.cshtml gave an error when opening a completed order in the background.
The error shown in the edit-orders.cshtml back-end was:
NullReferenceException: Object reference not set to an instance of an object at TeaCommerce.Umbraco.Application.Views.Orders.EditOrder.SetTransactionPayment(PaymentMethod paymentMethod)
at TeaCommerce.Umbraco.Application.Views.Orders.EditOrder.OnLoad(EventArgs e)
In the ordering process the result from the TC.generatePaymentForm javascript call was that it generated a javascript error because the result from the POST to FormPost.aspx returned an object with only:
{
eventName: "GeneratePaymentForm"
}
The required "data" object with the generated form was nowhere to be found.
Unfortunately the error went away after restarting the applicationpool again, so for testing purposes this was very annoying.
After an upgrade to the webserver (faster / more cpu) this issue seemed to happen more often, every now and then we had an angry customer on the phone telling us their clients cannot complete orders.
What we have tried so far
Logging everything in the PaymentProviders process to see which settings are send to the payment provider.
This had the result that we noticed the payment providers were not even called. It went wrong before anything was happening to the PaymentProviders.
Update Teacommerce to the latest version on the v2 range (2.3.3), updated Umbraco to the latest v6 range (6.2.6). Had no effect.
We created a simple plain ASPX page which only retrieved a hard coded payment provider using the API, somewhat like this:
We monitored this page every minute to check when it fails. Unfortunately we noticed that it happened more than we thought after an ApplicationPool reset.
This made us think that the TeaCommerce.PaymentProviders.dll was not correctly loaded.
Which is exactly the case. When you remove the TeaCommerce.PaymentProviders.dll file from the BIN folder, the resulting behavior is exactly the same as we sometimes experience.
Further investigation shown us that we experience this issue on more TeaCommerce 2.3.3 / Umbraco v6 webshops, but only when the following steps happened:
Application Pool Restart
The first request needs to be to a request to a plain aspx page, or ashx page without Umbraco context.
Than for the entire duration of the
Because we use automated calls to a plain aspx or ashx alot for background processing of certain tasks, like updating stock, etc. This happens sometimes.
Even stranger is the fact that we did not notice this on all webshops, so investigating further into the differences between webshops we noticed this issue and which we do not, it revealed that when there are several other modules loaded in the web.config HTTP Modules, like the ImageResizer plugin, or when using OWIN based .NET addons are used, the startup after AppPool restart seems to take a couple of seconds longer, but the Payment Providers are correctly loaded.
Conclusion
This is utterly strange behavior, and we think it has something to do with the timing/initialization of the TeaCommerce application. It seems to be failing when things are happening too early or not in sync.
Unfortunately the Umbraco Logs are completely the same on a successful startup and a failed one which do not load the payment providers. No errors or info messages.
Question
Is this a known issue or can we do some more testing to prevent this from happening. Is there any extra logging or debugging we can enable to see the exact difference of the site startup?
I do not have the complete code of Teacommerce v2 so debugging is somewhat tedious.
Maybe somebody from the core team who can help us? Any help would be very appreciated.
When you use the payment provider service and calls Get - when the service is constructed - this code runs:
public PaymentProviderService() {
_paymentProviderTypes = new Dictionary<string, Type>();
//Load all assemblies and see if any types has the PaymentProvider attribute - loads the types and registers them
foreach ( Assembly assembly in AppDomain.CurrentDomain.GetAssemblies() ) {
try {
foreach ( Type type in assembly.GetTypesWithAttribute<PaymentProviderAttribute>() ) {
string alias = type.GetSingleAttribute<PaymentProviderAttribute>().Alias;
if ( !_paymentProviderTypes.ContainsKey( alias ) ) {
_paymentProviderTypes.Add( alias, type );
} else {
throw new Exception( "A payment provider with the alias - " + alias + " - has already been added" );
}
}
} catch ( Exception ex ) {
if ( ex is ArgumentException && ex.Message.StartsWith( "A payment provider with the alias" ) ) {
throw;
}
}
}
}
public IPaymentProvider Get( string paymentProviderAlias ) {
IPaymentProvider paymentProvider = null;
if ( _paymentProviderTypes.ContainsKey( paymentProviderAlias ) ) {
paymentProvider = (IPaymentProvider)Activator.CreateInstance( _paymentProviderTypes[ paymentProviderAlias ] );
}
return paymentProvider;
}
Any of this that makes you have an idea of what might be wrong? Could it be the AppDomain.CurrentDomain.GetAssemblies()? If its not Umbraco that is in the process then this might give back some wrong assemblies and then the payment provider service is constructed but with no providers as it could not find them in the assemblies it got from AppDomain.CurrentDomain.GetAssemblies()
Thanks for your reply. It definitely is a strange case. But I can reproduce it on every shop we've tested using the steps provided in the opening post.
Umbraco is being loaded (I can see that in the Umbraco Logs), the Umbraco logs show no difference between a succesfull startup and a unsuccesfull startup.
Is it possible for you to add some logging to that section (i.e. which assemblies are in CurrentDomain.GetAssemblies()
and compile a new DLL for me (version 2.3.3) so I can test it and provide the output so we can debug this somewhat further?
I've tested it on both a successful startup and a not successful one and noticed that the logging shows:
Unsuccessful startup:
- 95 assemblies listed (including TeaCommerce.Api.Web, TeaCommerce.Api.Web.PaymentProviders.Invoicing and TeaCommerce.Api.Persistence), but NOT the paymentproviders binaries.
Successful startup:
- 201 assemblies listed (including all TeaCommerce and Payment Provider binaries).
However both attempts includes twice the following exception:
System.NotSupportedException: The invoked member is not supported in a dynamic assembly.
at System.Reflection.Emit.InternalAssemblyBuilder.GetExportedTypes()
at TeaCommerce.Api.Common.AssemblyExtensions.GetTypesWithAttribute[T](Assembly assembly)
at TeaCommerce.Api.Web.PaymentProviders.PaymentProviderService..ctor(ILoggingService logging)
The "The invoked member is not supported in a dynamic assembly" exception is nothing to talk about. That is just because some code in App_Code is behaving a little different that normal .Net compiled code. I just logged all things for your test to help us a little more.
So looks like the problem is the CurrentDomain.GetAssemblies() that returns something different when you run a normal aspx page first. So could you try in your own code and see if there is an assembly method that gives the correct dll files in both cases?
Did some testing and research and I found out that AppDomain.CurrentDomain.GetAssemblies() does not list all assemblies when they are not loaded yet (.NET uses deferred loading).
Using the alternative method: BuildManager.GetReferencedAssemblies() does give all assemblies.
Is it possible for you to generate a new DLL for this version for me? (or release an update also for Teacommerce v2)?
I think this will definitely solve the problem.
I see what you mean. But the problem is that it is in System.Web. We won't have Tea Commerce rely on that in the Api project as that is not a web project. So is there a way to do that same but don't use the System.Web dll file?
I understand what you mean, but the code you referenced is in the PaymentProviderService from the TeaCommerce.Api.Web.dll, which (according to DotPeek) already references and uses System.Web.
I don't think it's a problem to use the BuildManager there, or am I wrong?
If it is a problem, I think the only alternative way is to iterate over the BIN folder to load all DLL's. That is because not any other (loaded) assembly references the TeaCommerce.PaymentProviders.dll directly.
Using for instance Assembly.GetExecutingAssembly().GetReferencedAssemblies() I never get to the TeaCommerce.PaymentProviders binary, because none of the projects has a direct reference to it. Besides, if I use a different assembly to put my custom paypent providers this assembly will never be found.
Besides, in my opinion the approach using the BIN folder directly is more un-clean then using a method from System.Web in the TeaCommerce.Api.Web assembly.
For your info, here is some piece of code I used to iterate over the bin folder (which also loaded the unreferenced TeaCommerce.PaymentProviders.dll):
string binPath = System.IO.Path.Combine(System.AppDomain.CurrentDomain.BaseDirectory, "bin");
foreach (string dll in Directory.GetFiles(binPath, "*.dll", SearchOption.AllDirectories))
{
try
{
Assembly loadedAssembly = Assembly.LoadFile(dll);
Response.Write(loadedAssembly.FullName + "<br />");
}
catch (FileLoadException loadEx)
{ } // The Assembly has already been loaded.
catch (BadImageFormatException imgEx)
{ } // If a BadImageFormatException exception is thrown, the file is not an assembly.
}
It is okay that we use it in the payment providers as that is in the Api.Web, but we use some of the same code in Tea Commerce in the DependencyContainer class in TeaCommerce.Api.dll and would like to find a method that works in both cases. Else we might have another case where something 3 party suddenly doesn't work because that is not loaded correctly. What we want in the dependency container is to loop all dll files and find the ones that has used the SupressDependency attribute.
Do you have an idea how to solve that with out using the System.Web dll?
I think you could do something like this, based on your piece of code earlier this thread.
If you'd like I can test this piece of code for you if you can provide me with some new dll's where this has changed.
var appPath = AppDomain.CurrentDomain.BaseDirectory;
var files = Directory.GetFiles(
appPath + "\\bin", "*.dll",
SearchOption.AllDirectories);
foreach (string file in files)
{
try
{
var assembly = Assembly.LoadFile(file);
foreach (Type type in assembly.GetTypesWithAttribute<PaymentProviderAttribute>())
{
string alias = type.GetSingleAttribute<PaymentProviderAttribute>().Alias;
if (!_paymentProviderTypes.ContainsKey(alias))
{
_paymentProviderTypes.Add(alias, type);
}
else
{
throw new Exception("A payment provider with the alias - " + alias + " - has already been added");
}
}
}
catch (Exception ex)
{
if (ex is ArgumentException && ex.Message.StartsWith("A payment provider with the alias"))
{
throw;
}
}
}
Very strange issue on startup Teacommerce
We have a very strange issue on one of our TeaCommerce v2.3.3 / Umbraco v6 webshops.
Symptoms
Sometimes after an IIS ApplicationPool recycles the PaymentProviders are not correctly loaded and this breaks all functionality in Teacommerce. We noticed this for the first when clients were not able to continue to the payment URL in the ordering process. When investigating, we also noticed the edit-orders.cshtml gave an error when opening a completed order in the background.
The error shown in the edit-orders.cshtml back-end was:
In the ordering process the result from the TC.generatePaymentForm javascript call was that it generated a javascript error because the result from the POST to FormPost.aspx returned an object with only:
The required "data" object with the generated form was nowhere to be found.
Unfortunately the error went away after restarting the applicationpool again, so for testing purposes this was very annoying.
After an upgrade to the webserver (faster / more cpu) this issue seemed to happen more often, every now and then we had an angry customer on the phone telling us their clients cannot complete orders.
What we have tried so far
Logging everything in the PaymentProviders process to see which settings are send to the payment provider.
This had the result that we noticed the payment providers were not even called. It went wrong before anything was happening to the PaymentProviders.
Update Teacommerce to the latest version on the v2 range (2.3.3), updated Umbraco to the latest v6 range (6.2.6). Had no effect.
We created a simple plain ASPX page which only retrieved a hard coded payment provider using the API, somewhat like this:
_
We monitored this page every minute to check when it fails. Unfortunately we noticed that it happened more than we thought after an ApplicationPool reset.
This made us think that the TeaCommerce.PaymentProviders.dll was not correctly loaded.
Which is exactly the case. When you remove the TeaCommerce.PaymentProviders.dll file from the BIN folder, the resulting behavior is exactly the same as we sometimes experience.
Further investigation shown us that we experience this issue on more TeaCommerce 2.3.3 / Umbraco v6 webshops, but only when the following steps happened:
Because we use automated calls to a plain aspx or ashx alot for background processing of certain tasks, like updating stock, etc. This happens sometimes.
Even stranger is the fact that we did not notice this on all webshops, so investigating further into the differences between webshops we noticed this issue and which we do not, it revealed that when there are several other modules loaded in the web.config HTTP Modules, like the ImageResizer plugin, or when using OWIN based .NET addons are used, the startup after AppPool restart seems to take a couple of seconds longer, but the Payment Providers are correctly loaded.
Conclusion
This is utterly strange behavior, and we think it has something to do with the timing/initialization of the TeaCommerce application. It seems to be failing when things are happening too early or not in sync.
Unfortunately the Umbraco Logs are completely the same on a successful startup and a failed one which do not load the payment providers. No errors or info messages.
Question
Is this a known issue or can we do some more testing to prevent this from happening. Is there any extra logging or debugging we can enable to see the exact difference of the site startup? I do not have the complete code of Teacommerce v2 so debugging is somewhat tedious.
Maybe somebody from the core team who can help us? Any help would be very appreciated.
Hi Rody
That is a strange case. And not something we have ever heard of from anyone else.
The code for Tea Commerce 2 is not open source, but the code for v3 is also most the same. So here is the code that starts Tea Commerce: https://github.com/TeaCommerce/Tea-Commerce-for-Umbraco/blob/master/Source/TeaCommerce.Umbraco.Configuration/ApplicationStartup.cs
When you use the payment provider service and calls Get - when the service is constructed - this code runs:
Any of this that makes you have an idea of what might be wrong? Could it be the AppDomain.CurrentDomain.GetAssemblies()? If its not Umbraco that is in the process then this might give back some wrong assemblies and then the payment provider service is constructed but with no providers as it could not find them in the assemblies it got from AppDomain.CurrentDomain.GetAssemblies()
Kind regards
Anders
Hi Anders,
Thanks for your reply. It definitely is a strange case. But I can reproduce it on every shop we've tested using the steps provided in the opening post.
Umbraco is being loaded (I can see that in the Umbraco Logs), the Umbraco logs show no difference between a succesfull startup and a unsuccesfull startup.
Is it possible for you to add some logging to that section (i.e. which assemblies are in CurrentDomain.GetAssemblies() and compile a new DLL for me (version 2.3.3) so I can test it and provide the output so we can debug this somewhat further?
Hi Anders,
I've tested it on both a successful startup and a not successful one and noticed that the logging shows:
Unsuccessful startup: - 95 assemblies listed (including TeaCommerce.Api.Web, TeaCommerce.Api.Web.PaymentProviders.Invoicing and TeaCommerce.Api.Persistence), but NOT the paymentproviders binaries.
Successful startup: - 201 assemblies listed (including all TeaCommerce and Payment Provider binaries).
However both attempts includes twice the following exception:
Do you have any idea on this?
Thanks!
The "The invoked member is not supported in a dynamic assembly" exception is nothing to talk about. That is just because some code in App_Code is behaving a little different that normal .Net compiled code. I just logged all things for your test to help us a little more.
So looks like the problem is the CurrentDomain.GetAssemblies() that returns something different when you run a normal aspx page first. So could you try in your own code and see if there is an assembly method that gives the correct dll files in both cases?
Kind regards
Anders
Did some testing and research and I found out that AppDomain.CurrentDomain.GetAssemblies() does not list all assemblies when they are not loaded yet (.NET uses deferred loading).
Using the alternative method: BuildManager.GetReferencedAssemblies() does give all assemblies.
See this StackOverflow topic
Is it possible for you to generate a new DLL for this version for me? (or release an update also for Teacommerce v2)? I think this will definitely solve the problem.
I see what you mean. But the problem is that it is in System.Web. We won't have Tea Commerce rely on that in the Api project as that is not a web project. So is there a way to do that same but don't use the System.Web dll file?
I understand what you mean, but the code you referenced is in the PaymentProviderService from the TeaCommerce.Api.Web.dll, which (according to DotPeek) already references and uses System.Web.
I don't think it's a problem to use the BuildManager there, or am I wrong?
If it is a problem, I think the only alternative way is to iterate over the BIN folder to load all DLL's. That is because not any other (loaded) assembly references the TeaCommerce.PaymentProviders.dll directly.
Using for instance Assembly.GetExecutingAssembly().GetReferencedAssemblies() I never get to the TeaCommerce.PaymentProviders binary, because none of the projects has a direct reference to it. Besides, if I use a different assembly to put my custom paypent providers this assembly will never be found.
Besides, in my opinion the approach using the BIN folder directly is more un-clean then using a method from System.Web in the TeaCommerce.Api.Web assembly.
For your info, here is some piece of code I used to iterate over the bin folder (which also loaded the unreferenced TeaCommerce.PaymentProviders.dll):
It is okay that we use it in the payment providers as that is in the Api.Web, but we use some of the same code in Tea Commerce in the DependencyContainer class in TeaCommerce.Api.dll and would like to find a method that works in both cases. Else we might have another case where something 3 party suddenly doesn't work because that is not loaded correctly. What we want in the dependency container is to loop all dll files and find the ones that has used the SupressDependency attribute. Do you have an idea how to solve that with out using the System.Web dll?
https://documentation.teacommerce.net/net-api/override-default-providers/
Kind regards
Anders
I think you could do something like this, based on your piece of code earlier this thread.
If you'd like I can test this piece of code for you if you can provide me with some new dll's where this has changed.
is working on a reply...