A big mess in orders and payments after update to 1.8.5
Hi,
I have updated our customer solution from Vendr 1.8.0 to 1.8.5 (Umbraco 8.15.1). Payment provider: Bambora Checkout.
There were no such problems before update.
Their customers begun complaining, that there are unknown or wrong payments being captured from their cards/bank accounts.
I'm trying to investigate the problem, but can see, that there is a total mess i Vendr order table. There are no orders with transactionID(s) from Bambora. When I'm looking on dates, some orders are saved only with CART number. Some are with ORDER numbers, but not finalized.
Example 1:
Payments from Bambora:
orders in database for the customer:
I guess the capture on 150 DKK may come from the not finalized order, looking on create time.
Example 2:
There is an order with a date/time that agrees with payment date. It has a order number, but is not finalized and is missing payment ID
I don't know if it was better, but TeaCommerce was sending Cart number to payment provider, so it was easier to track payment problems.
There are no errors in the umbraco log around order/payment timestamps.
Is there any other place I can look to investigate the problem?
This is very strange! I can’t see how a patch update would cause this, but I’ll review the release and see if there is anything I can suggest.
I’m really unsure how the gateway could just process random payments. Weirdly the orders you show in Bambara seem to have an order number against them so the order it relates to must have gone to the checkout.
Is there any way the Bambara provider hasn’t been accidentally setup on another install somewhere? Although even this I’m bemused how it could just randomly charge people 🤔
PS you can see the Order ID in Bambora is the Order Number from Vendr with the hyphens removed.
I can’t fully recall how Bambora works, but does it remember customer cards? As I can’t for the life of me understand how a random payment would just occur.
Yes, OrderID in Bambora is order number without hyphens. And I'm trying to use this numbers to find orders in Vendr. But it doesn't help, if there is only Cart number i Vendr.orders :(
Yes, it's very strange.
No, this payment provider is used only in this one place.
If I could guess, I would think, that some paid orders are missing in Vendr.
All I have now is some mail from customer's customers with screenshots, that their accounts were debited many times or with wrong amounts.
I can see payments in Bambora dashboard, but I can't find anything in Vendr. There are no orders with orderId/transactionID saved in Bambora.
Customar claims, that they haven't deleted anything self in Umbraco/Vendr/orders.
I'm investigating other payment since the last update and trying to find a pattern.
The "new" thing in our code is a custom OrderFinalizedHandler. We save ordered items in a custom table. The handler reads the finalized order and "converts" orderlines to records in the custom table. But there is no manipulation on the order. Can it be a problem anyway?
public class OrderFinalizedHandler : NotificationEventHandlerBase<OrderFinalizedNotification>
{
private readonly ILogger _logger;
private readonly MealOrdersConverter _ordersConverter;
public OrderFinalizedHandler(ILogger logger, MealOrdersConverter ordersConverter)
{
_logger = logger;
_ordersConverter = ordersConverter;
}
public override void Handle(OrderFinalizedNotification evt)
{
var task = Task.Run(() => _ordersConverter.ConvertVendrOrderToMealOrders(evt.Order));
task.ContinueWith(t =>
{
var processedItemsCount = t.Result;
_logger.Info<OrderFinalizedHandler>($"===OrderFinalized=== ConvertVendrOrderToMealOrders. OrderId: {evt.Order.Id} OrderNumber: {evt.Order.OrderNumber} OrderLinesCount: {evt.Order.OrderLines.Count()} OrderLinesProcessed: {processedItemsCount}");
});
}
}
ConvertVendrOrderToMealOrders just reads some order properties and loops through order lines (and reads order lines properties).
Is there no customer object linked to the transaction? (I just tried logging into my Bambora test account but all the historic orders aren't working 🤦♂️)
Have you tested a transaction you know was processed by Vendr?
I can see, that most of the "wrong" transactions (if not all) is paid with Mobilepay, which is a mobile payments solution activated in Bambora. But it means, that user leaves webbrowser (on the phone), switch to another app to accept the payment, and returns to the browser.
I try to test it, but can it lead to loose the order data, somehow?
I will be also strange, because they were using Mobile pay befor update without problems...
Hmmm, it really shouldn't. The session should still be maintained, so it should know when they return to the site.
The only real thing I can think to do is to pull the source code for the Bambora provider, update it with a ton of logging in it and see if the payment provider actually has anything to do with it. Does it instigate the payment? Does it get notified of the payment?
I honestly can't think how a payment would even trigger with no record of the order in the database. Which is why I wondered if the Bambora details weren't being used on another install.
RE these payments. Are the customers saying they didn't instigate them at all? Or that they are just for the wrong amounts. If it's that they didn't instigate them, then I'd also question how could that be? Vendr only ever instigates a payment through the checkout flow. The only other actions it performs is via buttons in the back office to capture / refund / cancel transactions, and then the rest is listening for changes and just syncing them to Vendr.
There are customers that instigate payments after they log in to the system. They go through payment, but something goes wrong after successful payment. I can see these payments in Bambora portal, but can't find the order in Umbraco/Vendr.
For some transactions I can find a record in vendrOrder when I know the member Id. But in almost all wrong cases, the ordernumber in Bambora is not existing in vendrOrder (so it's very hard to find something/anything...)
I could see, that some orders that were not finalized (and I could find them), has older createDate. It could mean, that customers "reused" existing order. I don't think this is the problem, but I'm investigating all.
I changed teh Cookie Timeout in shop settings from 525600 to 30 minutes. Does it mean, that all "existing" orders older than 30 minutes will be invalidated or does it apply only to new orders?
It's a shame Bambora doesn't hold more meta data as for most other payment provider we pass an Order Reference that basically consists of the order ID and the order number so even if the order number somehow didn't persist, the order ID should at least match so you could tell if the order existed but was in a previous state for some reason.
I do find it odd though that, if the orders you are locating in the DB are in fact the the ones the transactions are for, the fact they are so wrong is strange as it potentially means either multiple persists of the order are not happening but Vendr seems to think they are, or, the order is being reset to a previous state or completely deleted. Either option is completely baffling.
Older createDates will indeed just signify when the order was initially created, which yes, based on the cookie value could be days, weeks or even months before they actually finalize. Changing the cookie length though will indeed only affect new cookies.
Again, the only thing I can think to do is the custom Bambora provider with a ton of logging in to see what actions are actually occurring. Maybe logging the order ID and order Number so you can try and cross reference the exact order with a transaction.
I can find this order by orderId i the database, but its orderNumber is null.
I can't reproduce it on staging, nor localhost, so it is not easy to test...
In changlog to v. 1.8.3 is a "change": "Updated the payment controller actions to not continue with processing if the order reference on file does not match that in the URL."
Can it have something to do wit it?
I'm about do downgrade Vendr back to 1.8.0. I hope it's doable.
So those order ID's exist in the database but they have no Order Number? Wow, that really shouldn't be possible. Clearly the order was given an order number or that URL wouldn't have been created, but why it's not persisted I don't know.
The ONLY thing I can think of, is if there is something in a custom event handler somewhere that is re-persisting the order to a state prior to the order number being assigned, but even this I'm not sure how it would be achievable in such a way that part of the system thinks the update has completed but in reality it hasn't.
I don't believe the 1.8.3 update would be a problem as, if anything, it's more restrictive and prevents processing when the order number doesn't exist.
UPDATE: Actually, another way an order can have it's order number cleared is the customer opens the payment gateway in say a new window, but then in the previous window modifies the order in such as way that changes the order price / requires re-calculation. This would force the order number to be cleared, expecting a new one to be re-created when they go back to the review step (ie, this is required behavior for if a customers cancels out of the checkout process and updates the order. They are given a new order number).
There is potential that maybe there is also an event handler somewhere updating the order in much the same way. ie, something is forcing the order to recalculate after the customer has been sent to the payment gateway and thus it's resetting the order number.
With all of this though, I can only see how this could allow an order to continue without it finalizing, but I can't see how it would be issuing random payment amounts 🤔
Hi Matt,
I've came to the same conclusion, that order number is being cleared.
I have a checkout page, where customer can still change quantity or remove order line. This is invoked via API, so checkout page is not reload - this is why it "remembers" the order number, but it gets cleared in the backend.
So I now understand, why callback from bambora can't find the order.
It's by design. It's an edge case, but there are some payment gateways that don't let you process the same order number twice so if the order changes after BeginPaymentForm is called, then this will reset the order number such that when BeginPaymentForm is called again, this will generate a new order number.
The expected flow though is this.
Create / Update Order
Review stage (No modifications to order should occur here, but it's here you call BeginPaymentForm to setup the payment on the gateway)
User redirected to payment gateway
Either complete payment, or cancel (customer may have forgotten something)
If canceling, go back to site and reset order number
Customer can update order
Customer goes back to review step where BeginPaymentForm is called and the order is given a new order number.
Go back to step 3
So yea, if you are rendering the payment form, but then allowing the order to be modified, this will be resetting the order number, but also not updating the order details that were sent to Bambora and so potentially this is your miss-matching payment amounts too.
If you are doing a single page checkout, you might want to do something like what we do on Vendr.net which is to have a checkout page were people can update whatever they like on a single page, but then this posts through to another page on your site which it's this page that renders the BeginPaymentForm and uses some javascript to auto submit that form and send them to the payment gateway. We show a message like "Redirecting to Payment Gateway".
Really glad we were able to find the culprit. I may look at what we can do about adding some logging if the order number gets cleared so we can at least know that is what is happening and then we can understand why.
Yea, this is different between Vendr and TeaCommerce. TeaCommerce doesn't actually generate an order number until after payment is complete at the payment gateway, but it also does a weird thing in that it duplicates the entire order if you were to make a modification. The issue with this is that people are used to using the Order Number to reference things, but in TC the only thing you have is a Cart Number which is never really displayed to the customer in any other circumstance. This becomes even worse in TC when you know that most payment gateways now confirm their payments asynchronously via webhook and so quite often it means you don't even have an order number you can display on the confirmation page as usually the webhook and the redirect to the confirmation page happen at the same time and so you can't guarantee the webhook will hit first, finalizing the order and generating the order number.
In Vendr we took the option to generate the Order Number prior to going to the payment gateway so that you have a consistent number you display through the checkout flow. Obviously, as we have now learnt, this also comes with some pitfalls, but IMO these were more logical than in TC (ie, customers expect to see an Order Number)
I hope this at least helps explain why things are the way they are.
A big mess in orders and payments after update to 1.8.5
Hi,
I have updated our customer solution from Vendr 1.8.0 to 1.8.5 (Umbraco 8.15.1). Payment provider: Bambora Checkout. There were no such problems before update.
Their customers begun complaining, that there are unknown or wrong payments being captured from their cards/bank accounts.
I'm trying to investigate the problem, but can see, that there is a total mess i Vendr order table. There are no orders with transactionID(s) from Bambora. When I'm looking on dates, some orders are saved only with CART number. Some are with ORDER numbers, but not finalized.
Example 1:
Payments from Bambora:
orders in database for the customer:
I guess the capture on 150 DKK may come from the not finalized order, looking on create time.
Example 2:
There is an order with a date/time that agrees with payment date. It has a order number, but is not finalized and is missing payment ID
I don't know if it was better, but TeaCommerce was sending Cart number to payment provider, so it was easier to track payment problems.
There are no errors in the umbraco log around order/payment timestamps.
Is there any other place I can look to investigate the problem?
Regards
Tomasz
Hi Thomasz
This is very strange! I can’t see how a patch update would cause this, but I’ll review the release and see if there is anything I can suggest.
I’m really unsure how the gateway could just process random payments. Weirdly the orders you show in Bambara seem to have an order number against them so the order it relates to must have gone to the checkout.
Is there any way the Bambara provider hasn’t been accidentally setup on another install somewhere? Although even this I’m bemused how it could just randomly charge people 🤔
PS you can see the Order ID in Bambora is the Order Number from Vendr with the hyphens removed.
I can’t fully recall how Bambora works, but does it remember customer cards? As I can’t for the life of me understand how a random payment would just occur.
Yes, OrderID in Bambora is order number without hyphens. And I'm trying to use this numbers to find orders in Vendr. But it doesn't help, if there is only Cart number i Vendr.orders :(
And yes, Bambora remebers obfuscated card number
Hi, Matt
Yes, it's very strange. No, this payment provider is used only in this one place. If I could guess, I would think, that some paid orders are missing in Vendr.
All I have now is some mail from customer's customers with screenshots, that their accounts were debited many times or with wrong amounts.
I can see payments in Bambora dashboard, but I can't find anything in Vendr. There are no orders with orderId/transactionID saved in Bambora.
Customar claims, that they haven't deleted anything self in Umbraco/Vendr/orders.
I'm investigating other payment since the last update and trying to find a pattern.
The "new" thing in our code is a custom OrderFinalizedHandler. We save ordered items in a custom table. The handler reads the finalized order and "converts" orderlines to records in the custom table. But there is no manipulation on the order. Can it be a problem anyway?
ConvertVendrOrderToMealOrders just reads some order properties and loops through order lines (and reads order lines properties).
Regards
Tomasz
I can't see anything wrong with this, but I'm guessing you aren't seeing any of those log statements for the given orders right?
One more thing. Customer's licence expired on the 11th august. It means, that I can't upgrade theirs Vendr to 1.8.6, unless they buy licence upgrade?
Tomasz
Hi Thomasz,
RE upgrade, yes that would be the case unfortunately, however I'm not yet convinced this is an error that an upgrade would currently resolve.
So looking at the Bambora payment provider, the only other "unique" thing we store is the customers email address https://github.com/vendrhub/vendr-payment-provider-bambora/blob/v2/dev/src/Vendr.PaymentProviders.Bambora/BamboraCheckoutPaymentProvider.cs#L57 I wonder if you check the orders table, if this can help you identify the orders.
Also, are you load balanced at all?
Matt
I can't see customers email address in Bambora transaction detials:
No, there is no any load balancer, as far as I know (customer hosts the solution on their server).
Hi Tomasz,
Is there no customer object linked to the transaction? (I just tried logging into my Bambora test account but all the historic orders aren't working 🤦♂️)
Have you tested a transaction you know was processed by Vendr?
Matt
I can see, that most of the "wrong" transactions (if not all) is paid with Mobilepay, which is a mobile payments solution activated in Bambora. But it means, that user leaves webbrowser (on the phone), switch to another app to accept the payment, and returns to the browser.
I try to test it, but can it lead to loose the order data, somehow?
I will be also strange, because they were using Mobile pay befor update without problems...
Tomasz
Hmmm, it really shouldn't. The session should still be maintained, so it should know when they return to the site.
The only real thing I can think to do is to pull the source code for the Bambora provider, update it with a ton of logging in it and see if the payment provider actually has anything to do with it. Does it instigate the payment? Does it get notified of the payment?
I honestly can't think how a payment would even trigger with no record of the order in the database. Which is why I wondered if the Bambora details weren't being used on another install.
RE these payments. Are the customers saying they didn't instigate them at all? Or that they are just for the wrong amounts. If it's that they didn't instigate them, then I'd also question how could that be? Vendr only ever instigates a payment through the checkout flow. The only other actions it performs is via buttons in the back office to capture / refund / cancel transactions, and then the rest is listening for changes and just syncing them to Vendr.
There are customers that instigate payments after they log in to the system. They go through payment, but something goes wrong after successful payment. I can see these payments in Bambora portal, but can't find the order in Umbraco/Vendr.
For some transactions I can find a record in vendrOrder when I know the member Id. But in almost all wrong cases, the ordernumber in Bambora is not existing in vendrOrder (so it's very hard to find something/anything...)
I could see, that some orders that were not finalized (and I could find them), has older createDate. It could mean, that customers "reused" existing order. I don't think this is the problem, but I'm investigating all.
I changed teh Cookie Timeout in shop settings from 525600 to 30 minutes. Does it mean, that all "existing" orders older than 30 minutes will be invalidated or does it apply only to new orders?
Anyway thank you for your time. I appreciate it.
Regards Tomasz
Hmmm, very strange indeed.
It's a shame Bambora doesn't hold more meta data as for most other payment provider we pass an Order Reference that basically consists of the order ID and the order number so even if the order number somehow didn't persist, the order ID should at least match so you could tell if the order existed but was in a previous state for some reason.
I do find it odd though that, if the orders you are locating in the DB are in fact the the ones the transactions are for, the fact they are so wrong is strange as it potentially means either multiple persists of the order are not happening but Vendr seems to think they are, or, the order is being reset to a previous state or completely deleted. Either option is completely baffling.
Older createDates will indeed just signify when the order was initially created, which yes, based on the cookie value could be days, weeks or even months before they actually finalize. Changing the cookie length though will indeed only affect new cookies.
Again, the only thing I can think to do is the custom Bambora provider with a ton of logging in to see what actions are actually occurring. Maybe logging the order ID and order Number so you can try and cross reference the exact order with a transaction.
Hi again, Investigating continues... I can see in the IIS log, that both calls returns error 500:
/umbraco/vendr/payment/callback/bambora-checkout/6250f883-6c91-4c79-9308-dca4fee8e551/ORDER-0651-0861-J6N3/umbraco/vendr/payment/continue/bambora-checkout/6250f883-6c91-4c79-9308-dca4fee8e551/ORDER-0651-0861-J6N3I can find this order by orderId i the database, but its orderNumber is null.
I can't reproduce it on staging, nor localhost, so it is not easy to test...
In changlog to v. 1.8.3 is a "change": "Updated the payment controller actions to not continue with processing if the order reference on file does not match that in the URL."
Can it have something to do wit it?
I'm about do downgrade Vendr back to 1.8.0. I hope it's doable.
Regards
Tomasz
Hi Thomasz,
So those order ID's exist in the database but they have no Order Number? Wow, that really shouldn't be possible. Clearly the order was given an order number or that URL wouldn't have been created, but why it's not persisted I don't know.
The ONLY thing I can think of, is if there is something in a custom event handler somewhere that is re-persisting the order to a state prior to the order number being assigned, but even this I'm not sure how it would be achievable in such a way that part of the system thinks the update has completed but in reality it hasn't.
I don't believe the 1.8.3 update would be a problem as, if anything, it's more restrictive and prevents processing when the order number doesn't exist.
UPDATE: Actually, another way an order can have it's order number cleared is the customer opens the payment gateway in say a new window, but then in the previous window modifies the order in such as way that changes the order price / requires re-calculation. This would force the order number to be cleared, expecting a new one to be re-created when they go back to the review step (ie, this is required behavior for if a customers cancels out of the checkout process and updates the order. They are given a new order number).
There is potential that maybe there is also an event handler somewhere updating the order in much the same way. ie, something is forcing the order to recalculate after the customer has been sent to the payment gateway and thus it's resetting the order number.
With all of this though, I can only see how this could allow an order to continue without it finalizing, but I can't see how it would be issuing random payment amounts 🤔
Hi Matt, I've came to the same conclusion, that order number is being cleared.
I have a checkout page, where customer can still change quantity or remove order line. This is invoked via API, so checkout page is not reload - this is why it "remembers" the order number, but it gets cleared in the backend.
So I now understand, why callback from bambora can't find the order.
But is it a bug or a feature, that
clears the OrderNumber property?
But I'm very happy that I finally found this...
Regards
Tomasz
Hi Thomasz,
Ahhh, we got it!
It's by design. It's an edge case, but there are some payment gateways that don't let you process the same order number twice so if the order changes after
BeginPaymentFormis called, then this will reset the order number such that whenBeginPaymentFormis called again, this will generate a new order number.The expected flow though is this.
BeginPaymentFormto setup the payment on the gateway)BeginPaymentFormis called and the order is given a new order number.So yea, if you are rendering the payment form, but then allowing the order to be modified, this will be resetting the order number, but also not updating the order details that were sent to Bambora and so potentially this is your miss-matching payment amounts too.
If you are doing a single page checkout, you might want to do something like what we do on Vendr.net which is to have a checkout page were people can update whatever they like on a single page, but then this posts through to another page on your site which it's this page that renders the
BeginPaymentFormand uses some javascript to auto submit that form and send them to the payment gateway. We show a message like "Redirecting to Payment Gateway".Really glad we were able to find the culprit. I may look at what we can do about adding some logging if the order number gets cleared so we can at least know that is what is happening and then we can understand why.
Matt
Hi Matt,
Maybe I'm still soaked with TeaCommerce and was unaware of this behaviour (changing order number) in Vendr, but it makes sense.
Case closed! :)
Thank you again for help :)
Regards
Tomasz
Hi Tomasz,
Yea, this is different between Vendr and TeaCommerce. TeaCommerce doesn't actually generate an order number until after payment is complete at the payment gateway, but it also does a weird thing in that it duplicates the entire order if you were to make a modification. The issue with this is that people are used to using the Order Number to reference things, but in TC the only thing you have is a Cart Number which is never really displayed to the customer in any other circumstance. This becomes even worse in TC when you know that most payment gateways now confirm their payments asynchronously via webhook and so quite often it means you don't even have an order number you can display on the confirmation page as usually the webhook and the redirect to the confirmation page happen at the same time and so you can't guarantee the webhook will hit first, finalizing the order and generating the order number.
In Vendr we took the option to generate the Order Number prior to going to the payment gateway so that you have a consistent number you display through the checkout flow. Obviously, as we have now learnt, this also comes with some pitfalls, but IMO these were more logical than in TC (ie, customers expect to see an Order Number)
I hope this at least helps explain why things are the way they are.
Matt
is working on a reply...