A/B Testing, Feature Roadmap, Extensibility and Technical Support
Please note: this is a set of questions I asked while doing an evaluation of this extension for our company. They requested that I post the questions and responses to this forum for others to reference.
Q1. The Single page A/B codeless test is amazing – content editors could set this up without any outside help. On the multi-page A/B test, is there any way to do the same type of content setup? For example if I wanted to enable an A/B test for a certain document type – it would be ideal to be able to add the variant content to each page of that type with the split editor and have that be included in the test. If not is this something planned for the future? I ask because there is a test we just did exactly like this on our site which required a lot of manual setup the way we do it now.
Answer: => The single page A/B test is indeed fantastic 😊. I think this the best part of the uMarketingSuite at this moment. The reason why we do not use the splitview for the multiple page setup is that I don’t exactly know how this would work. Let’s say that you selected 10 pages during the setup of the A/B Test. Would you go through all these 10 pages in splitview to adjust the content for variant B or variant C? And how good would your test be at that moment. Because you’re probably testing all kinds of different texts on different pages I don’t think your AB Test would be really reliable. The idea is that you test one thing and test whether it performs better or worse compared to the original. If you change to many variables, you cannot really interpret the results correctly on our opinion.
In a single page you can test for a specific page whether the optimization of content gives you better results.
In a multipage test it’s possible to add-in additional css or javascript to do something about the template of the website.
When A/B Testing you should test one hypothesis / change per time in our opinion.
Maybe you could explain a bit more in detail what you hope to achieve. A workaround for your case could be to do some development yourself. We’re documenting & opening up the APIs of the uMarketingSuite this sprint so you can maybe use these to implement your exact requirement!
Q2. The Split URL testing feature is not listed on the roadmap that I saw – is there any rough idea when this will be available?
Answer: => We’re constantly discussing the roadmap with our uMarketingSuite partners (https://www.umarketingsuite.com/partners/). We’ve created a partnership model (https://www.umarketingsuite.com/partners/become-a-umarketingsuite-partner/) so partners can influence the roadmap, but also support the uMarketingSuite financially. We’re still a startup so we constantly have to prioritize our resource in the right way.
At this moment we do not have an exact date for the split url feature, because none of our partners saw this at a priority at the moment.
Once again; I think it can be implemented yourself once we’ve opened up the APIs, although this is probably more work than expected.
Q3. The technical documentation on programmatic extensibility is still in a to-do state – we do a lot of custom extension to Umbraco for our website so we would be really interested in any details you can provide here. Concern is both to make sure our customizations don’t break your rendering pipeline but also to see what we can do in terms of advanced tests such as the multi-page test scenario I mentioned in point 1.
Answer: => We’re documenting as much as we can do in this sprint. Also you get the possibility to create your own segment parameters. We will have this available mid-December 2020!
Q4. How is technical support handled with full licenses? Is there a separate support channel or contact point?
Answer=> At this moment it’s handled via email. Depending on the questions we can setup a meeting with the dev-team if needed.
The analytics and A/B testing are indeed working on our development server now.
From your responses I had a few clarifications but I think that the Extensibility API in December is going to be the key to doing some of the types of testing we had to do in the past. Before I go into that, I did notice it’s a bit tricky to test the alternate content – I have to use an incognito window / another browser and hard reload to bypass caching. I am ok with that, but I was wondering if there is a way to “force” a variant e.g. can I use a query string parameter or something to force it to place my session in the “B” group?
The main objectives here with an Umbraco integrated solution for any custom test (besides having the analytics baked in) would be:
Be able to assign the visitor to an A/B group
Be able to determine which group you are in programmatically for custom / advanced testing
Ideally be able to use the split editor on a single page to store the “alternate” content without having to make document type changes for complex testing scenarios – see my example cases below.
Reduce as much as possible the need to add special code or document type changes for testing
Some of the more advanced testing scenarios we have had to deal with:
Multiple documents affected – our product detail pages (house plans in our case) all needed to change images, templates and other details as part of the A/B test. These all had one document type but were separate pages. This is where it would be ideal to have a multi-document A/B test and be able to go edit each of the pages involved and use the split editor to set up the alternate content but have it be associated with the multi-document test. It was necessary to change all of them to be visually consistent.
Changes that affect other places – e.g. when a document is queried via the Umbraco XPath queries in code, we would need to get the “alternate” version of content. For example, if we changed a single content page for an A/B test, in some places we will pull in a thumbnail or other details in other places from that content page programmatically. When we query for that document in XPath, will it pull the alternate content if you are in that test group?
Changing the template for a group of pages or the entire site
Varying content for a document type that is rendered via a partial view and surface controller
excellent questions again and I've decided to post them on this forum so others can benefit from our Q & A as well.
I was wondering if there is a way to “force” a variant e.g. can I use a query string parameter or something to force it to place my session in the “B” group?
=> You have a preview mode in the backoffice. If you click on the previewmode in the backoffice you will see the parameter that is added to the url to force this specific variant to render. The requirement is though that you're logged into the backoffice to see this alternative variant. So it depends if you want it to use it for demonstrations purposes, or that you really want to force visitors in a specific group? And why would you do that (just a question for my understanding)
Be able to assign the visitor to an A/B group
=> This is currently not possible and is not a priority at this moment on the roadmap. In A/B Testing the variant (original, variant A, variant B, variant C, etcetera) is a based on randomness (like a true A/B Test should be). We can make it extensible at a later moment where you can implement your own distribution algorithm. But I would love to understand the requirement first. Why would you do this?
Be able to determine which group you are in programmatically for custom / advanced testing
=> This will be part of the "Developer extensibility"-sprint of December and January. At that moment you can determine this from code. I will keep you posted about that!
Ideally be able to use the split editor on a single page to store the “alternate” content without having to make document type changes for complex testing scenarios – see my example cases below.
=> Can you explain this in a bit more detail? What kind of information would you like to store?
Reduce as much as possible the need to add special code or document type changes for testing
=> Yes, that is always the best!
And about your specific cases:
Multiple documents affected – our product detail pages (house plans in our case) all needed to change images, templates and other details as part of the A/B test. These all had one document type but were separate pages. This is where it would be ideal to have a multi-document A/B test and be able to go edit each of the pages involved and use the split editor to set up the alternate content but have it be associated with the multi-document test. It was necessary to change all of them to be visually consistent.
=> This can only be done with some custom coding I'm afraid. The idea of A/B Testing is to test a single element / page to observe whether your change has a significant impact. In this case multiple pages / elements will be tested at the same moment and the results will become vague. What did contribute to better results in this case?
An alternative could be to do this via personalization where you setup a segment and have an alternative view for this segment.
Changes that affect other places – e.g. when a document is queried via the Umbraco XPath queries in code, we would need to get the “alternate” version of content. For example, if we changed a single content page for an A/B test, in some places we will pull in a thumbnail or other details in other places from that content page programmatically. When we query for that document in XPath, will it pull the alternate content if you are in that test group?
=> The beauty of the uMarketingSuite is that we will take care of this for you as long as you're using ModelsBuilder! We will make sure that we grab the correct "Title" or "Image"-attribute for you. So this will hopefully work out of the box.
Changing the template for a group of pages or the entire site
=> This something you need to handle yourself at this moment. I think it could be pretty straightforward once we delivered the "Developer extensibility"-sprint. At a later moment this can also something for on the roadmap of uMarketingSuite.
Varying content for a document type that is rendered via a partial view and surface controller
=> As long as the content is filled on the document the uMarketingSuite will take care of this.
A/B Testing, Feature Roadmap, Extensibility and Technical Support
Please note: this is a set of questions I asked while doing an evaluation of this extension for our company. They requested that I post the questions and responses to this forum for others to reference.
Q1. The Single page A/B codeless test is amazing – content editors could set this up without any outside help. On the multi-page A/B test, is there any way to do the same type of content setup? For example if I wanted to enable an A/B test for a certain document type – it would be ideal to be able to add the variant content to each page of that type with the split editor and have that be included in the test. If not is this something planned for the future? I ask because there is a test we just did exactly like this on our site which required a lot of manual setup the way we do it now.
Answer: => The single page A/B test is indeed fantastic 😊. I think this the best part of the uMarketingSuite at this moment. The reason why we do not use the splitview for the multiple page setup is that I don’t exactly know how this would work. Let’s say that you selected 10 pages during the setup of the A/B Test. Would you go through all these 10 pages in splitview to adjust the content for variant B or variant C? And how good would your test be at that moment. Because you’re probably testing all kinds of different texts on different pages I don’t think your AB Test would be really reliable. The idea is that you test one thing and test whether it performs better or worse compared to the original. If you change to many variables, you cannot really interpret the results correctly on our opinion.
In a single page you can test for a specific page whether the optimization of content gives you better results. In a multipage test it’s possible to add-in additional css or javascript to do something about the template of the website.
When A/B Testing you should test one hypothesis / change per time in our opinion.
Maybe you could explain a bit more in detail what you hope to achieve. A workaround for your case could be to do some development yourself. We’re documenting & opening up the APIs of the uMarketingSuite this sprint so you can maybe use these to implement your exact requirement!
Q2. The Split URL testing feature is not listed on the roadmap that I saw – is there any rough idea when this will be available?
Answer: => We’re constantly discussing the roadmap with our uMarketingSuite partners (https://www.umarketingsuite.com/partners/). We’ve created a partnership model (https://www.umarketingsuite.com/partners/become-a-umarketingsuite-partner/) so partners can influence the roadmap, but also support the uMarketingSuite financially. We’re still a startup so we constantly have to prioritize our resource in the right way.
At this moment we do not have an exact date for the split url feature, because none of our partners saw this at a priority at the moment.
Once again; I think it can be implemented yourself once we’ve opened up the APIs, although this is probably more work than expected.
Q3. The technical documentation on programmatic extensibility is still in a to-do state – we do a lot of custom extension to Umbraco for our website so we would be really interested in any details you can provide here. Concern is both to make sure our customizations don’t break your rendering pipeline but also to see what we can do in terms of advanced tests such as the multi-page test scenario I mentioned in point 1.
Answer: => We’re documenting as much as we can do in this sprint. Also you get the possibility to create your own segment parameters. We will have this available mid-December 2020!
Q4. How is technical support handled with full licenses? Is there a separate support channel or contact point?
Answer=> At this moment it’s handled via email. Depending on the questions we can setup a meeting with the dev-team if needed.
Aaah, you even typed the answers as well :)!
Perfect! Let me know if there's anything else I can do for you!
Jeffrey
And a few more questions of Paul:
The analytics and A/B testing are indeed working on our development server now.
From your responses I had a few clarifications but I think that the Extensibility API in December is going to be the key to doing some of the types of testing we had to do in the past. Before I go into that, I did notice it’s a bit tricky to test the alternate content – I have to use an incognito window / another browser and hard reload to bypass caching. I am ok with that, but I was wondering if there is a way to “force” a variant e.g. can I use a query string parameter or something to force it to place my session in the “B” group?
The main objectives here with an Umbraco integrated solution for any custom test (besides having the analytics baked in) would be:
Some of the more advanced testing scenarios we have had to deal with:
Hi Paul,
excellent questions again and I've decided to post them on this forum so others can benefit from our Q & A as well.
=> You have a preview mode in the backoffice. If you click on the previewmode in the backoffice you will see the parameter that is added to the url to force this specific variant to render. The requirement is though that you're logged into the backoffice to see this alternative variant. So it depends if you want it to use it for demonstrations purposes, or that you really want to force visitors in a specific group? And why would you do that (just a question for my understanding)
=> This is currently not possible and is not a priority at this moment on the roadmap. In A/B Testing the variant (original, variant A, variant B, variant C, etcetera) is a based on randomness (like a true A/B Test should be). We can make it extensible at a later moment where you can implement your own distribution algorithm. But I would love to understand the requirement first. Why would you do this?
=> This will be part of the "Developer extensibility"-sprint of December and January. At that moment you can determine this from code. I will keep you posted about that!
=> Can you explain this in a bit more detail? What kind of information would you like to store?
And about your specific cases:
=> This can only be done with some custom coding I'm afraid. The idea of A/B Testing is to test a single element / page to observe whether your change has a significant impact. In this case multiple pages / elements will be tested at the same moment and the results will become vague. What did contribute to better results in this case?
An alternative could be to do this via personalization where you setup a segment and have an alternative view for this segment.
=> The beauty of the uMarketingSuite is that we will take care of this for you as long as you're using ModelsBuilder! We will make sure that we grab the correct "Title" or "Image"-attribute for you. So this will hopefully work out of the box.
=> This something you need to handle yourself at this moment. I think it could be pretty straightforward once we delivered the "Developer extensibility"-sprint. At a later moment this can also something for on the roadmap of uMarketingSuite.
=> As long as the content is filled on the document the uMarketingSuite will take care of this.
Hope these answers help you a bit,
kind regards,
Jeffrey
is working on a reply...