Press Ctrl / CMD + C to copy this to your clipboard.
This post will be reported to the moderators as potential spam to be looked at
Please note: this is a set of questions I asked while doing an evaluation of this extension for our company. They requested that I post the questions and responses to this forum for others to reference.
Q1. The Single page A/B codeless test is amazing – content editors could set this up without any outside help. On the multi-page A/B test, is there any way to do the same type of content setup? For example if I wanted to enable an A/B test for a certain document type – it would be ideal to be able to add the variant content to each page of that type with the split editor and have that be included in the test. If not is this something planned for the future? I ask because there is a test we just did exactly like this on our site which required a lot of manual setup the way we do it now.
Answer: => The single page A/B test is indeed fantastic 😊. I think this the best part of the uMarketingSuite at this moment. The reason why we do not use the splitview for the multiple page setup is that I don’t exactly know how this would work. Let’s say that you selected 10 pages during the setup of the A/B Test. Would you go through all these 10 pages in splitview to adjust the content for variant B or variant C? And how good would your test be at that moment. Because you’re probably testing all kinds of different texts on different pages I don’t think your AB Test would be really reliable. The idea is that you test one thing and test whether it performs better or worse compared to the original. If you change to many variables, you cannot really interpret the results correctly on our opinion.
In a single page you can test for a specific page whether the optimization of content gives you better results.
When A/B Testing you should test one hypothesis / change per time in our opinion.
Maybe you could explain a bit more in detail what you hope to achieve. A workaround for your case could be to do some development yourself. We’re documenting & opening up the APIs of the uMarketingSuite this sprint so you can maybe use these to implement your exact requirement!
Q2. The Split URL testing feature is not listed on the roadmap that I saw – is there any rough idea when this will be available?
Answer: => We’re constantly discussing the roadmap with our uMarketingSuite partners (https://www.umarketingsuite.com/partners/). We’ve created a partnership model (https://www.umarketingsuite.com/partners/become-a-umarketingsuite-partner/) so partners can influence the roadmap, but also support the uMarketingSuite financially. We’re still a startup so we constantly have to prioritize our resource in the right way.
At this moment we do not have an exact date for the split url feature, because none of our partners saw this at a priority at the moment.
Once again; I think it can be implemented yourself once we’ve opened up the APIs, although this is probably more work than expected.
Q3. The technical documentation on programmatic extensibility is still in a to-do state – we do a lot of custom extension to Umbraco for our website so we would be really interested in any details you can provide here. Concern is both to make sure our customizations don’t break your rendering pipeline but also to see what we can do in terms of advanced tests such as the multi-page test scenario I mentioned in point 1.
Answer: => We’re documenting as much as we can do in this sprint. Also you get the possibility to create your own segment parameters. We will have this available mid-December 2020!
Q4. How is technical support handled with full licenses? Is there a separate support channel or contact point?
Answer=> At this moment it’s handled via email. Depending on the questions we can setup a meeting with the dev-team if needed.
Aaah, you even typed the answers as well :)!
Perfect! Let me know if there's anything else I can do for you!
And a few more questions of Paul:
The analytics and A/B testing are indeed working on our development server now.
From your responses I had a few clarifications but I think that the Extensibility API in December is going to be the key to doing some of the types of testing we had to do in the past. Before I go into that, I did notice it’s a bit tricky to test the alternate content – I have to use an incognito window / another browser and hard reload to bypass caching. I am ok with that, but I was wondering if there is a way to “force” a variant e.g. can I use a query string parameter or something to force it to place my session in the “B” group?
The main objectives here with an Umbraco integrated solution for any custom test (besides having the analytics baked in) would be:
Some of the more advanced testing scenarios we have had to deal with:
excellent questions again and I've decided to post them on this forum so others can benefit from our Q & A as well.
=> You have a preview mode in the backoffice. If you click on the previewmode in the backoffice you will see the parameter that is added to the url to force this specific variant to render. The requirement is though that you're logged into the backoffice to see this alternative variant. So it depends if you want it to use it for demonstrations purposes, or that you really want to force visitors in a specific group? And why would you do that (just a question for my understanding)
=> This is currently not possible and is not a priority at this moment on the roadmap. In A/B Testing the variant (original, variant A, variant B, variant C, etcetera) is a based on randomness (like a true A/B Test should be). We can make it extensible at a later moment where you can implement your own distribution algorithm. But I would love to understand the requirement first. Why would you do this?
=> This will be part of the "Developer extensibility"-sprint of December and January. At that moment you can determine this from code. I will keep you posted about that!
=> Can you explain this in a bit more detail? What kind of information would you like to store?
And about your specific cases:
=> This can only be done with some custom coding I'm afraid. The idea of A/B Testing is to test a single element / page to observe whether your change has a significant impact. In this case multiple pages / elements will be tested at the same moment and the results will become vague. What did contribute to better results in this case?
An alternative could be to do this via personalization where you setup a segment and have an alternative view for this segment.
=> The beauty of the uMarketingSuite is that we will take care of this for you as long as you're using ModelsBuilder! We will make sure that we grab the correct "Title" or "Image"-attribute for you. So this will hopefully work out of the box.
=> This something you need to handle yourself at this moment. I think it could be pretty straightforward once we delivered the "Developer extensibility"-sprint. At a later moment this can also something for on the roadmap of uMarketingSuite.
=> As long as the content is filled on the document the uMarketingSuite will take care of this.
Hope these answers help you a bit,
is working on a reply...
Write your reply to:
Image will be uploaded when post is submitted