Documenting your AB Email Testing

[chriskowalczyk9]

Hi All,

I'm giving this another go-round here, but I'm looking for help on best practice documentation for Email AB testing. If anyone can help, I would definitely appreciate it!

Here is my problem:

I want to start performing email testing on a routine basis, but I don’t want to do a test just based on gut-feel. I want to be able to say, “Based on ABC, we should test by doing XYZ.” From that point, I want to specifically document everything prior to performing a test, and this is what I need help with.

I can ask the question a different way by saying, what do I need to consider before performing the test? From there, I can decide what to document.

Based on the example of a Subject Line test, I would have something like this:

1. Test Type
   1. Subject Line Test
      1.      Should it be a standard AB Subject line test or a 50/50 Random Sample?
         
         
2. Test Variables
1. Open Rate
   1.      What to include for a subject line
      1. What was it before?
      2. What do we want to do differently?
         
         
3.  Hypothesis
   1. Open Rate
      1. I think that using an emoji at the front end of the subject line will increase the open rate vs. not having one at all
         
         
4. Audience
   1. The minimum amount of audience needed to test a subject line is ‘XYZ’
      1. How big/small is our audience?
      2. If audience is too small, what to do next?
      3. What type of audience is it?
         1. Industry, Decision Makers, etc.
            
            
5. Test Length
   1. How long should we test the theory?
      1. Best practice for Subject line test length?
      2. Send 1-3 Emails?
      3. A time span of 1 month?

So using this example, what questions would I be missing? Do you see a better way of documenting this? A different layout? Using Word vs. Excel vs. Powerpoint? Is there a best practice way of capturing this information that I’m not thinking about?

Again, any help would be appreciated! Looking to nail this down!

You can see my previous post that didn't really provide the answer I was looking for here (I think I worded the question incorrectly) - https://nation.marketo.com/t5/product-discussions/tracking-and-logging-your-a-b-tests/m-p/152507#M10... 

Please let me know what you think!

Cheers,

Chris

[Michael_Florin]

I remember having done this in project management tool. We had set up the respective fields - Test Scenario, Hypothesis, Long Term Impact, Future Actions, Success/Failure - and asked test requesters and executors to fill these fields.

But eventually the tool doesn't matter, I think. What really matters is that you consistently do it, and share the knowledge among anyone who should know. That's the real challenge. 🙂