champion/challenger emails within an engagement program

[Anonymous]

Hi there, I have a few questions about email testing. 

I've read how to set up a champion/challenger email test by subject line. I'm interested in setting this up for all the emails in one of my engagement programs. Do I just follow the steps outlined on this page (see link below) for each email in the program and then drag the emails into the stream? Or do I need to do something else for this to work in an engagement program? 

https://community.marketo.com/MarketoArticle?id=kA050000000LGSqCAO

When you set up the Distribution, it seems this is dictating the percentage of leads that get each version of the email. So if you set up 20% challenger and 80% champion, and you add 100 leads to the engagement program, 20 would get the challenger, 80 would get the champion. 

Is there a way to set this up so instead Marketo sends the challenger to say 20% and the champion to 20% and then whichever version wins, Marketo sends to the remaining 60%? Maybe this is only possible with single regular emails, not engagement programs, since engagement programs have set cadences? 

Last thought, if the subject line testing works like I explained two paragraphs up, how do you take advantage of the testing results? Do you just evaluate who wins a round and then, based on the outcomes and the sample size, you might choose to run with the winner a larger percentage of the time for a while and see how things go? I imagine you're looking for trends over time but in my experience testing results don't often yield clear trends. I'd be curious to hear some of your best practices for interpreting and acting on testing results.

Thanks,

David

 

[Josh_Hill13]

My understanding of this is that it will send 15% of the list the challenger item (or items) and then send the rest of the list whichever email wins. It kind of assumes you have the first email as the "champ" or original item. So it may be a good idea to run that "Champ" first, then come back in a couple of weeks and setup the challenger system.

In this situation, you will run the Champ until you have more challengers. over the long run, you may (or may not) receive a boost with whichever email wins the tests. I'd keep a sheet to monitor this over time. The idea is to keep testing and incrementally increase the win rate for whatever you want: opens, clicks, fill outs.

[Anonymous]

Thanks Josh. Always helpful. I appreciate it. 

To implement the test, do I just follow the steps outlined on that page (see URL from my previous post) for each email in the program and then drag the emails into the stream? Or do I need to do something else for this to work in an engagement program? 

[Mark_Farnell]

Note to Josh - is that true for emails in engagement as well?  I thought that was only relevant in the Email Send campaign, which you can't use in Engagement.  And that in other testing you have to manually declare a champion for future sends to get that champion.

I've posted a discussion here about how we're able to test champion/challenger on batch campaigns in engagement programs.  Initially we seeing success - would appreciate the community's views as well. Champion/Challenger Batch Solution #lessonslearned