A/B Test in Engagement Program not sending Challengers

[Anonymous]

I have an A/B champion test on subject lines set up for one of my email programs inside an engagement program. 

We set it live this weekend where those who fill out a form are triggered to enter the stream. My cadence is set for every day and I've recieved over 40 people this weekend to enter the stream.

So far my results are showing that the program is only sending the champion email and no challengers. I have not declared a winner yet.

Any reason why this may be happening?

[Anonymous]

Hi Anthony,

If you're using programs in engagement streams I can only imagine the complexity to properly champion challenger for Marketo. It would be wonderful to have, but hard to build out for them.

Looks like we will need to do our reporting outside the "out of box" functionality of champion challenger.

Im always in favor for more robust documentation. Marketo cant do it all though right?

Post your support update, be curious to hear what they say. 

[Anonymous]

Well it seems pretty straightforward. I created a Smart Campaign that sends the email within the program as instructed in the product docs when you want to use Programs instead of individual emails in Engagement Programs. When I export the report of the Champion/Challenger it shows that all the emails were sent for the Champion. It literally skipped the challengers. I would think in the back end programming of Marketo they could easily fix this with some type of command in the code to split the list into the various percentages as specified in the Champion Challenger.

I guess another alternative if I wanted to use Programs within Engagement Programs would be in the Flow to do a Random sample 50 percent and have it send the two emails. Reporting-wise it would be more manual but I would see the two subject lines in the report. Then I could track how it is performing an manually remove one of the emails. Not as easy as just "declaring a winner" but a workaround nonetheless.

I think it was an oversight on their part when they created Champion/Challenger. The documentation says clearly: "Champion/Challenger emails only work with trigger campaigns and engagement programs. For batch mailings, use the email program's A/B testing functionality."

[Anonymous]

My champion/challenger has been sending out correctly. From my understanding you must place the direct email in the stream and it cannot be within a program. 

I also have no cadence set for this champion.challenger as I also hear it must be triggered instead of batch sent. 

If the above meets your campaign setup and you still  have an issue, I would open a ticket with support. I have yet to hear on a final OK this has been fixed. 

[Anonymous]

I have the same issue you had with Champion Challenger. It seems you can't use a Program within the Engagement stream. You have to use the individual emails in order to use the Champion Challenger functionality. It didn't work initially but then I tested it with just the email itself in the stream and it worked. But nowhere does it say that it wouldn't work in the documentation. I opened a ticket with Support.

[Dory_Viscoglio]

We also had a champion/challenger test that didn't go out (properly approved) as scheduled on July 15. Cheryl, can you confirm if this has been fixed? 

[Anonymous]

I'm running into the same problem. All tests approved, and I'm receiving the emails to give me an update, but none of the challengers are being sent. Is there something that Support can do to fix this?

It would be very helpful to be able to A/B test within engagment programs.

[Anonymous]

Don't think so. I'm not experiencing issues with my engagement program not sending emails to those added to the stream....

[Anonymous]

Has this issue been resolved?  We did a champion challenger email on 7/15 and it doesn't look like the challenger was sent. 

[Anonymous]

Thank you Cheryl. I appreciate the update.

[Anonymous]

I have confirmed that there is a bug.  Engineering is working on a fix - it might take a day or so to fix, test and push it out.

[Anonymous]

Called in a ticket. Still waiting to be resolved.

From diving in further, the engagement program is not correctly sending the champion challenger...

[Anonymous]

Hi Timothy,

It might be best to call into Marketo support to see if they can help.  

Thanks,
Cheryl

[Anonymous]

Its just one email split into an A/B subject line test (2 competitors). Everything has been approved. My send sample test properly sends all 3 variations. 

[Dory_Viscoglio]

Timothy, has everything been approved? Both of the emails as well as the program itself need to be approved in order for the send to work properly.

[Anonymous]

25% to the Champ and 75% to the Challengers 

[Anonymous]

What percentage do you the Champ versus Challenger set to?