A/B Testing

[Anonymous]

If i want to A/B test two different emails, how do i send it to only 50 people? I have a database that is 60,000. Can i use decimal percentages? (ie. 0.083 for 50 out of 60,000)

[Anonymous]

Andrew,

First, I don't think you can go below a full decimal point in Marketo (1%). You'll need to manually create a static list then manually add names (if you're only doing 50, I'd add the second and fifth entries per letter of the alphabet)

Second, let me suggest that an A/B test divided among 50 users is very, very small and the results are not likely to yield much relevant value.

Prior to getting Marketo, we used to subject line testing using groups of between 100 and 250, think that would for for us (we also did not have much of a choice due to a limited data pool).

With Marketo we've learned that to get a decent staistical indicator we need to come in at around 2000 messages per option tested. Based on those results we test with groups of 5-7k more, which then tends to give us some staistical relevance. Hope this helps.

[Anonymous]

I second Keith's recommendation regarding the size of the test group. With an A/B split you'd want at least 4000 people total in the recipient group.

In terms of the list, you could manually add to a static list, as suggested, or just keep adding segmentation criteria to a Smart List until you're down to the desired count. Just be sure that the send count on the campaign as viewed in the "Run" screen matches your total desired count, as various factors can make the acutal send number differ from the Smart List total count.

Happy testing!