SOLVED

Re: A/B Test Day of Week

Go to solution
davcamp22
Level 1

A/B Test Day of Week

I just want to see if an email performs better on one day versus the next. Here is what I did: 

Screen Shot 2022-02-28 at 5.40.56 PM.png

Will this send 50% on one day and 50% the next?

1 ACCEPTED SOLUTION

Accepted Solutions
Katja_Keesom
Level 10 - Community Advisor

Re: A/B Test Day of Week

Hi @Darshil_Shah1 , the DB size will not change, as the audience will be picked the moment the smart campaign is run and will not be evaluated again after the wait step. Your comment about the random samples being disconnected from eachother is totally true though, so this flow will indeed not work.

I would replace the third flow step with a remove from flow if the email was already sent and then send the email on day 2 to all people remaining in the flow.

View solution in original post

8 REPLIES 8
Darshil_Shah1
Level 10 - Community Advisor

Re: A/B Test Day of Week

Random Sample will randomly pick 50% of the qualified leads and put them through the respective flow step (in your case Send email) - It is not guaranteed that the set of leads chosen randomly after a day's of wait step in flow step 3 would be mutually exclusive with the set of leads Marketo chose in the flow step -1 - you can remove people based on the email send activity prior to the next send the email send step from the campaign after a day's wait step.

If you're using Email Send program you may wanna use “Date/Time” A/B Testing

Katja_Keesom
Level 10 - Community Advisor

Re: A/B Test Day of Week

Hi @Darshil_Shah1 , the DB size will not change, as the audience will be picked the moment the smart campaign is run and will not be evaluated again after the wait step. Your comment about the random samples being disconnected from eachother is totally true though, so this flow will indeed not work.

I would replace the third flow step with a remove from flow if the email was already sent and then send the email on day 2 to all people remaining in the flow.

Darshil_Shah1
Level 10 - Community Advisor

Re: A/B Test Day of Week

Ahh, yes that is true - smh missed that! Thanks, @Katja_Keesom!

davcamp22
Level 1

Re: A/B Test Day of Week

Hi there,

Still working on this... if you have a second is this what you meant? When waiting one day it will know to activate the other 50%? Seems like I'm missing something?

Screen Shot 2022-03-10 at 4.51.09 PM.png

Katja_Keesom
Level 10 - Community Advisor

Re: A/B Test Day of Week

You need to set your Default choice to Do Nothing (remove the value that is in there currently), otherwise everyone will be removed from the flow.

After that last flow step, you do still need to send the email to all people who are left in the flow, so a simple send email without any further choices.

davcamp22
Level 1

Re: A/B Test Day of Week

Thanks so much for your help. I think I got it.. this is where i'm at. let me know if this is what you were talking about? 

 

day of week flow.png

 

Darshil_Shah1
Level 10 - Community Advisor

Re: A/B Test Day of Week

Looks good!

EllenSchwier
Level 4

Re: A/B Test Day of Week

For easy visibility on the results of the test, the standard A/B test on date/time in an email blast program is really good.  But I agree with the others on everything else!