Random Sample Failed

[michellechopin]

Hello Community!

I'm working with one of our Program Managers to send out a multi-email campaign that has testing on every email going out (either subject line or whole email). The first two emails have gone out (one last week, one today) and when I run the report to see the two emails (and determine the winner), I only see that Version B has gone out to the Random Sample but no Default option (Version A):

Here's how I set up the Flow Step in the campaign (I do this regularly for multi-email campaigns and have done nothing different!), and I can't figure out why only the Random Sample sent out but not the Default:

This is a new campaign in a new workspace but the leads it's going to were all uploaded to the related partition, so I don't think it's that. This is the first time I've had an issue using Random Sample so I'm at a loss as to what the issue is but I'm not able to provide testing results to my PM because the test hasn't worked as intended!

At worst, I can create individual email programs for each email test send, but I'd rather keep the program as clean as possible by using Batch Smart Campaigns to do these tests (the manual review of the test isn't a problem).

Any insight or suggestions are much appreciated 😊

Michelle

[Darshil_Shah1]

It'd help if you could confirm a couple of items -

1. Do you see the send email activity for email 02a (the one in the default choice step) in the campaign's result?

2. Did all the leads receive the b version or just the 50% of the audience were sent the b version and the rest were not sent any of the versions?

[michellechopin]

Hey @Darshil_Shah1 ,

Always happy to give more insight!

1. I see no send activity for Email 02a (or Email 01a) in the campaign results. Just the Send Email and Email Delivered details for Email 02b (and Email 01b)
2. Only the 50% assigned by the random sample got an email; the other 50% got nothing

It's like the campaign completely ignored the Default step after it assigned the random sample - I've never seen that before, which is why this is entirely baffling to me!

Let me know if I can share anything else to help uncover this Friday mystery!

Michelle

[Darshil_Shah1]

@michellechopin, thank you for the details! 

This is really strange - as long as the campaign's flow step was set as per the snapshot that you have shared in your post, I can't think of a reason where-in the entire 50% of the random sample identified for the default choice wouldn't be sent out the email!

You could try creating a SL with the exact same criteria that you've got in your campaign, and include a filter in AND saying to only include people who are not member of the send email campaign. This should roughly give you the list of 50% of the people who were didn't flow through this campaign (until a lot of original audience got updated such that they don't qualify for the filter criteria now). Once you have the list of people in the SL, maybe you can just open the up a few of the leads and see if all's good with them (do they really still exist in the db, were they Marketable during the time of email send, etc).

Please let us know if you have any questions! We'd be happy to help as much as we can to get to the bottom of this.

[Katja_Keesom]

If email 2b actually went to the intended 50% sample, I expect the other 50% actually are member of the smart campaign, otherwise the audience size doesn't make sense. I would also check in the audit trail whether the smart campaign was changed aound the time of sending and whether email 2a was actually approved at the time of sending. This is indeed very strange behaviour, so checking the obvious is necessary I'm afraid...

[michellechopin]

Thank you both for your insight/suggestions, @Katja_Keesom and @Darshil_Shah1!

With those, here are some updates (this covers both email sends):

• Checked the Campaign Members tab and all 693 people are showing in there (no one is left in the Qualified People tab); this is also confirmed on the summary page of the Smart Campaign too
• Ran a smart list that included leads who were in the send list and those who are NOT in either of the send campaigns - list comes back blank (makes sense given that all 693 people are listed in the campaigns tab for each)
• Ran a smart list that included leads who were in the send list and did NOT receive ANY of the four emails (because if I just run it as not receiving Email 1a/2a, then it pulls in everyone) - there are 402 people showing (so not quite 50% but there could be other factors in the 50% being off, like unsubscribes, marketing suspended)
• Emails used were all approved before the campaign was finished/set live

I took a break from this while in a meeting and came back to my Smart List to see who didn't receive any emails.... for some reason, duplicates were created of most/all leads without email addresses...I worked with two uploads and looking at those files every single contact has an email address so I have NO idea what happened upon upload (there's nothing showing in the audit other than the list upload itself).

Is there a chance that Marketo managed to randomly select everyone who had an email address, leaving the other ~50% as those that didn't? If so, that's likely what caused the test to fail here.

I'm just double checking my lead list and will set the next email to go out as a normal test with the issue resolved, and see if the test works this time.

THANK YOU for the note to check the obvious - it's usually something like this when I have an issue (thank goodness, as it's an easy fix!).

Michelle

[Darshil_Shah1]

So I think people who didn't get either of the emails were bad records without an email address value. I'm not totally surprised since a bad CSV file can cause such mishaps while importing people into the list. I've witnessed a case where-in a list upload with mere 10 records caused an import of 1900 records (of course the CSV file was bad, and 1890 of all the records didn't have any field/value in them, just the lead id being assigned by the system).

Also, Marketo's send email flow step filters people without an email address before sending out the email, though people would still be member of the campaign and would flow normally through the rest of the flow steps in the campaign if any, e.g., CDV, Change Score, etc. However, it's highly unlikely that the random sample randomly selected all the people with email address value for sending the email 02b in choice step 1.

A quick round of testing confirmed that Marketo creates random samples before blocking non-Marketable people (unsubscribed, empty email address value, marketing suspended, etc.). Albeit all of this, seeing 0 sends of the email in the default choice is something that I still keep thinking about (assuming those people were not deleted from the DB altogether making them not appear in the campaign results/members list). I'll try to dig this up more tomorrow and run more tests to check if I'm missing out on anything here.

[michellechopin]

Hey @Darshil_Shah1 ,

You are right about those people not getting the emails as having bad/missing data - that's how I noticed that email addresses were missing because I ran the smart list to see who had gotten NO email at all, and only those people showed up.

Turns out that the error wasn't on the .CSV upload itself but the field mapping - for some reason we have two email address fields, E-mail and Email Address, but only use the latter in Marketo. However, the download from CRM uses E-mail as the column header and I didn't catch it on the mapping screen so Marketo ignored the data in that field and uploaded everything else. Then when I did a second upload a few days later to include some new leads, I properly mapped the Email Address field that time, but Marketo had nothing to match the leads with, creating all the duplicates.

A great example of why data/field integrity/planning is so important and a bad example of the human error that can happen when manually importing lists (which I really dislike doing for that reason).

To your point about Marketo creating the random sample before blocking people, that does indeed continue the mystery as to why only those with email addresses got Email B and none got Email A - it seems like a very unlikely scenario (but I have bought a lottery ticket today just in case 😋). I can confirm that those people were not deleted before the email campaigns ran (they showed in the Qualified Members and the Campaign Results tab), but since discovering the issue, I've deleted those leads without email addresses from the database as the list is being used in two more email sends this week and next (and I didn't want anymore duplicates in our database than we already have!).

I never intend to find these mysteries but when I do, it always sends me down a rabbit hole of trying to figure things out! I appreciate your testing to answer some of the questions we have and if you learn anything more then let me know 😀

Michelle

[Darshil_Shah1]

Gotcha, having multiple email address fields sounds scary (you could possibly run into countless such issues)! I'd suggest planning to map them together and build workflows in CRM and/or Marketo around it to achieve/do what you're currently doing using two Email Address fields. And yes, thank you so much for keeping us updated and confirming items as you continue to dig into this more in your instance.

To your point about Marketo creating the random sample before blocking people, that does indeed continue the mystery as to why only those with email addresses got Email B and none got Email A - it seems like a very unlikely scenario (but I have bought a lottery ticket today just in case 😋). I can confirm that those people were not deleted before the email campaigns ran (they showed in the Qualified Members and the Campaign Results tab), but since discovering the issue, I've deleted those leads without email addresses from the database as the list is being used in two more email sends this week and next (and I didn't want anymore duplicates in our database than we already have!).

I never intend to find these mysteries but when I do, it always sends me down a rabbit hole of trying to figure things out! I appreciate your testing to answer some of the questions we have and if you learn anything more then let me know 😀

Michelle

---

I'd really like to bring this to the notice of the Product team when I get a chance! I shall update this thread in case I get any updates/insights from them. Till then, good luck with your lottery ticket! 🙃

[michellechopin]

@Darshil_Shah1 We're currently in the process of launching a new instance of Marketo and a HUGE part of it has been cleaning up fields and ensuring that we're only using one per applicable data point. It's been a long, but fulfilling and insightful exercise and we're looking forward to presenting the team with cleaner data, easier-to-find data, and data they can rely on.

While I did buy lottery tickets, no big win (yet 😉).

Thank you again for your insight and always being quick to help out,

Michelle