How to temporarily suppress an external list against a single program

[SanfordWhiteman]

Right, that's how it works.

It's not called for every lead in the suppression list but for every lead in your database that's in danger of receiving an unwanted send.

[Anonymous]

Got it. So given that a typical email deployment will often go to thousands of names, and given that the size of the suppression list(s) are in the thousands (if not more), would you agree that this proposed process will be hugely inefficient and resource-intensive, particularly for the server that would host the endpoint for the webhook?

I guess what we're hoping for is a process that can make the suppression list available to Marketo in some kind of a batch, but it's sounding as though that may not be possible.

[SanfordWhiteman]

would you agree that this proposed process will be hugely inefficient and resource-intensive

Well, no, I wouldn't agree with that at all.

Scanning a list of 1,000,000 names in memory is child's play. The webhook will still complete at just over wire speed (~100ms), and the endpoint server should barely see a blip.  In all, this is lightweight stuff. The only reason it might seem heavy is a number like "1 million" does still mean something when it comes to long (and definitely long and wide) database tables or on-disk spreadsheets, but it's negligible when you're talking about a match against a cached array of names.

Even a seeming batch process is still not an "all at once" process, that is, it's ultimately going to match each name against the list so it's going to be dependent on the speed of an individual lookup. Though it could be highly parallelized on the same machine in a way that you can't as easily do over a network.

[SanfordWhiteman]

P.S. Ran a benchmark against a static list of 1,000,000 random email addresses (average length 28 characters). I made sure the email address I was looking for was not found (thus full scan) and didn't add an early exit as you'd be able to do with a pre-sorted list.

Results in milliseconds across 20 runs x 1,000,000 names:

     24 21 23 22 24 32 26 31 22 21 21 23 23 24 21 24 27 24 22 19

with an exit halfway through the list it was linearly faster:

     15 10 16 10 11 13 12 12 13 18 13 9  9  11 12 9  13 15 12 11

So I'd definitely be confident about running this over tens of thousands of names. I mean, of course it would be better to filter the names over time, the best validation is the one you don't need to do, etc. but I wouldn't classify it as inefficient.

(I used FlowBoost/JS to run the test but you could run it your browser or in any other language, too.)

[Ed_Burton1]

Hi Sanford Whiteman,

I'm reading through your explanation and I want to try this in our instance. We have a excel file with about 200,000 inactive or "dead" emails. I'm thinking we can use your approach to setup a simple webhook to filter out these bad emails from all of our campaigns or programs.

I'm new to Marketo, but have some programming knowledge. My question is, how did you setup the web server with the "Inactive emails" and do the comparison? Did you do that on your server or within Marketo? I think I could wright a little script that would take 10the emails from Marketo, 2)compare those to the inactive list, 3)then update a field in Marketo with "Good" or "bad", and 4)remove those now identified "bad" emails from the smart list/program. Am I paraphrasing you correctly? 

Right now, we use excel to locate and remove all the bad email on each and every list import...kind of a pain. Any thoughts?

Thank you in advance for the help!

Ed.

[SanfordWhiteman]

Did you do that on your server or within Marketo?

The comparison is done on the remote server. You pass the {{Lead.Email Address}} in the payload (JSON-encoded POST in this case, but you can use anything).

2)compare those to the inactive list, 3)then update a field in Marketo with "Good" or "bad", and 4)remove those now identified "bad" emails from the smart list/program. Am I paraphrasing you correctly?

Yes, the remote process looks up an email (since webhooks fire for one lead at a time) in the sheet. You might choose to cache the sheet in memory or read it from disk each time -- benchmark it to see if it matters.

[Grégoire_Miche2]

Create a static list, name it whatever you want. for instance "Customer X list"

Then create a smart list with the following filters:

lead was created - constraint: list is Customer X List.

This smart list will five you the leads that where unique to you customer and created when you ran the import.

When the email is sent, delete the members of this smart list, not the members of the list.

I prefer to do this in individual programs, not in the (lead) database. I prepare a program template with all the element above + the email + a smart campaign to send it (or else use an email program type), + another smart campaign to delete the leads, so everything is self contained.

Each time we have to run the operation for a customer, we clone the program template, prepare the email, import the data and have the smart campaign send the email. We name the program by the customer or lead source name and the date.

and of course,  +1 on the comments on the performance of the whole operation if you are in the millions of lines....

-Greg

[Brent_Baker1]

We have a customer with a list they want suppressed that is over 95k contacts. We only pay for <60k contacts so there is no way can we import the list so here is what I did.

1. Export your entire list.
2. Copy email addresses to a clean excel document from both your list and the suppression list.
3. Find Duplicates and show with highlight.
4. Filter by only showing the highlighted records.
5. Import this list into a static list.

There you go, you have a list to suppress that only contains the contacts that were already in your database.

[SanfordWhiteman]

This is definitely a good call, Brent, if your Marketo database doesn't allow for leads to be created via any means other than pre-certified list upload. But if forms or any other self-service feature is in use, or any dynamic CRM lead sync...