The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?

[Victor_Herrero]

Hi,

This is almost a philosophical question on the origin of things...

I have a Smart Campaign that looks something like this:

Smart list

Trigger: Person is created

Filter: Country is xxxx

Filter: Person is NOT member of "Smart List yyyy"

Flow

Send email zzzz

My question would be: when a person is created, does it get added to my "Smart List yyyy" fast enough to NOT qualify?

My intended behavior is that they don't qualify. I want to use this method to handle exceptions but I need to know if it works and if so, it it is reliable.

I could always let them qualify and remove them from the flow if they belong to the list, but I have a feeling its more convenient, better for MKTO performance and perhaps more maintainable to do this in the smart list rather than inside the flowsteps.

Thanks!

[Ronen-Was-SRpro]

HI Victor,

If the person doesn't exist already in the list I don't think it will work however what you can do is:

1. Smart campaign:

Trigger: Person is created

Flow: Wait a few minutes.

Request campaign.

2. Smart campaign:

Trigger: Campaign requested

Filter: Country is xxxx

Filter: Person is NOT a member of "Smart List yyyy"

Flow: Send Email

[SanfordWhiteman]

If the person doesn't exist already in the list I don't think it will work

It's a Smart List, though. So "membership" in the list is dynamic; there's no separate step to add to the list.

The better way to phrase the question is "does the person match the filter parameters of the Smart List?" And that actually depends on what those parameters are. For example, if you trigger a webhook when somebody comes in, then the fields updated by the webhook may be updated a second or so later. If in the meantime you see if they're a member of a Smart List that filters on those field values, they may or be not be at that time (thus the outcome is unreliable). But a filter on data that already exists will find them.

[Floyd_Alvares2]

Just adding my 2 cents:

"My question would be: when a person is created, does it get added to my "Smart List yyyy" fast enough to NOT qualify?"

When a member of a smart list filter (A) is referenced in a smart list within a smart campaign (B) or within another smart list (C). Then when B or C are executed, they will filter of each of the fields/criteria within the other smart list(A). So it is not entirely accurate to check if they would qualify fast enough in the smart list.

However, the chicken and egg or in my terms we refer to them as the race condition will be needed to be taken into consideration.

Example: a record may fill out a form on the website where the record populates certain field information (example country). Now if you run a smart campaign with the trigger "Fills out Form + Country value = YYYY". There will be a race condition between the trigger (Fills out Form) which is based on an activity in Marketo versus the update condition that Marketo is updating a record in Marketo (which is not based on the activity). However both these actions will happen at the same time.

The same logic applies for filters like Data Value Changes and Person is created.

The solution to this would be to either run batch campaigns for actions that do not need immediate effect OR leverage the solution recommend by Ronen Wasserman

I hope this made sense

Thanks

Floyd

[SanfordWhiteman]

To be clear, in this particular case there's not a race condition.

If the Smart List qualification filters are based on data values that exist on the lead at the point of entry into the system (i.e. form fields in the same form post that is their Web Form Fillout source on a ​New Person ​activity) they will be members of the SL at the time the Person Created trigger fires.

[Victor_Herrero]

I see what you mean.

As long as the source form includes the filters that you are using on the filter smart list, the trigger will fire correctly, because the necessary information will exist on the record on creation.

You seem to be 100% sure about the speed of dynamic Marketo lists, but are all records constantly being added to and removed from smart lists on creation / update instantaneously?

Is marketo THAT fast? Is it constantly re-evaluating membership for all records on each update?

I didn't want to reveal / bore with the details of the campaign since I also wanted an answer that will help me in the future as well, but in this particular case, upon creation, all information needed for the Trigger and the Smart List is there.

So the setup should be reliable.

[SanfordWhiteman · Accepted solution]

but are all records constantly being added to and removed from smart lists on creation / update instantaneously?

It's not about "adding" or "removing" in any tangible sense. It's about whether queued-up database updates have been committed (you can think of "committed" as meaning "hardened" or "finished" or "made available to other processes") before the same database is searched by another process.

And in this particular case they have. In many other cases, the order of updates and searches is unreliable. This simply isn't one of them.

P.S. There's a fascinating world out there of newfangled database locking and concurrent access methods. These are basic building blocks of systems design studied since the 1960s, but people have gone back to the drawing board recently with the explosion of internet traffic and the need for better and better multi-user performance.  Locking is made much more complex w/distributed systems (i.e. more than one server, as any large organization has, and more than one client), and distributed systems may accept being loosely consistent, meaning every server does not have the same data at every point in time. But again: not to worry in this case.

[Victor_Herrero]

Ok, so after every update / creation, first smart list qualification is determined / fixed and then all other search processes take place.

When it comes to smart list memberships, I find it fascinating that there is no actual field in Marketo's own database (or each of our instances) where for each record there is a "true/false" value somewhere.

How does it really work?

[SanfordWhiteman]

I find it fascinating that there is no actual field in Marketo's own database (or each of our instances) where for each record there is a "true/false" value somewhere. How does it really work?

It works like any database search query.

For a greatly simplified example, in SQL

select id from users where filter1 = 'something' and filter2 = 'something else'

returns a result set with the matching user IDs at that point in time.  It doesn't write anything back to the users table. (The database user's permissions need not even allow it to update anything at all, just select.) The same query issued again can have a different result set.

Yes, caching heuristics and intermediate tiers in many systems may store and reuse the entire result set for a certain period: "until the underlying table/s are known to have been updated" is the lossless way, while "until a fixed number of seconds have gone by" is the lossy but sometimes necessary way.  (Try as you might to avoid the latter by tweaking indexes, analyzing query plans, etc., you'll end up caching something somewhere in high-volume and/or legacy environments.)

But regardless of whether caching is employed, it would never be feasible to store on the lead level whether each person matched each query of an effectively infinite number of queries. Some architectures flatten (denormalize) known-interesting data back to the individual record level. Indeed, that's a form of storing somebody's match-iness back on the record, and that's how Marketo segments work.  But you could never do this for all possible queries.

[Victor_Herrero]

So you are saying that each smart list is basically a query result that gets updated every time the trigger is triggered, and until all query results have been delivered (i.e. first name is changed, so all smart lists and processes that are using that as a trigger are re-queried and the result returned) all other processes are on hold?

returns a result set with the matching user IDs at that point in time.  It doesn't write anything back to the users table. (The database user's permissions need not even allow it to update anything at all, just select.) The same query issued again can have a different result set.

you mean since its a query you don't need to write, you are just reading?

Yes, caching heuristics and intermediate tiers in many systems may store and reuse the entire result set for a certain period: "until the underlying table/s are known to have been updated" is the lossless way

I take it Marketo operates like this most of the time with its smart lists right?

But regardless of whether caching is employed, it would never be feasible to store on the lead level whether each person matched each query of an effectively infinite number of queries. Some architectures flatten (denormalize) known-interesting data back to the individual record level. Indeed, that's a form of storing somebody's match-iness back on the record, and that's how Marketo segments work.  But you could never do this for all possible queries.

So marketo segments DO generate a field on the records that stores membership data... this paragraph I don't understand too well.