18 Replies Latest reply on Dec 7, 2018 7:05 PM by Sanford Whiteman

    The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?

    Victor Herrero

      Hi,

       

      This is almost a philosophical question on the origin of things...

      I have a Smart Campaign that looks something like this:

       

      Smart list

      Trigger: Person is created

      Filter: Country is xxxx

      Filter: Person is NOT member of "Smart List yyyy"

       

      Flow

      Send email zzzz

       

      My question would be: when a person is created, does it get added to my "Smart List yyyy" fast enough to NOT qualify?

       

      My intended behavior is that they don't qualify. I want to use this method to handle exceptions but I need to know if it works and if so, it it is reliable.

      I could always let them qualify and remove them from the flow if they belong to the list, but I have a feeling its more convenient, better for MKTO performance and perhaps more maintainable to do this in the smart list rather than inside the flowsteps.

       

      Thanks!

        • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
          Ronen Wasserman

          HI Victor,

           

          If the person doesn't exist already in the list I don't think it will work however what you can do is:

          1. Smart campaign:

          Trigger: Person is created

          Flow: Wait a few minutes.

          Request campaign.

           

          2. Smart campaign:

          Trigger: Campaign requested

          Filter: Country is xxxx

          Filter: Person is NOT a member of "Smart List yyyy"

           

          Flow: Send Email

            

          1 of 1 people found this helpful
            • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
              Sanford Whiteman

              If the person doesn't exist already in the list I don't think it will work

              It's a Smart List, though. So "membership" in the list is dynamic; there's no separate step to add to the list.

               

              The better way to phrase the question is "does the person match the filter parameters of the Smart List?" And that actually depends on what those parameters are. For example, if you trigger a webhook when somebody comes in, then the fields updated by the webhook may be updated a second or so later. If in the meantime you see if they're a member of a Smart List that filters on those field values, they may or be not be at that time (thus the outcome is unreliable). But a filter on data that already exists will find them.

              1 of 1 people found this helpful
                • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                  Floyd Alvares

                  Just adding my 2 cents:

                   

                  "My question would be: when a person is created, does it get added to my "Smart List yyyy" fast enough to NOT qualify?"

                   

                  When a member of a smart list filter (A) is referenced in a smart list within a smart campaign (B) or within another smart list (C). Then when B or C are executed, they will filter of each of the fields/criteria within the other smart list(A). So it is not entirely accurate to check if they would qualify fast enough in the smart list.

                   

                  However, the chicken and egg or in my terms we refer to them as the race condition will be needed to be taken into consideration.

                   

                  Example: a record may fill out a form on the website where the record populates certain field information (example country). Now if you run a smart campaign with the trigger "Fills out Form + Country value = YYYY". There will be a race condition between the trigger (Fills out Form) which is based on an activity in Marketo versus the update condition that Marketo is updating a record in Marketo (which is not based on the activity). However both these actions will happen at the same time.

                   

                  The same logic applies for filters like Data Value Changes and Person is created.

                   

                  The solution to this would be to either run batch campaigns for actions that do not need immediate effect OR leverage the solution recommend by Ronen Wasserman

                   

                  I hope this made sense

                  Thanks

                  Floyd

                    • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                      Sanford Whiteman

                      To be clear, in this particular case there's not a race condition.

                       

                      If the Smart List qualification filters are based on data values that exist on the lead at the point of entry into the system (i.e. form fields in the same form post that is their Web Form Fillout source on a New Person activity) they will be members of the SL at the time the Person Created trigger fires.

                      1 of 1 people found this helpful
                        • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                          Victor Herrero

                          I see what you mean.

                          As long as the source form includes the filters that you are using on the filter smart list, the trigger will fire correctly, because the necessary information will exist on the record on creation.

                          You seem to be 100% sure about the speed of dynamic Marketo lists, but are all records constantly being added to and removed from smart lists on creation / update instantaneously?

                          Is marketo THAT fast? Is it constantly re-evaluating membership for all records on each update?

                           

                          I didn't want to reveal / bore with the details of the campaign since I also wanted an answer that will help me in the future as well, but in this particular case, upon creation, all information needed for the Trigger and the Smart List is there.

                          So the setup should be reliable.

                            • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                              Sanford Whiteman

                              but are all records constantly being added to and removed from smart lists on creation / update instantaneously?

                              It's not about "adding" or "removing" in any tangible sense. It's about whether queued-up database updates have been committed (you can think of "committed" as meaning "hardened" or "finished" or "made available to other processes") before the same database is searched by another process.

                               

                              And in this particular case they have. In many other cases, the order of updates and searches is unreliable. This simply isn't one of them.

                               

                              P.S. There's a fascinating world out there of newfangled database locking and concurrent access methods. These are basic building blocks of systems design studied since the 1960s, but people have gone back to the drawing board recently with the explosion of internet traffic and the need for better and better multi-user performance.  Locking is made much more complex w/distributed systems (i.e. more than one server, as any large organization has, and more than one client), and distributed systems may accept being loosely consistent, meaning every server does not have the same data at every point in time. But again: not to worry in this case.

                              1 of 1 people found this helpful
                                • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                  Victor Herrero

                                  Ok, so after every update / creation, first smart list qualification is determined / fixed and then all other search processes take place.

                                   

                                  When it comes to smart list memberships, I find it fascinating that there is no actual field in Marketo's own database (or each of our instances) where for each record there is a "true/false" value somewhere.

                                  How does it really work?

                                    • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                      Sanford Whiteman

                                      I find it fascinating that there is no actual field in Marketo's own database (or each of our instances) where for each record there is a "true/false" value somewhere. How does it really work?

                                      It works like any database search query.

                                       

                                      For a greatly simplified example, in SQL

                                       

                                      select id from users where filter1 = 'something' and filter2 = 'something else'
                                      

                                       

                                      returns a result set with the matching user IDs at that point in time.  It doesn't write anything back to the users table. (The database user's permissions need not even allow it to update anything at all, just select.) The same query issued again can have a different result set.

                                       

                                      Yes, caching heuristics and intermediate tiers in many systems may store and reuse the entire result set for a certain period: "until the underlying table/s are known to have been updated" is the lossless way, while "until a fixed number of seconds have gone by" is the lossy but sometimes necessary way.  (Try as you might to avoid the latter by tweaking indexes, analyzing query plans, etc., you'll end up caching something somewhere in high-volume and/or legacy environments.)

                                       

                                      But regardless of whether caching is employed, it would never be feasible to store on the lead level whether each person matched each query of an effectively infinite number of queries. Some architectures flatten (denormalize) known-interesting data back to the individual record level. Indeed, that's a form of storing somebody's match-iness back on the record, and that's how Marketo segments work.  But you could never do this for all possible queries.

                                      1 of 1 people found this helpful
                                        • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                          Victor Herrero

                                          So you are saying that each smart list is basically a query result that gets updated every time the trigger is triggered, and until all query results have been delivered (i.e. first name is changed, so all smart lists and processes that are using that as a trigger are re-queried and the result returned) all other processes are on hold?

                                           

                                          returns a result set with the matching user IDs at that point in time.  It doesn't write anything back to the users table. (The database user's permissions need not even allow it to update anything at all, just select.) The same query issued again can have a different result set.

                                          you mean since its a query you don't need to write, you are just reading?

                                           

                                          Yes, caching heuristics and intermediate tiers in many systems may store and reuse the entire result set for a certain period: "until the underlying table/s are known to have been updated" is the lossless way

                                          I take it Marketo operates like this most of the time with its smart lists right?

                                           

                                          But regardless of whether caching is employed, it would never be feasible to store on the lead level whether each person matched each query of an effectively infinite number of queries. Some architectures flatten (denormalize) known-interesting data back to the individual record level. Indeed, that's a form of storing somebody's match-iness back on the record, and that's how Marketo segments work.  But you could never do this for all possible queries.

                                          So marketo segments DO generate a field on the records that stores membership data... this paragraph I don't understand too well.

                                            • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                              Sanford Whiteman

                                              So you are saying that each smart list is basically a query result that gets updated every time the trigger is triggered, and until all query results have been delivered (i.e. first name is changed, so all smart lists and processes that are using that as a trigger are re-queried and the result returned) all other processes are on hold?

                                              I'm sort of saying that, but I don't want you to get the idea that triggers block on all prospective or pending updates. Some things you might think of as an update from a user-facing perspective (like from a form post or list upload) aren't committed to the database for some time.  Only after they're fully committed can there then be a question of whether a filter will see them or not.

                                               

                                              Yes, caching heuristics and intermediate tiers in many systems may store and reuse the entire result set for a certain period: "until the underlying table/s are known to have been updated" is the lossless way

                                              I take it Marketo operates like this most of the time with its smart lists right?

                                              It depends.

                                               

                                              Let's just say that for your particular case a caching layer will never get in the way.

                                               

                                              So marketo segments DO generate a field on the records that stores membership data... this paragraph I don't understand too well.

                                              Correct.  This flattened field is accessible in Velocity, for example. 

                                               

                                              But you only have 20 Segmentations in your whole instance. That's very different from the infinite number of Smart Lists that can be created -- and modified -- throughout your database.

                                • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                  Victor Herrero

                                  Ok, so my takeaway from your answer is that its unreliable, same as Ronan, you are saying that if the data is already there when the trigger happens - it's relaible.

                                  Otherwise, you can't be sure if it's going to work a 100% of the time.

                                  • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                    Victor Herrero

                                    Here is another suggestion...

                                     

                                    Never use "person created" as a trigger in your instance's smart campaigns. Always use instead "added to list: "Person created" ".

                                    Create the list "Person created" via a smart campaign with the trigger "Person is created" (only this once, throughout your whole instance).

                                    Perhaps this would add enough delay and guarantee that the record has been created with at least all of the values it first came with, so by the time you check for "added to list" in your campaign all of the information should be there already.

                                     

                                    My concern here would be whether the instance could become sluggish, or whether there is an actual limit to the people that can be in a list.

                                     

                                    What do you think?

                                      • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                        Sanford Whiteman

                                        As above: depends on the source of the data value. If you do Change Data Value and then Add to List, then the trigger on Added to List can be filtered on the new data value with 100% reliability.

                                         

                                        If you do Call Webhook and then Add to List, then try to trigger on Added to List and filter on the data value, you have a race condition and your setup is not reliable.

                                         

                                        If you Call Webhook and trigger on Data Value Changes, that's 100% reliable.

                                         

                                        I never use Wait steps to anticipate data value changes. Not only is Wait clunky and itself unreliable, there's always another way.

                                        1 of 1 people found this helpful
                                    • Re: The Chicken or the egg? - What happens first: Added to smart lists or "person created" trigger?
                                      Victor Herrero

                                      Hi Ronen,

                                       

                                      Thanks for you answer!

                                      So you suggest I break that into two campaigns that happen in sequence.

                                      But doing that implicates duplicating the numbers of campaigns... perhaps in terms of maintainability not the best idea