19 Replies Latest reply on Jan 11, 2017 1:59 AM by Diederik Martens

    Boolean Count

    Josh Hampleman

      We have multiple boolean fields used in part of our segmentation.

      What I would like to do is sum up the "True" values of these fields and denote this number in a separate field.

       

      Is there a way to do this inside of Marketo?  The boolean values change on a weekly basis.

      Thank you

       

      And if it can avoid using a Score Field on the original boolean field that would be best as multiple processes check the field and could leave to inflated numbers

       

      Message was edited by: Josh Hampleman

        • Re: Boolean Count
          Courtney Grimes

          Hi Josh,

           

          This is an instance where it may not make much sense to have Marketo itself do this—but counting the sum of True boolean values would be pretty easy to do with a Salesforce formula field or Dynamics calculated field. You'd then just echo that output into a standard field.

           

          This also gets around the issue with scoring, as you mentioned—it will just recalculate every time one of those boolean values change.

          • Re: Boolean Count
            Justin Norris

            I like Courtney's suggestion best, although if you need a solution that doesn't involve a round trip to the CRM, you can also check out FlowBoost (do a search in these forums for more info) or the Hoosh Excel webhook.

             

            Also, I know you mentioned you didn't want a score field, but...

             

            I'm not sure why you couldn't have a central processing campaign that you could schedule on a recurring basis. At that point you could zero out the field and recalculate it, e.g.,

             

            If Boolean A = True, booleanCounter +1

            If Boolean B = True, booleanCounter +1

            etc.

             

            Then you can reference that booleanCounter field anywhere you want, but the calculation of it is centralized, so I don't believe it would lead to inflated numbers.

            1 of 1 people found this helpful
              • Re: Boolean Count
                Josh Hampleman

                I'll look into FlowBoost as both Sanford and yourself had mentioned it.

                 

                After I said I didn't want a score field I began to think about a weekly process as you suggest where it +1s around the various booleans and then at the end of the week it zeros out before the next the weekly update.  I was too narrow in my original comment as I was thinking of adding the scoring section to each of the flows that attribute to the fields.  This is why I thought it would be inflated.

                 

                Thank you

              • Re: Boolean Count
                Sanford Whiteman

                Because we do everything we used to do in SFDC in FlowBoost* :

                 

                  var totalTrue = FBMath.sum( {{lead.booleanFieldOne}}, {{lead.booleanFieldTwo}}, {{lead.booleanFieldThree}} );
                

                 

                (Booleans automatically coerce to 0 or 1 when added together.)

                 

                Or if you want to learn native JS (recommended!) instead of using shortcuts:

                 

                  var booleanFields = [
                    {{lead.booleanFieldOne}},
                    {{lead.booleanFieldTwo}},
                    {{lead.booleanFieldThree}}
                  ],
                  totalTrueTake2 = booleanFields.filter(function(itm){ return itm; }).length,
                

                 

                or another native JS approach:

                 

                     totalTrueTake3 = booleanFields.reduce(function(prev,next){ return prev + next; })
                

                 

                or if you wanted to be really primitive (I wouldn't do this because it invites typos with a lot of fields):

                 

                     totalTrueTake4 = {{lead.booleanFieldOne}} + {{lead.booleanFieldTwo}} + {{lead.booleanFieldThree}};
                

                 

                *Depending on # of calls per month, would be fine w/FB Community (e.g. free) edition.

                • Re: Boolean Count
                  Josh Hampleman

                  We have nineteen boolean fields and only scoring off of three took roughly two hours to completely update the new score field.

                  So I don't think this is going to work out for us. 

                   

                  Speaking with our Communicators I believe we need something more advanced regardless and I don't know if it will be possible.

                  Based on the segments used in a current mailing I would need a count of those referenced in the message.

                  The number of segments and the segments being used change from quarter to quarter.

                    • Re: Boolean Count
                      Sanford Whiteman

                      We have nineteen boolean fields and only scoring off of three took roughly two hours to completely update the new score field.

                      So I don't think this is going to work out for us.

                      Are you sure that the performance is actually linear (i.e. with 6 does it take 4 hours)? How many leads did you run against in those 2 hours?

                        • Re: Boolean Count
                          Josh Hampleman

                          Not sure if it's linear but two hours is still too much time.  I believe in those two hours roughly 27k leads were updated

                            • Re: Boolean Count
                              Sanford Whiteman

                              I'll run some benchmarks on my primary instance to see how FlowBoost fares in comparison, but I'm confused about why 2 hours is too much time for a weekly batch.

                              • Re: Boolean Count
                                Sanford Whiteman

                                Benchmarked FB and was able to get 125ms/call, so that'd be about 60min as opposed to your 120min.  Not instantaneous, but I think I'm impressed rather than disappointed! I'm wondering what your specific objective is (or what leads you to think Marketo can be made to meet it ).

                                • Re: Boolean Count
                                  Diederik Martens

                                  Hi Josh,

                                   

                                  It is probably due to Marketo's webhooks. They are NOT designed for high volumes (anything over 1000). Marketo strongly discourages webhooks for this. They recommend a ReST API instead. Though that also has a limit of 300 leads max per call.

                                   

                                  The only native ReST API integrated formula suite is CalcItNow.com (also on Marketo launchpoint: CALCITNOW - Field formula calculations – Marketo LaunchPoint ).

                                   

                                  with a complex scoring formula with multiple fields involved I was able to update over 3000 leads in less than 1 minute (please note the 5 minute sync interval before changes are picked up though).

                                   

                                  Just PM me if you have any questions.

                                   

                                  Best,

                                    • Re: Boolean Count
                                      Sanford Whiteman

                                      The claim of a webhook max of 1000 (per day? per batch?) is mythical FUD. It is not supported by real production experience.

                                       

                                      That REST can be batchified and thus much faster in toto (unless you run out of API calls) than outbound integrations was never in dispute. I hope you can find selling points that are based on empirical evidence: it should be enough to prove substantively higher daily capacity with real-world numbers, not imaginary ones.

                                        • Re: Boolean Count
                                          Diederik Martens

                                          Hi Sanford,

                                           

                                          You are right. I don't have data to back up the webhook maximum load. I'm merely quoting what Marketo is saying during their API office hour calls. I just suggested this might explain the performance that is being experienced (27K in 2 hours). I'm not trying to attack anyone here. I don't know enough about FB. I do know formulas is just one of the many things it can do right? But as Josh mentioned he's looking for higher performance on formulas specifically, I just mentioned our solution as an alternative. (P.s. FB looks really powerfull from the screenshots I could find on your blog). Do you have moe documentation? I might have some customers who might be interested in it.

                                           

                                          ---

                                           

                                          Hi Josh,

                                           

                                          I just ran a test on our uswest server (at AWS) with 25K records, each with 10 booleans, to store the outcome in the lead score field. All records were loaded, calculated, and stored back within 15 minutes. This in batches of 300. So about 10 seconds for each batch.

                                           

                                          This could be faster, but capacity is intentially limited to 1 batch per formula per customer at the same time. The only exception is for the first batch of 300. So if 10 formulas, and less than 300 leads per formula need to be calculated. All 10 could run at the same time. But anything after 300 leads (so probably for bigger manual data changes you do), the next batches are sequential. This translates to the 300 per 10 seconds. I hope this helps. You can do a free trial by the way!

                                            • Re: Boolean Count
                                              Sanford Whiteman

                                              Thanks for clarifying, Diederik. On-the-ground tests show webhooks are fine with 50,000+ calls per day. This isn't specific to FB (though the fact that we autoscale helps, since we never run out of juice on our side). I'll run some more benchmarks this week and post about it.

                                               

                                              With webhooks, the faster you can consistently return results (as opposed to point-in-time/synthetic benchmarks) and the smaller you keep your payloads (both in and out) the better. Building for webhook traffic involves specific challenges that would look like microoptimizations in any other case, like worrying about 100ms differences in process (re)start times, maintaining burstable performance, things like that.

                                               

                                              I wasn't clear from this thread if Josh ended up calling a webhook at all, or whether he was talking about Change Data Value performance (fully within Marketo); he didn't try FlowBoost. But without making those optimizations, a service that works great for a handful of hits will eventually crumble.

                                               

                                              With the above optimizations in mind, though, you can get "wire speed" performance from webhooks, that is, if your 'hook maintains a roundtrip time of 120ms from a barebones HTTP client, Marketo isn't adding substantive overhead to that. So we can get ~8 results/sec for a synchronous webhook and maintain that rate through ~500/min and ~30,000/hr. (Async style, for when you just fire-and-forget to a remote service/database, is about 25% faster.)

                                               

                                              But for sheer performance, you can rip way more out of REST, no doubt whatsoever.

                                               

                                              I can see why Marketo support is championing the REST API because, well, it looks good to have your own API be the integration of choice! Also, as far as maintaining control, what we see as the amazing power of webhooks might look like a scary "Wild West" situation from another angle. After all, the story goes that Munctions were taken away because they were perceived as too dangerous, or at least too dangerous to be a native function! I think that's weird given that you can wipe out data in many ways in Marketo (including a REST app, if it has the right permissions) but I guess offering one more way directly to admins was seen as bad karma, and webhooks seem like they occupy the same space.

                                               

                                              I think CalcItNow is making perfect use of Marketo REST, since field updates are totally bulkified and won't run into the bizarre confusions of the Activity API. My principal problem with the Marketo API is that the microscopic number of calls must be shared across all your integrations, and I'm continually horrified that it's advertised for non-bulk applications (a.k.a. DoS fests). In a controlled context, there's nothing better.

                                               

                                              Finally, like you said, FlowBoost is indeed aiming beyond string & number manipulation, but we're darn good at that, too, so we can't deny that FB and CalcItNow are going head-to-head.  I imagine we should come up with some official differentiators if we're both going to live in this town.

                                                • Re: Boolean Count
                                                  Diederik Martens

                                                  Thank you too for clariying. I can imagine Marketo pushing for the API of course.

                                                   

                                                  CalcItNow does also have two webhooks. One as stand-alone webhook. The other specifically for Marketo. you can the webhook, but it actyally uses the API to store process the data. The webhook merely takes the leadid as input.

                                                   

                                                  It seems there is indeed some competition. Though one of the other solutions that are being developed right now, likely will also have some competition (not with you). I'm not particularly interested in going head-to-head. I just want to provide people a solution to their challenges, the same way I do with contracting in marketing ops.

                                                   

                                                  I think FB allows you to do more advanced raw coding. CalcItNow is more wysiswyg wizard style. Making it easy for the less techy people to create something. Maybe it's like Eloqua vs Marketo? Eloqua has the ability to do more complex flow structures. The flipside is that it's harder to understand what that workflow does, simply by looking at it. I created CalcItnow 1.0 back in 2014, which was only used by people I know (not openly purchasable). Now 2.0 is publicly available.

                                                   

                                                  P.s. We also have Hoosh in our space...

                                                   

                                                  The other solutions planned for 2017 (hopefully befoe the summit) are in a different area than CalcItNow and FB. So no worries ;-)