About Andy_N

Andy_N · ‎01-13-2021

Can you give examples of "any service t hat uses a shared counter "? Also, what do you mean by "a shared counter"?

Andy_N · ‎01-12-2021

The concern with our previous experiments was the skews to our KPIs because different segments of our clients weren't represented equally in both groups. What do you mean by webhook-powered services that can randomize on certain characteristics? Can you give an example of a webhook that's used for strict round-robin distribution?

Andy_N · ‎01-07-2021

Hey everyone, Working now with our data team on the best way for us to split the audience for the email A/B test campaigns. Until now we’ve been randomizing people for that purpose in our CRM using signup time and/or native A/B testing capabilities of Marketo when sending emails. We’ve noticed though that it might be not ideal as some segments of our clients were overrepresented in one group vs another and this skewed success metrics since different segments are treated differently by sales. In general, different segments use our product differently so if they aren't represented roughly equally in A and B then we'll have an issue. I there anyone here who has experience spliting their audiences into A and B themselves and how do you do that? Which formulas, algorithms do you use? Any insights are appreciated. Thank you! Andy

Andy_N · ‎12-15-2020

I'm pretty sure I know what's the issue. One variation which had a gif went mostly to the promo/others tab..or maybe even the spam folder where the other went to the primary inbox that's why such a big gap in open rates though the subject and preview texts are the same.

Andy_N · ‎12-14-2020

Most actions affect one group of our users more than others. So when we run A/B tests we want to make sure that the 2 groups are roughly equivalent so the differences in their behavior can be attributed. If you're asking which specific distribution (binomial, normal, t- distribution, Chi-Square, etc) we want... then unfortunately I don't have the answer to that. Do you know which distribution is used by Marketo for A/B tests?

Andy_N · ‎12-14-2020

Knowing this how do you run A/B tests in Marketo to produce accurate results?

Andy_N · ‎12-14-2020

I've done an A/B test using the whole email variation test type with an audience of 3000 people. The A/B test was done on everyone (I’ve selected 100% thus I haven’t selected any winner criteria). So far 1 email got 45% more opens than the other (that’s about 60 people). I’m confused because the subject and preview text are the same… the low opens one even had slightly more delivered (10+ people) than the other one. The audience for the test is big enough to make a statistically significant conclusion that the subject line or/and preview text in As is much better than in B however it’s not the case since both are the same. Any idea why that would happen? Which criteria does Marketo use to divide the audience into A and B? How Marketo makes sure that audience A has the same attributes as the audience B? I feel like Marketo didn’t divide the audiences “fairly” My issue now is how can I rely on Marketo to run the A/B test for us?

Andy_N · ‎07-22-2020

We are currently cleaning our CRM and getting rid of many fields. It would be very helpful if I could get a spreadsheet with all the fields that Marketo is currently using. Basically, I need a list that shows only fields that have something in “Field Used By” so we make sure not to touch those in our CRM. I'm writing this here as I was told there is no feature or functionality available in Marketo to export only Used by fields and there is no workaround. Thank you, Andy Naumenko

Online Status	Offline
Date Last Visited	Tuesday

Date Registered	‎01-21-2020 01:17 PM
Date Last Visited	Tuesday
Total Messages Posted	8

Top Badges Earned

Re: Does anyone run A/B tests without relying on M...

Re: Does anyone run A/B tests without relying on M...

Does anyone run A/B tests without relying on Marke...

Re: Which criteria does Marketo use to divide the ...

Re: Which criteria does Marketo use to divide the ...

Re: Which criteria does Marketo use to divide the ...

Which criteria does Marketo use to divide the audi...

"Field Used By" Spreadsheet for Field Management