Skip navigation
All Places > Products > Blog > Author: Sanford Whiteman
1 2 3 4 Previous Next

Products

60 Posts authored by: Sanford Whiteman

When an HTML document requests additional HTML content over Ajax (XMLHttpRequest), the browser doesn’t run the code inside <script> tags in the response.[1]

 

This is so even when the response is an otherwise-living HTML document.[2] The innards of <script> tags are still downloaded, of course, but they’re neutralized as raw text (e.g. source code) and not executed.

 

Why this matters for SPAs

Some classic multi-page apps can be (relatively) easily upgraded to (theoretically) faster single-page apps (SPAs) just by changing how links work.

 

Instead of reloading a whole new page on link click, the initial page stays put, while the next page’s URL is fetched via a background Ajax call. Then part of the main page is replaced by part of the HTML response. It’s far easier on the network, ideally easier on CPU, and faster for the end user.

 

But this change comes at a price. Previously working JS that set up form behaviors, dynamic styles, etc. won’t run when a page serves merely as a passive source of HTML markup.

 

If an Ajax framework so chooses, it might find all the scripts in the response, copy out the inner text of each, and reinject it inside new <script> elements.[3] Then and only then will the code be executed.

 

But the strictest web frameworks don’t allow you to “resuscitate” scripts in this way (even when you control both the outer page and the inner Ajax-fetched page, so there’s no security risk). Squarespace’s proprietary CMS framework is one of these strict ones.

 

New Squarespace templates are Ajax-powered SPAs

In the newest gen of Squarespace templates (note: my sole SS site is on a very old version and still looks fine, thank you very much!) such as Brine, your entire site is actually a single page.

 

Once that initial page is loaded, other would-be “pages” are actually HTML fragments dropped inside a placeholder “dynamic content” element. (A full HTML page is fetched under the hood using Ajax, then the interesting parts are clipped out and injected into the placeholder, replacing its previous contents.)

 

As you follow nav links like Shop or Blog and click further into Shop products and Blog posts, you’re doing all that within the same main document:

 

 

If you refresh your browser, the main document does reload once, then the process continues: new “pages” are again Ajax-injected as you click, keeping the same main doc. (The browser URL does change, so to those unfamiliar with the History API, it might look like a new page loaded. But nope, watch the F12 Network tab and you’ll see it’s all Ajax.)

 

So Squarespace pages serve in two capacities:

 

(1) They can serve as an active main document — if you link to them from a Marketo email, hit Refresh, paste them into your browser and hit Enter, etc. — and in these cases they will run your header or footer JavaScript one time, on page load.

(2) But they can also serve merely as passive sources of HTML content, fetched dynamically into the main document. In this case, they’ll replace the page’s dynamic content, including <div>s and <span>s and <img>s and all that stuff, but won’t run any code.

 

Therefore, if you have JavaScript on your page, you need to make sure that it’s ready for both cases.

 

You can’t skip the more traditional case (1). Synchronous <script> or <script src>, external <script src> that’s asynchronous and/or deferred, code that adds listeners for load or DOMContentLoaded — all must run on a traditional page load, because there’s no special event for non-Ajax-created content.[4]

 

But your code must also account for case (2). It’ll run only once during main document load, so during that run it must add an event listener function that’ll be triggered every time a new “page” is loaded via Ajax. The listener won’t always find something to do, but it needs to always check for possible tasks, like adding a Marketo form if newly injected content matches a certain pattern.

 

Squarespace insiders apparently use the jargon “Ajax-enabled code” to describe JavaScript built to handle both cases. Yes, sure it’s “Ajax-enabled,” but such a generic term doesn’t help a developer. A clearer way to put it is: your <script> may run only once per browser session, so it also needs to set up listeners for ongoing changes to the HTML DOM after that first run.

 

Adapting the Marketo form embed

The standard Forms 2.0 embed code is like so:

 

<script src="//pages.example.com/js/forms2/js/forms2.min.js"></script>
<form id="mktoForm_9999"></form>
<script>
MktoForms2.loadForm("//pages.example.com", "123-ABC-456", 9999);
</script>

 

Straightforwardly, the embed:

  • loads the common Forms 2.0 JS library
  • creates an easily findable empty <form> element
  • runs loadForm, which downloads the form descriptor from Marketo, creates the initial contents of the <form>, and sets up all the JS event handlers for validation and Visibility Rules and all that jazz

 

Understandably, the embed code expects to run synchronously, as part of a main document’s (script-enabled!) page load. Its goal is to get the form into the page as quickly as possible (the form being the most important part of a landing page) so it doesn’t include any extra logic. It rightly remains neutral, as it couldn’t fit all manner of unpredictable embedding situations.

 

But clearly the default embed code won’t function on a Squarespace page when that page falls into case (2) above. When the “page” is merely an Ajax-injected block of HTML, the empty <form> element would be rendered (for what that’s worth) but neither top nor bottom <script> would be executed. So there would be no Marketo form.

 

So we need to get smarter. We need JS that:

 

  • Loads the Forms 2.0 JS library — as there’s no reason to redundantly load this static JS — to create the global MktoForms2 object and hold it in reserve, even if the current view isn’t supposed to have a form.
  • Seeks in the current HTML for a recognized parent element. It could be anything, just wherever you want to place the form. For example, if you want every blog post to have a newsletter signup form at the top, see if the page is in “Single Blog Item” mode and find the <article> element on that page.
    • If there’s no matching parent element, don’t do anything for now.
    • If there’s a form-worthy parent element, check to see if it already has a <form> inside (from earlier navigation). If it’s already there, don’t do anything for now.
    • If there’s a parent, but no form yet, inject the <form> using standard DOM createElement, then run loadForm().
  • Adds an event listener to run itself again when there’s new page content. This is the key to being Squarespace-aware. By listening for new “pages” to be loaded via Ajax, we automatically re-check if the parent element has come into view, and inject the form inside it if applicable.

 

The code

For this demo, I wanted to display a Marketo form at the top of every Blog item (not trying to win any design awards here!):

 

 

So first I determined the CSS selector for the parent HTML element, the first match for "article.BlogItem" in Squarespace’s single-Blog-item view:

 

Then added this code to the global Footer (the surrounding <script> tags are required) under Settings » Advanced » Code Injection:

<script>
(function(){

var instanceURL = "//pages.example.com",
munchkinId = "ABC-123-456",
formid = 9999,
parentContainer = "article.BlogItem", // where the form goes
formPositionBefore = true, // put <form> :before other content in parent
pathInfoField = "SquarespaceSubPage"; // hidden field to pass SS sub-page


/* -- NO NEED TO TOUCH BELOW THIS LINE! -- */

var mktoFormsJS = document.createElement("script");
mktoFormsJS.src = instanceURL + "/js/forms2/js/forms2.min.js";
mktoFormsJS.addEventListener("load", function(e){
// check on initial full page load
injectMktoForm();
// recheck on Ajax "page" load
window.addEventListener("mercury:load", injectMktoForm);
MktoForms2.whenReady(function(form){
var fieldsObj = {};
fieldsObj[subPageInfoField] = document.location.pathname;
form.addHiddenFields(fieldsObj);
});
});
document.head.appendChild(mktoFormsJS);

function injectMktoForm(e){
var parentsVisible = document.querySelectorAll(parentContainer),
formVisible = document.querySelector("form#mktoForm_" + formid + ":not(:empty)");

// expect only one eligible parent for simplicity
if ( parentsVisible.length != 1 || formVisible ) {
return;
}

var mktoFormParent = parentsVisible[0],
mktoFormEl = document.createElement("form");

mktoFormEl.id = "mktoForm_" + formid;
mktoFormParent.insertBefore( mktoFormEl, formPositionBefore ? mktoFormParent.firstChild : null );
MktoForms2.loadForm(instanceURL,munchkinId,formid);
}

})();
</script>

 

As you can see, the code assumes you have a single Marketo form that goes in a consistent page location in Squarespace. You can extend the code greatly, to deal with multiple possible parents and multiple form IDs, but the core logic remains the same.

 

One extra frill

You may wonder about the JS variable pathInfoField. It’s optional, but you can set it to a Marketo form field name. The code will add a corresponding hidden field, set to the URL path at the time of form submission:

 

 

This aids in building Smart Campaign filters. You see, because of the SPA architecture, the Referrer property of each Filled Out Form activity is the last main document — that is, the last real Squarespace page, not the Ajax-injected “page” they’re currently viewing — so it’s not a fine-grained constraint.

 

It’s possible to hack harder so the sub-page appears as the Referrer, but that’s complex enough that it’d have to have another post of its own.

 

 

 


NOTES

[1] As I’m ever curious (and hope you are too)  this is the exact section of the XMLHttpRequest standard that specifies that scripts must be neutralized:

 

[2] That is, if you pass contentType="document", you get a fully parsed HTML Document object, but the <script> tags are still disabled.

 

[3] Or, less securely but still effectively, eval() it.

 

[4] A “content loaded via any method” custom event could always fire, for consistency – it would be equivalent to DOMContentLoaded in the case of the initial pageview – but in Squarespace’s case it doesn’t.

Some small signals from end users stressing some can be disproportionately meaningful.[1] One such signal: that they started to fill out a form, so you can compare/intersect with people who eventually submitted the form.

 

The delta between the 2 is, of course, people who were distracted/discouraged/disengaged (or, in the interest of completeness, people who couldn’t submit the form due to forces outside their control).

 

Of course, we need to define our terms.

 

Does “started” mean “focused – either by clicking, tapping, or tabbing – a visible form field”?

 

Or does “started” mean “changed a value” in a visible form field? This definition might seem more exact at first, but the problems are (a) there’s no need for someone to change a value in order to submit a form – remember Pre-Fill – and (b) there’s no single DOM event that can account for all types of net changes, making the accompanying JavaScript very complex.

 

There are other directions you can take with “started,” and all manner of JS hooks you can build (veering quickly into way-too-much-to-be-useful). In a later post, I may feature some of those takes. But today I’m going to stick with the simple definition: Did someone move focus to the form?

 

Here’s how that’s done:

MktoForms2.whenReady(function(form){
var formEl = form.getFormElem()[0],
formId = form.getId();

formEl.addEventListener("focus", logFormStart, true);

function logFormStart(e){
formEl.removeEventListener("focus", logFormStart, true);
Munchkin.munchkinFunction("visitWebPage",{
"url" : document.location.pathname + "#!/inPage/startedForm/" + formId
});
}
});

 

Then you’ll see a hit one supplementary hit per pageview, naturally in the Activity Log:

 

 

/startedForm also logs if they submit the form without touching any fields, because clicking Submit also fires a focus event. (This is A Good Thing.)

 

Note the hit will stay in the Anonymous Activity Log if the person never converts, but moves to the standard (Known Lead) Log after their session is associated. As with all Visit Web Page activities, there’s ongoing backfill.

 

I’ll leave it to you to come up with meaningful Smart Lists using this info.

 

Why log Visit Web Page instead of Click Link?

Because Munchkin clickLink blocks the browser UI thread. And if you don’t know what that is: trust me, it’s A Bad Thing.

 

I’d go far as to say all custom Munchkin code that uses clickLink for in-page events (non-unloading events) instead of visitWebPage is lowkey broken – admittedly including some of my own older code. Someday I’ll get to a blog post specifically about this, but for now: trust me.

 

 


 

 

NOTES

[1] I feel strongly that logging every minor-to-meaningless page interaction is wrong. That not only creates chaff on the server side, it’s also disrespectful to users with low-quality connections. I’m in a big city, but still stuck on slow DSL, people!

 

In addition, Munchkin’s append-only logic is particularly ill-suited for storing a continuous spray of superseding events. The proper back end for that is one that can consolidate superseding events into one. Example: video tracking. If someone watched 60s of a video, they also watched 30s, so only the superseding value should be stored. Of course, I still use Munchkin to log YouTube plays... but I know it’s wrong in the grand scheme.

While you should be loading embedded forms from your Marketo LP domain, it turns out just changing the URL is not enough.

 

Using a non-Marketo-owned domain to load forms2.min.js and call loadForm() seems to avoid collateral damage from Tracking Protection (Firefox’s built-in feature or the equivalent plugin for another browser). The form will at least show up on the page if you do that.

 

But – I’m sure you will be dismayed to learn – the form still won’t post under TP! You also need to upload a file to Design Studio and add a tiny bit of JavaScript. Then and only then are you good to go.

 

First, download this file:

 

marketo-xdframe-relative.html

 

(Note: the Marketo Nation site gates files through an “Offsite Link” popup. So click above, then right-click the link + Save Link As. Make sure you get the .html file itself, and don’t end up with a snapshot of this blog post!)

 

Next, upload the file to your Design Studio. To be clear, you’re not creating an LP, you’re uploading the file as static HTML. This may not be something you’ve done before (you probably upload images, CSS, and PDFs) but Marketo actually supports any kind of file.

 

Screenshot after it’s been uploaded to Design Studio:

 

 

Then replace your embed code with this ever-so-slightly different version:

 

<script src="//pages.example.com/js/forms2/js/forms2.min.js"></script>
<form id="mktoForm_9999"></form>
<script>
MktoForms2.setOptions({
formXDPath : "/rs/123-ABC-456/images/marketo-xdframe-relative.html"
});
MktoForms2.loadForm("//pages.example.com", "123-ABC-456", 9999);
</script>

 

Where:

 

  • 123-ABC-456 represents your Munchkin ID
  • pages.example.com represents your Marketo LP domain
  • 9999 is o’course your form ID

 

A live demo (though all the code is above):

 

MktoForms2 :: XDFrame relative path for Tracking Protection

 

The why

Using your LP domain enables forms to show up. But there’s a special asset, only used upon submission, that will still be blocked by default. Naturally, this only reveals itself when you run end-to-end tests.

 

See, embedded forms use an IFRAME message relay (XDFrame) for cross-domain posts (there are other ways to do it, but they’re not as backward-compatible and the IFRAME method works fine + fast in all browsers, even back to IE8).

 

But the IFRAMEd document loads forms2.min.js from an absolute URL with your instance hostname, i.e. app-xxnn.marketo.com:

 

Oops! Tracking Protection ain’t gonna let that load, for the same reason it doesn’t like a Marketo domain in the main document. (You can’t avoid TP by using an IFRAME, that would be silly.) As you can see, the replacement XDFrame file uses a relative URL (/js/forms2/js/forms2.min.js), so it loads from your Marketo LP domain like the rest of the assets. Presto! No marketo.com for TP to get all paranoid about.

 



NOTES

forms2.min.js inside the IFRAME is only necessary because of its bundled  jQuery library, specifically the MktoForms2.$.ajax wrapper method. While not technically necessary, using jQuery keeps parity with the Forms 2.0 API.

New developers (and non-developers) often mistake values that might be converted to, converted from, or treated sort of like Booleans for a language’s actual Boolean type

 

This leads to internal contradictions like:

 

I have a Boolean with field with values “on” and “off”.

 

and

 

If my Boolean is set to 1, I want to do this. If it's set to 0, do this other thing.

 

If the value is really a string like “on” or a number like 1, you’re not (or no longer) working with a Boolean.

 

Boolean vs. sorta-boolean-ish-sometimes

In general, if a programming language includes a Boolean type, then a Boolean variable can have one of two canonical[1] values:

 

    one value will be the unquoted keyword true

    the other will be the unquoted keyword false

 

The keywords might be lowercase (true/false), uppercase (TRUE/FALSE), or mixed-case (True/False); some languages are case-insensitive.

 

But the quoted strings true” and “false”, the numbers 0 and 1, the numeric strings “0” and “1”, let alone strings like “Yes” and “No” or “Y” and “N” or “opt-in” and “opt-out” or the empty string “” — with rare exceptions in the world of SQL[2] (but not in Marketo) none of those are Boolean values.

 

Sure, those pairs of values might be a lot of useful things:

 

    labels for radio buttons on a form that map to a Boolean on the back end…

    values associated with a custom checkbox-lookalike field

    workarounds because some aspect of an app doesn't support Booleans…

    values that convert predictably to/from Booleans in a specific framework or language…

    the way an app displays Booleans in emails or web pages (hint, hint)

 

But not real Booleans, and the difference matters.

 

Just one of the reasons it matters in Marketo-land

A recent Velocity question from a client brought the difference into high relief:

 

  

With our Product Registration CO, how come the condition #if($product.isRecalled)always seems to match, even if isRecalled is set to false?

 

I immediately looked at their Custom Object setup in Admin, and check it out:

 


 

Bingo! The field isn’t a Boolean. It’s an Integer, presumably sent to Marketo exactly as it’s stored in their in-house SQL database.  And they were trying to use it in a Boolean context (an #if() condition) and expecting 1 and 0 to be truthy and falsy respectively. 

 

But Velocity doesn’t work that way. (Java doesn’t work that way, either: in Java, a strictly typed language, if something ain’t a Boolean, you’ll get a fatal type mismatch error, truthiness and falsiness do not apply.)

 

Note even real Boolean fields (CO or Person fields) require special treatment once they get into Marketo’s Velocity, as I’ve written about before. But here the field wasn’t even a Boolean to begin with! I’ll show you the slickest way to do the required conversion in Velocity in a moment.

 

But first, a little more about real Booleans vs. boolean-ish values, to help you code and/or talk to coders.

 

Truthiness doesn’t change the truth

I don’t blame folks for thinking 0 and 1 are low-level Boolean values at first. There are a bunch of things that can lead you down that path.

 

MISLABELING

Many SaaS apps don’t do you any favors by calling anything vaguely yes-ish/no-ish a “boolean” to sound cool (I guess?). It would be better to call such things “YepNopes” than to be misleading about the real type.

 

THE STRING-CENTRIC WEB

Bedrock web technologies operate exclusively on strings. Take a standard x-www-form-urlencoded form post or a URL query param: those are all characters on the wire.[3]

 

TRUTHINESS

This is a big one. Some languages assign “truthiness” or “falsiness” to certain non-Boolean values, so they don't have to be manually converted to the Boolean datatype to be used in a Boolean context. This can make for less complex-looking code (for what that’s worth) but with a cost of risk and just bad learnin’.

 

Make no mistake: just because a language considers if(1) or if("1") to be true, that doesn’t mean 1 and “1” are Booleans. They’re merely treated, on-the-fly, as Boolean true or false. And a value that’s temporarily truthy in an if() condition may not even be true when permanently converted using toBoolean(), bizarre as that might seem!

 

Moreover, truthiness rules are totally different across languages. One language’s truthy is another language’s falsy — or another language’s fatal TypeError. I’m not harping on hard-to-find cases: 0 and empty “” are falsy in JavaScript; they're both truthy in Velocity.

 

THE ELUSIVENESS OF SQL BOOLEAN

SQL databases are probably the main way people come to think 0 and 1 are Booleans, when in fact they’re just a very common way of storing things that were/will be Booleans.

 

SQL — the ANSI SQL standard, not all products based on that standard — does have a column type BOOLEAN. However, of the big 5 SQL products, only Postgres supports it. Microsoft SQL, MySQL, Oracle, and DB2 don’t. So, to avoid database vendor lock-in, you must use some other SQL type to store values that will be converted to/from Booleans in your app code. The narrowest possible INT, with values restricted to 0 and 1 via a constraint, is very widely used. (Though it’s not the only choice.)

 

Back to Marketo

You know I couldn’t resist that technical dive. But let me flip modes and get back to the matter at hand.

 

Velocity is more forgiving than Java, so it doesn’t throw an error[4] when non-Booleans are used in a Boolean condition. Velocity uses a very simple rule: anything other than null or the real Boolean false is truthy. That means an empty String, a non-empty String, an Object or List (regardless of whether it has any  properties/items), and 0 and 1 and all other Integers are all going to match #if($something)!

 

Now it’s clear why the client’s attempt to use #if($product.isRecalled) was doomed to fail.

 

So what’s the simplest way to turn 1/0 to true/false?

 

There’s clearly a plodding way to do it:

 

#if( $product.isRecalled.equals(0) )
#set( $product.isRecalledBoolean = true )
#else
#set( $product.isRecalledBoolean = false )
#end

 

But that’s going to be cumbersome to call every time. What about converting the values to Booleans using ConvertTool?

 

#set( $product.isRecalledBoolean = $convert.toBoolean($product.isRecalled) )

 

Nope. ConversionTool’s toBoolean() only treats the String "true" as convertible to Boolean true. Numbers 0 and 1 will both convert to Boolean false. (Notice how that’s the opposite of 0 and 1’s truthiness? That’s the kind of unpredictability I’m talking about!)

 

But there’s a secret weapon we can use to keep the conversion to a (slightly longer) one-liner: DisplayTool’s plural() method.

 

DisplayTool.plural() would be better called DisplayTool.notOneOrMinusOne(), because 0 is considered plural:

 

 

This quirk is great for our case, because now we can do:

 

#set( $product.isRecalledBoolean = $convert.toBoolean($display.plural($product.isRecalled,"true"))

 

 Here’s how this one-line contraption works:

 

      
  •  plural checks if the first argument ($product.isRecalled) is what it considers “singular” or “plural”   
          
    • if the argument is singular (that is, for our purposes, if it’s 1 – since we’re saying it’s never anything other than 0 or 1) then it returns the second argument, the string "true"
    •     
    • if the argument is plural (that is, if it’s 0) then it returns the second argument appending the letter “s”, to make the string "trues"
    •    
  •   
  • $convert.toBoolean then processes the returned value, which is always "true" or "trues"   
          
    • toBoolean("true") converts to Boolean true
    •     
    • toBoolean("trues") converts to Boolean false
    •    

 

So we’ve implemented the desired 0→false, 1→true conversion. Cool, eh?

 

 

 


 

Notes

 

[1] “Canonical” means a value that can be both written to and read from the variable. I use this qualifier because some languages allow alternate values to write a Boolean, but you’ll never see those values when you read a Boolean. For example, in XML, 0 and 1 are synonyms for true and false when setting an xs:boolean. But the values read back out are always true or false.  And the Postgres example below applies as well.

 

[2] Postgres goes bizarrely further than just the standard TRUE/FALSE keywords (though those are the preferred values for setting SQL BOOLEANs). They allow lots of aliases to be used for setting values: the strings “true”/“false” and “t”/“f”, even “yes”/“no” and “on”/“off” and a few more. Stranger still IMO is that the output values are the strings “t” and “f”. So you would say the canonical values are “t” and “f” in that particular dialect (since those are the only values that can be both written and read). Weird exception.

 

[3] Funny to call XML “newer” than anything as it’s so ancient, but both XML and JSON reserve special Boolean values (as well as distinguishing numbers from numeric strings) even though the overall encoding is still character-based. But regular form POSTs and GET params are strings through and through, until/unless the server chooses otherwise.

 

[4] This forgiveness is not necessarily a good thing. As I’ve droned on about before, Velocity suppresses a lot of fatal errors to keep your app running, but not all types by any means: you can trivially cause a fatal error by .get()-ing the 1st item in an empty List!

Community user JW asked a surprisingly unprecedented (AFAIK) question:

 

I am looking to send out a communication to our customers that includes that persons Policy Number. Unfortunately our legal department will not let us send the entire {{lead.Policy Number}} digitally. We are permitted to send the last 4 digits however...  with Velocity Scripting can I get it to show as *****3253?

 

Let’s be clear: this isn’t relevant to credit card numbers or SSNs because those fields would never be in Marketo in the first place! (Right, guys?)

 

But other fields – like Policy Number here, or other account/lead info like the person’s phone number – legitimately would be present in Marketo and your CRM, yet might be nice to partially redact in emails. In the unlikely event that an email is intercepted, the person’s privacy is protected, but there’s still enough data to know they’re the right recipient.

 

So here’s a quick one-liner to mask out 1234567890 to ******7890:

 

${lead.PolicyNumber.replaceAll(".(?=.{4,}$)","*")}

 

In brief, it’s a regex lookahead and replace. The pattern matches any character that has 4 or more characters between it and the end of the string, ergo, all characters except the last 4.

 

(Yes, there are ways to do this with indexOf and substring instead, possibly more efficiently than the regex… if you’re timing to the microsecond, that is. But as always with Velocity, saving a few lines of code is worth a few μs.)

For arithmetic in Velocity, the only safe approach is the so-called “tools-ish” one: use MathTool methods like $math.add, $math.sub, and  $math.mul.

 

The rule

Aside from the one exception later in this post, you should not use the Java-like math operators (+ and -).

 

The core reason is the plus sign + and minus sign - have different syntax rules in Velocity! You can easily create fatal errors – or worse, lines that are silently ignored – by forgetting that - must be surrounded by spaces.

 

That’s right, only one of these is correct Velocity syntax for subtraction:

 

#set( $a = 99 )
#set( $b = $a - 1 ) ## correct
#set( $b = $a -1 ) ## fatal ParserException!
#set( $b = $a- 1 ) ## fatal ParserException!
#set( $b = $a-1 ) ## non-fatal, but doesn’t subtract!!!

 

So I always teach people to use #set( $b = $math.sub($a,1) ). It’s longer, but you’ll never break your code by switching sub and add.

 

The exception

Despite the above, in the course of developing my Base64 encoding in Velocity code, I discovered a unique requirement that could only be met by purposely using the + operator instead of $math.add.

 

This is the only exception I’ve found, and it doesn’t override my recommendation above. But it is – to me and I hope to the couple of people like me out there – fascinating.

 

So.

 

In Base64 encoding, one of the steps requires Integers to be converted to binary Strings, 42 → “00101010”. (If this is complete gibberish to you, you can reread this post when you have a couple more years of development under your belt!)

 

To get those Integers, you need to convert signed Bytes (-128 through 127) to their equivalent all-positive Integers (0 through 255).

 

To convert signed Bytes, you need to use a bitwise AND. But the traditional AND operator, the & symbol in Java and most other languages, isn’t available in Velocity.

 

Since you can’t use notation like #set( $c = $a & $b ) in Velocity (that’s a fatal parser error) you need to find a method-based way – a “tools-ish” way, somewhat ironically – to do a bitwise AND: something like #set( $c = $a.and($b) ).

 

Miraculously, such a method does exist (after you do some hunting) so long as $a is a Java BigInteger.  

 

Hmm. So now you need to be able to construct a BigInteger from an Integer.

 
Getting a BigInteger

A BigInteger isn’t something you can explicitly create in Marketo’s Velocity (post-June 2019, that is), only implicitly. There’s one way to cheat it, and that is to set a variable to just outside of the range of a Long. If you know off the type of your head that the max positive Long is 9,223,372,036,854,775,807 (~9 quintillion) then you can add 1 to that:
    

#set( $aBigInteger = 9223372036854775808 )
${aBigInteger.class} ## will be class java.math.BigInteger

 

 

Skipping the magic

But let’s say you don’t know that magic number – you only know that Java’s

concrete integer Number types  are, in order of increasing width, Byte → Short → Integer → Long → BigInteger. (I think it’s reasonable for a developer to know that progression, but not reasonable to expect someone to remember that 9,223,372,036,854,775,807 is special.)

 

If you didn’t know the magic Long-BigInteger boundary, how would you make sure Velocity allocated a type big enough for a BigInteger? You’d want to say:

 

Gimme a Number that can hold Long.MAX_VALUE + 1.

 

Of course to do that, you‘d have to get a handle on a Long (in order to refer to the constant Long.MAX_VALUE). Since Velocity doesn’t give you a Long unless you’ve asked for something bigger than an Integer, you have to say:

 

First, gimme a Number with the value 0. I know that’s an Integer, since that’s the default type for whole numbers in Velocity.


Then, gimme another Number that can hold ThisMustBeAnInteger.MAX_VALUE + 1. That’s gonna be a Long.

 

Finally, gimme yet another Number that fits ThisMustBeALong.MAX_VALUE + 1. That third number must be a BigInteger.

 

And here is where + acts differently from $math.add.

 

MathUtils vs. MathTools

Hidden in the doc for Velocity’s MathUtils (part of Velocity’s internal API, not for public consumption) you’ll find this gem:

 

So now we know Velocity can do – in fact is explicitly documented to do – exactly what we need. If we add 1 to the max value of a Long, the result will be a BigInteger (and will also have the correct value, obviously).

 

Except this only applies to the literal + operator. MathTools’ $math.add also does addition, but doesn’t implement the “overflow correction” logic.

 

Let’s compare.

 

Using the + operator:

#set( $Integer = 0 )
Integer $Integer.class.getName() = $Integer
#set( $Long = $field.in($Integer).MAX_VALUE + 1 )
Long $Long.class.getName() = $Long
#set( $BigInteger = $field.in($Long).MAX_VALUE + 1 )
BigInteger $BigInteger.class.getName() = $BigInteger

 

The output will be:

Integer java.lang.Integer = 0
Long java.lang.Long = 2147483648
BigInteger java.math.BigInteger = 9223372036854775808


Exactly what we need. $BigInteger is a BigInteger we can use as a factory to make more BigIntegers ($BigInteger.valueOf) and then do fancy stuff like .and() and .or() and  .xor() which would otherwise be impossible in Velocity.

 

Let’s try using $math.add:

#set( $Integer = 0 )
Integer $Integer.class.getName() = $Integer
#set( $Long = $math.add($field.in($Integer).MAX_VALUE,1) )
Long $Long.class.getName() = $Long
#set( $BigInteger = $math.add($field.in($Long).MAX_VALUE,1) )
BigInteger $BigInteger.class.getName() = $BigInteger

 

 

The output will be:

Integer java.lang.Integer = 0
Long java.lang.Long = 2147483648
BigInteger java.lang.Long = 9223372036854775807

 

 

Whoa. We attempted to add 1 to the largest Long, but the attempt silently failed. $BigInteger (despite the name) remains a Long, and it never gets incremented: it’s still Long.MAX_VALUE or 9,223,372,036,854,775,807.

 

It’s notable, but doesn’t change the rule

The likelihood that you’d otherwise need to perform BigInteger-range arithmetic within Marketo is basically zero.

 

I mean, it would have to be something like… you’re using a webhook to take the squares of lead scores, and you have a score in the millions and must raise it to the power of 2 in a Velocity token. Let’s face it, relative to the chances you will at some point forget the spaces around -, that’s just not gonna happen.  The risk/reward overwhelmingly favors $math.add and $math.sub.

 

In this case we needed a BigInteger not because of its infinite range of values, but because we need to get at the cool methods it has – which Integers and Longs do not. (The underlying value is tiny, between -128 and 127 then converted to between 0 and 255.) So for this particular purpose, deliberately using + makes sense.

The removal of Reflection-based features from Marketo’s Velocity environment in June 2019 was a disappointment — however inevitable — to those of us who push email scripting to the limit.

 

Requirements like encoding, hashing, encryption, flexible rounding and randomization, XPath queries, and advanced filters/sorts on Custom Objects could no longer be met with Marketo alone. (Webhooks can fulfill a subset of those needs, but with nowhere near the performance and simplicity of Velocity.)

 

At the prodding of Community user TM, I set out to see if Base64 encoding, one of the Java methods that used to be callable from Velocity, could be written with “pure” Velocity and public Java methods, without relying on the Reflection API.

 

I was indeed able to do it... though unlike most of my Velocity work, this one wouldn’t File Under Fun!

 

You can call the #userlandBase64_v3 macro below in 4 ways:

 

#1: Basic usage

#userlandBase64_v3("some string")

The result uses the standard RFC4648 Base64 character set (A-Za-z0-9+/). (That is, the characters used/expected by browsers’ native btoa()/atob(), not a “URL-safe” variant.)

 

#2: Automatic URL encoding

#userlandBase64_v3("some string",true)

This option URL-encodes the input string (using $esc.url) before Base64-ing it, which is critical if (a) you have non-Latin-1 characters in the string and (b) you want to use the browser’s native atob() to decode the result.[1]

 

#3: Basic + URL-safe characters

#userlandBase64_v3("some string",false,"RFC4648URLSafePadding")

This result is always safe to embed in URLs directly, even in the path, because it uses - and _ in place of the more sensitive + and /.

 

#4: URL encoding + URL-safe characters: The works

#userlandBase64_v3("some string",true,"RFC4648URLSafePadding")

The safest route, as it accounts for both non-Latin-1 in the input string and for possible URL conflicts.[2]

 

Get the code

I don’t want to digress into how the code works in this post, but perhaps there’ll be a Code Anatomy follow up... my version of the code has lots of comments!

 

Enjoy...

 

#**
* Base64 encoding in Velocity without Reflection
* @version v3 2010-10-11
* @author Sanford Whiteman, TEKNKL
* @license MIT
* @param $inputString String String to transform
* @param $alsoURIEncode Boolean [default false] URI-encode the String first (support browser-native atob())
* @param $charset String ["RFC4648" | "RFC4648URLSafePadding" | default "RFC4648"] 64-character set
*#
#macro( userlandBase64_v3 $inputString $alsoURIEncode $charset)
#set( $Base64PaddingChar = "=" )
#set( $Base64ByteChunkSize = 3 )
#set( $Base64BitGroupSize = 6 )
#set( $Base64Charsets = {
  "RFC4648" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
  "RFC4648URLSafePadding" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
} )
#set( $Base64Charset = $display.alt($Base64Charsets[$charset],$Base64Charsets["RFC4648"]) )
#set( $uriEncode = $display.alt($alsoURIEncode, false) )
#set( $Integer = 0 )
#set( $Long = $field.in($Integer).MAX_VALUE + 1 )
#set( $BigInteger = $field.in($Long).MAX_VALUE + 1 )
#set( $String = "" )
#set( $Base64Final = [] )
#if( $uriEncode )
#set( $inputString = $esc.url($inputString) )
#end
#set( $inputBytes = $inputString.getBytes() )
#set( $inputList = [] )
#set( $void = $inputList.addAll( $inputBytes.subList(0,$inputBytes.size())) )
#set( $inputSizeB = $inputList.size() )
#set( $padding =  $math.mod($inputSizeB, $Base64ByteChunkSize) )
#if( $padding > 0 )
#foreach( $pad in [1..$math.sub($Base64ByteChunkSize,$padding)] )
#set( $void = $inputList.add($null) )
#end
#end
#foreach( $byteGroup in [0..$math.sub($math.idiv($inputList.size(),$Base64ByteChunkSize),1)])
#set( $startOffset = $math.mul($byteGroup,$Base64ByteChunkSize) )
#set( $endOffset = $math.add($startOffset,$Base64ByteChunkSize) )
#set( $bytes = $inputList.subList($startOffset,$endOffset) )
#set( $integerList = [] )
#foreach( $byte in $bytes )
#if( $byte )
#set( $currentInteger = $convert.toInteger($byte) )
#set( $void = $integerList.add($convert.toInteger($BigInteger.valueOf($currentInteger).and($BigInteger.valueOf(255))) ))
#else
#set( $void = $integerList.add($null) )
#end
#end
#set( $binStrings = [] )
#foreach( $currentInteger in $integerList )
#set( $void = $binStrings.add($String.format("%8s",$Integer.toBinaryString($currentInteger)).replace(" ","0")) )
#end
#set( $allBinStrings = $display.list($binStrings,"") )
#set( $bitGroups = [] )
#foreach( $bitGroup in [0..$math.sub($math.idiv($allBinStrings.length(),$Base64BitGroupSize),1)] )
#set( $startOffset = $math.mul($bitGroup,$Base64BitGroupSize) )
#set( $endOffset = $math.add($startOffset,$Base64BitGroupSize) )
#set( $void = $bitGroups.add($allBinStrings.substring($startOffset,$endOffset)) )
#end
#foreach( $group in $bitGroups )
#if( !$group.contains("null") )
#set( $Base64CharsetPos = $Integer.parseInt($group,2) )
#set( $Base64Char = $Base64Charset.charAt($Base64CharsetPos) )
#else
#set( $Base64Char = $Base64PaddingChar )
#end
#set( $void = $Base64Final.add($Base64Char) )
#end
#end
$display.list($Base64Final,"")
#end

 

 


Notes

[1] Related browser-side decoding:

decodeURIComponent(
  atob(myEncodedString).replace("+"," ")
);

 

[2] Browser-side hint for this format:

decodeURIComponent(
  atob(
    myEncodedString.replace("-","+").replace("_","/")
  ).replace("+"," ")
);

 

Velocity does something magical when you just output a Lead field (without any other code at all).  The magic: it truly outputs the value as stored in Marketo. A {{lead.token}} doesn’t do that by default.

 

Community user AB wondered why a field containing an HTML table was revealing raw HTML (as text) in emails, as opposed to rendering the table.

 

That is, if a field looks like this in the UI:

 

 

Here’s what you see if you include {{lead.The Field With HTML}} in an email:

 

 

While you might have expected to see:

 

 

The reason you see the HTML-as-text is simple: Marketo HTML-encodes token values by default.

 

There’s nothing wrong, and a lot right, with this being the default behavior. It’s the same way that browsers deal with HTML-like text that’s not specifically inserted as HTML. To see what I mean, open a browser tab, go to  the Dev Tools Console, and run

 

document.body.insertAdjacentText("afterBegin","<table><tr><td>Stuff</td><tr></table>")

 

and you’ll see the code for a table, not a table!

 

In AB’s case the encoded value is:

 

&lt;table&gt;&lt;tr&gt;&lt;td&gt;Access Code&lt;/td&gt;&lt;td&gt;Remaining Uses&lt;/td&gt;&lt;td&gt;Start Date&lt;/td&gt;&lt;td&gt;End Date&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;hh7zh-lgyal&lt;/td&gt;&lt;td&gt;2.0&lt;/td&gt;&lt;td&gt;2018-12-31 08:00:00&lt;/td&gt;&lt;td&gt;2019-12-30 08:00:00&lt;/td&gt;&lt;td&gt;&lt;/table&gt;

 

Basic stuff, the < and > are encoded so the web browser/mail client won’t look for any deeper meaning in the text.

 

The old answer

The ready answer you might see for this is “Turn off HTML Encode Tokens in Field Management.”

 

I’ve given this answer myself. It works as a just-in-time fix. But I don’t feel it’s the right answer anymore, for multiple reasons:

 

  • you (the email author) may not have permission to make this change
  • if a field is used for multiple purposes, it may not be globally correct to leave the value unencoded
  • encoding is a secure default

 

The new answer

Using Velocity gives you local (per-email, even per-lead) control over whether the value is encoded.

 

To output a field without encoding, just create a {{my.token}} and drag that field to the canvas:

 

If you decide you do want to encode, Velocity has a method for that, EscapeTool.html:

 

${esc.html( $lead.TheFieldWithHTML )}

 

Now you’re in control of whether the output differs from the stored value.

 

Another related case

If you’re in the unlikely-but-not-unheard-of situation where an integration is plopping HTML into a field where you just want text, then it’s not just encoded vs. unencoded. You want the HTML tags outta there completely.

 

For that you use DisplayTool.stripTags:

 

${display.stripTags( $lead.TheFieldWithHTML )}

 

If you want to keep (whitelist) specific tags, pass all those tags as the 2nd+ arguments. With this VTL...

 

${display.stripTags( $lead.TheFieldWithHTML, "td", "tr" )}

 

... this original value...

 

<table>
<tr><td>Access Code</td><td>Remaining Uses</td><td>Start Date</td><td>End Date</td></tr>
<tr><td>hh7zh-lgyal</td><td>2.0</td><td>2018-12-31 08:00:00</td><td>2019-12-30 08:00:00</td></tr>
</table>

 

... gets output like so:

 

<tr><td>Access Code</td><td>Remaining Uses</td><td>Start Date</td><td>End Date</td></tr>
<tr><td>hh7zh-lgyal</td><td>2.0</td><td>2018-12-31 08:00:00</td><td>2019-12-30 08:00:00</td></tr>

 

Could be handy if you have a system that’s wrapping HTML in extra containers, disrupting your layout.

 

Note stripTags acts like textContent in a browser: all the matched opening and closing tags are removed, they’re not replaced by anything. So you might get unpretty output: <div>Sandy</div>Whiteman becomes SandyWhiteman (note the lack of space). Whether this works for your case depends on what’s coming in.

I obsess over questions like “What is a webhook?” like philosophers contemplate “What is consciousness?” (Or like others wonder “Are we merely 3-dimensional cartoons watched by N-dimensional beings?”)

 

It’s a different kind of search for truth, but better for your professional career.

 

  • Without the true (that is, RFC-based) answer to “What is a URL?” you’ll write a broken UTM parser.
  • If you use the term “CNAME” incorrectly, IT won’t take your requests seriously (or will botch the result).
  • If you don’t know what an SPF record can be, and when it, shall we say, ceases to be, your  deliverability will take a hit.
  • Without a (technical, not Shakespearean) dive into “What is a name?” you’ll mess up lots of things.

 

Standards, papers, and real-world experiences can reveal much wider definitions than you previously thought. The point of this post, for example, is that webhooks can take many more forms than you first imagine. And that should be exciting.

 

It starts from the start

The first mistake people make about webhooks is confusing (1) the webhook configuration (say, in Marketo Admin); (2) the webhook trigger (an event hook, which in Marketo means a triggered Smart Campaign); and (3) the remote web service that receives the webhook.

 

Far too frequently, (3) is mistakenly called the webhook. But it’s not.

 

You see, “the webhook” is the outbound HTTP request issued by a webhook-supporting app. It’s not the remote server that you connect to in hopes of a useful response – rather, it’s a feature of the platform that listens for the trigger. That is, for our purposes today, it’s a feature of Marketo.

 

Of course, without a webhook-compatible server on the other end, you won’t make successful use of webhooks! But you’re still sending a webhook, even if the other side is completely down.

 

The requirements for a server to be webhook-compatible are:

 

1. It runs over standard HTTP or HTTPS (that’s the “web” part). FTP servers, SMTP servers, servers that speak exotic binary protocols won’t cut it.[1]
2. It does not require multiple connections to perform a requested action. Webhooks are stateless. That means that OAuth-based systems requiring one HTTP connection to get an expiring token, followed by another connection do a lookup or update, aren’t webhook-compatible.[2]
3. It completes its requested action fast enough to handle invocation rates and within the HTTP timeout of the webhook side. “Fast enough” is a bit squishy because it depends on how many calls can be processed in parallel. Across the universe of webhooks, < 5s (including network time) is generally acceptable; < 1s is ideal. In Marketo specifically, < 30s is mandatory.[3]

 

Also note the creators of a service need not know it’s compatible to be compatible.[4]

 

The server doesn’t need to be a full-fledged service
You were supposed to notice something: the requirements above don’t include running a particular web language (PHP, Java, C#, JavaScript) nor any language at all!

 

That's because a webhook is still a webhook, even if it fetches static files – as long as the connection is over HTTP and finishes fast.

 

And a webhook need not fetch a different static file when called for different leads. Take a webhook you use to set default values. This could always get the same static JSON file defaults.json:

 

{
"FirstName" : "N/A",
"LastName" : "N/A",
"Company" : "[No Company Name]",
"Email" : "user@unknown.invalid"
}

 

This single ’hookable file serves a useful purpose, though it’s far from what people think of when you say “webhook”!

 

Taking it to the next level, you could use a set of static JSON files named by email address:

 

 

And a lead’s personal file could be dynamically fetched by the URL https://static.example.com/{{Lead.Email Address}}.json. For key accounts this would come in handy.

 

(You wouldn’t likely create tons of .json files by hand, in any case, but rather export them from a line-of-business app of some kind. The takeaway is that once static files are deposited on a webserver, no code needs to run to enrich a lead with their corresponding data.)

 

A static(ish) file that allows lookup

So far, the static file examples send back the  full contents of a static JSON file.

 

But what about doing a lookup within a large set of data to only return matches for the current lead?

 

That’s usually where people turn to a formal SQL or NoSQL database and, naturally, need to write the code to query that db. No longer is the webhook just fetching a prefab file with JSON. Instead, the webhook calls a web service, which is written in JavaScript, or C# or PHP or what-have-you. Then that service code queries the database using SQL or some other dialect, reading query info from the POST payload or from the GET query params.[5]

 

So when someone says, “Just use a webhook” (and I’m plenty guilty of this myself) unless they’re talking about calling an already-published service like Twilio SMS, they imply some kind of code. Perhaps simple code (to a developer’s eyes) but not no code.

 

But for certain types of data, it’s possible (and here is where this post starts to get strange) to create a webhook-compatible lookup service using only static files and standard HTTP headers. No code at all.

 

The key is HTTP Range (a.k.a. byte serving).

 

Homing in on Range:

The HTTP Range feature is amazingly ubiquitous: if you streamed a short video

today, you used it hundreds of times!

 

It works in such a simple way: you request a resource (URL) from a server and include a Range: header with a specific range of 1 or more bytes.[6]

 

The server, instead of responding with the entire resource, only sends you the bytes you requested.

 

So imagine you have this 26-byte, dirt-simple ASCII file shortsample.txt:[7]

 

ABCDEFGHIJKLMNOPQRSTUVWXYZ

 

And then you send this HTTP request:

 

GET /shortsample.txt
Range: bytes=0-4

 

The byte position always starts from 0. So this will get these first 5 bytes:

 

ABCDE

 

Presto! If you only wanted to peek at the first few characters in the file, you’ve just saved 80% of the potential network traffic. And now that you have the first 5, you can cache those and only get the remaining bytes, concatenating the files together afterward:

 

GET /shortsample.txt
Range: bytes=5-

 

FGHIJKLMNOPQRSTUVWXYZ

 

The Range: way of retrieving small segments at a time over standard HTTP is key to modern video streaming (Range Retrieval has been standard for 20 years, since RFC 2616, so it predates HTML5 video itself but has come into its own more recently). It’s also how we get pause/resumeable file downloads.

 

So how can Range: create a lookup-able, webhook-compatible service out of a single file? For that, you have to get inside my brain in the wee hours a few weeks ago (I never sleep because I’m thinking of things like this!).

 

Imagine the byte position (an Integer) can be a lookup key, and the character at each position (a one-letter ASCII String) can be the value at that key.

 

An example will — well, hopefully — help.

 

Say leads coming into your instance have a numeric License Code between 0 and 49 (in reality the range could start anywhere, just simplifying for now). The code indicates which kind of products they’re legally allowed to buy. Only people with certain codes are allowed to do business with your company. And it’s a binary/boolean thing: either they’re Allowed or Blocked, no nuances.

 

Create a text file like this that includes all 50 codes, in order from 0, with “A” for Allowed and “B” for Blocked:

 

ABBBAAAABBBABABBBAABBBBBAAAAAABABAAABAABAABAAABABA

 

Now, you can get the single byte at their License Code value to check if they’re allowed or not:

 

GET /licensecodes.txt
Range: bytes=23-23

 

That is, in a Marketo webhook’s Set Custom Headers config, where you can use {{lead.tokens}}:

 

Range: bytes={{Lead.License Code}}-{{Lead.License Code}}

 

 

That will return the one-character string

 

A

 

or

 

B

 

for every lead.

 

Did you have an “A-ha!” moment? I hope so.

 

Step by step

In the above example, the possible License Codes conveniently started at 0 and were all filled in through 49.

 

Let’s think of a somewhat more complex case, where the only codes used by the licensing bureau are 10-20 and (for whatever reason) 30-50.

 

Not a problem. Simply pad the file with 0s wherever you don’t have a possible value (you could use spaces or underscores or whatever, but 0s are easier to read on the blog, methinks):

 

0000000000BABABBBAABB000000000BABAAABAABAABAAABABA 

 

Byte #10 — a.k.a. License Code #10 — is still “B” for “Blocked”. Everything still works!

 

Why bytes and not bits?

If you’re with me so far you might be thinking, Why is he wasting whole bytes on letters, when there are 8 individual bits to fiddle with? I talk about that in the notes.[8]

 

We’re still getting underway (sorry!)

I know you’ve scrolled a lot already, but alas we’re only partway there.

 

Now, let’s walk through something that’s closer to a real-world scenario (I mean, the license code thing isn’t really that artificial, but I already had this other example in mind).

 

Imagine you‘re a US-based company and you assign every US ZIP code a certain grade that reflects its suitability for your product (please don’t assume I did anything but randomly generate a letter A-F, I didn’t pay attention to actual geography at all!). It’s a form of scoring, in other words.

 

Now, there are in theory 100,000 5-digit ZIP codes (00000-99999). In reality, only half of them are allocated (some are simply not used yet, some are permanently reserved). Here’s the top of an Excel sheet with the ones in use and their state/territory, in ascending order from the first one allocated (00501) to the last (99950), with lots of gaps of course:

 

 

Yes, I’m quick to admonish people for treating alphanumeric strings (like ZIP codes or, for other examples, credit card numbers or phone numbers) as if they’re truly numbers. So I’m not suggesting under any circumstances that ZIPs should be stored as Integers in a database. But for today’s purpose, a ZIP is usable as a numeric index. So let’s convert column A from Text to Number:

 

 

Then get the Sales team involved and assign grades:

 

 

Remember the lesson from the second License Codes example above. If we’re going to seek a specific byte position (ZIP 00501 = number 501 = byte 501 ), then every byte position must be occupied in some way. (To use technical jargon, you can’t have a “sparse array”, it needs to be “dense”.)

 

So we fill in the gaps so numbers that don’t have a state/territory are still present. Here’s how that looks, zoomed out a bit:

 

 

Then put a dummy grade of 0 for all the unused slots:

 

 

Now, the only thing we care about is Column C. That’ll be our database (if you will) of grades indexed by ZIP.

 

From here, we’re going to head over to a text editor to avoid any chaff that might come in from Excel.

 

Here’s just Column C, with carriage returns and/or line breaks:

 

We’ve got to strip out those CRLFs! They might add 1 or 2 bytes, throwing us totally off. Here’s the start of the file without CRLFs:

 

And another view after scrolling right a bit, so you can see the real grades after the hundreds of zeroes that have to kick it off (ZIP 00000/byte 0 through ZIP 00500/byte 500 are unassigned):

 

And finally, a deep look using a hex editor. You can see that the 501st byte, for ZIP code 00501, corresponds to the grade “B” (ASCII 0x42).

 

As far as the webhook-compatible file is concerned, we’re good to go! Now upload that file to a server somewhere (I’m using my Amazon S3 account) and let’s switch back to our beloved Marketo UI.

 

The webhook definition

Fetching the custom-crafted textfile with a lead-specific Range: header is simple. The main screen in Admin:

 

And the Set Custom Header dialog:

 

As you can see, we fetch the single byte at index {{lead.Postal Code}}.

 

Running a lead through the webhook, we see the expected single-byte response F:

 

 

So far, we’ve had success. The Range: is customized for the lead and honored by the remote S3 server, and we have a clean response.

 

Only the response is maybe too clean.

 

We’re not quite there

Here’s the rub.  Though there’s no particular response format that’s a requirement for being webhook-compatible, Marketo can only automatically map response values to lead fields if the response is valid XML or JSON. (It actually doesn’t matter whether the HTTP Content-Type is set, by the way, and Marketo, please don’t change this!)

 

In this case, the response is a single ASCII byte, which is totally valid text/plain of course but isn’t valid XML or JSON. Marketo doesn’t have a buit-in way to say “Map the entire response payload back to such-and-such field” so we have to take a more clunky approach (don’t worry, later in this endless post we’ll learn how to fix the clunk).

 

That clunky way is to use the undervalued Webhook is Called trigger. Since we only have 6 possible values, it’s easy enough to manage. You just need 6 Smart Campaigns with symmetrical structure, and the Response constraint matches each single-byte plain-text response:

 


Fixing the clunk

To avoid building multiple campaigns and use a standard webhook Response Mapping, we have to expand on another techno-ontological-philosophical question: “How do you generate a JSON response?

 

You probably have 2 valid answers in mind:

 

1. You return a static .json file stored on disk (that is, the entire file).
2. You generate the JSON text in server-side code, making sure to set your Content-Type to application/json.

 

Those are common, but what about an uncommon 3rd option: you store multiple valid JSON responses, end-to-end, in a single .txt file and use Range: to choose which block of JSON to return?

 

The file as a whole isn’t valid JSON. But you’re only reading parts, and those parts are valid.

 

For example, take this file:

 

{ "First Name" : "Marcus" }
{ "First Name" : "Cleopatra" }

 

 

That's not JSON taken all together (don’t mistake it for an array of 2 objects, as it’s missing the all-important [] and  , so would never parse correctly).

 

But the 29 bytes of the first line alone (including the CRLF) would be a single valid JSON object. And the 32 bytes of the second line alone would also be valid JSON.

 

Right now, the lengths of the 2 blocks are irregular. And since we don’t know in advance how long a First Name might be, that won’t yet work. So instead (for the sake of simplicity) let’s say it’s a maximum of 9 single-byte letters long (C-l-e-o-p-a-t-r-a). And we pad each line to exactly 32 bytes using harmless whitespace.

 

Using the ◌ character (in this post, not in the real file) to literalize the spaces and line break characters, and counting off the 32 bytes at the bottom:

 

{◌"First Name"◌:◌"Marcus"◌}◌◌◌◌◌
{◌"First Name"◌:◌"Cleopatra"◌}◌◌
{◌"First Name"◌:◌"Bob"◌}◌◌◌◌◌◌◌◌
00000000011111111112222222222333
12345678901234567890123456789012

 

Now we’ve created a file full of fixed width structured data. It doesn’t have any inherent meaning to the server hosting the file – it’s just like any text file – but to a client (i.e. Marketo) that knows how to seek into it, it’s like a giant, albeit simple, database of JSON objects.

 

We’re almost, sort of, there

With the above file structure, we know the width of an object is always 32. So if we want the Nth object, we seek to the position (N × 32 - 1) – recalling that positions start from 0 – and read the next 32 bytes.

 

But that li’l bit of multiplication there (N × 32) isn’t something we can do without... calling another more sophisticated webhook! Which would  defeat the purpose of this post. (Which was what again? Ah, to show how with a lot of ingenuity, you can make useful webhook-callable endpoints without writing any code.)

 

So we need to change our fixed block width to, well I’ll just cut to the chase: a multiple of 10.

 

Why? Because you can denote 10-byte zero-based ranges without doing the multiplication yourself.

 

Huh? Think about “0-9" – the first range of 10 bytes. “10-19" is the next range. 20-29 is next. Each of these strings can be created with simple variable substitution:

 

bytes={{some variable}}0-{{some variable}}9

 

Or with 100-byte blocks:

 

bytes={{some variable}}00-{{some variable}}99

 

100-byte blocks, though, could end up wasting a ton of space. Let’s not do that unless we have to. Instead let’s tighten our internal JSON objects as much as we can (the file doesn’t need to be human-readable, after all). If we use a single-character key v and remove other whitespace, we can get each block down to 8 bytes with an empty string value:

 

{"v":""}

 

Thus leaving us the headroom for 2 glorious bytes of string data! Since the example application here is single-letter grades A through F for every Postal Code, that's just fine.

 

And that's how the “Book of JSONs” file is laid out:

 

 

Now we need only set up a Marketo webhook that seeks 10 bytes at a time, offset by the Postal Code:

 

And the 10-byte response is valid JSON, ready for a standard Response Mapping back to our lead field:

 

Also notice how the Call Webhook dutifully records the HTTP 206 (Partial Content) as a success. Which it is!

 

Oh, and one more thing you can do with this is...

 

... basic score arithmetic (well, addition at least)

Y’know how Marketo can’t add 2 numbers on its own?

 

Here’s a primitive addition table created in Excel (via auto fill, I ain't that crazy!):

 

The column numbers are the first addend; the row numbers are the second addend.

 

You must consider the rows to start at Row 0, not Row 1 (something that Excel doesn’t allow, to this programmer’s dismay!). Count the columns from 0 in the same way, rather than as letters, so Excel’s Column D is Column 3.

 

The cell where a row & column meet is the sum of those numbers. For example, Excel cell D7 in zero-based terms = Column 3, Row 6. The value of that cell is 9 - which is the sum of 3 and 6!

 

By applying the same JSON-10-byte-block approach used above for ZIP codes, you can preload this addition table into Marketo as a bunch of individual files. Not to worry, I created the first 1000 columns for you, giving you the ability to sum any 2 numbers between 0 and 999.

 

This is one case where Marketo Sky (which, like most of you, I stil use only occasionally) shines over the legacy UI, since you can drag-and-drop multiple files. So download this file:

 

rangehook_mathjson_files.zip

 

Then unzip it and upload the contents to Design Studio into an appropriate folder:

 

 

Now you’re ready to do some simple addition.

 

Maybe you want to add the {{Lead.Demographic Score}} and {{Lead.Firmographic Score}} together into {{Lead.Aggregate Score}}.

 

You’d set up a webhook with the following settings (I’m too exhausted for another round of screenshots, but I trust you to know what to do!):

 

 

Where 123-ABC-456 is your Munchkin ID and na-sj01 is your direct asset URL (if you’re on instance app-sj01.marketo.com then the URL is na-sj01.marketo.com, etc.). Don’t use your Marketo LP domain here, it won’t work.

 

  • Custom Header: Rangebytes={{Lead.Firmographic Score}}0={{Lead.Firmographic Score}}9

 

  • Response Mapping: vAggregateScore

 

When run, this webhook will do a request like this, if Demographic Score is 12 and Firmographic Score is 30:

 

GET /rs/123-ABC-456/images/add-to-12.json
Range: bytes=300-309

 

Bytes 300-309 of add-to-12.json are, conveniently:

 

Giving us the correct sum, 42.

 

I’ll say it again: it ain’t pretty, but it is predictable and accurate. And it doesn’t require any external services, just a willingness to play on the wild side.

 

It’s not just a file download, it’s an indexed lookup!

It’s easy to confuse a Range request with a primitive file download. But it’s not, and the proof is in the performance.

 

Seeking the 1 billionth byte of a 1 GB file should be no slower than seeking the 1st byte. I’ve run convincing benchmarks with S3 and with IIS. Other HTTPds might not be as efficient, but that’s an engineering problem with those engines.[9]

 

Who is this for?

Hopefully, this post was interesting to anyone curious about HTTP, JSON, bytes and ranges and all that stuff.

 

But, to be clear, I don’t expect this far-out take on webhooks to be put into production by a totally non-technical person. It’s more for someone who may have some coding experience (from a past job or school) but – like most marketers – doesn’t have a corporate-approved place to host server-side code.

 

Even if you’re up to the challenge of writing secure, scalable webhook-compatible code... your employer may be (should be!) rightly uncomfortable about you “just” spinning up a production service somewhere in your personal cloud.

 

So the next best thing, for certain kinds of data, is a zero-code webhook running right out of your Marketo instance. No compliance issues, no security issues, no worries.

 

Or just enjoy the post as a peek into my sleepless mind.


Notes

[1] Counterexample: One of our clients runs an FTP server that accepts uploaded CSVs. A back end process periodically reads rows from the FTP’d files and upserts leads into Marketo. The deeper back end connects to Marketo via HTTP, sure – but the front end FTP server ain’t webhook-compatible.

 

[2] A webhook-compatible service needs to allow authentication + authorization info (if any) to be passed in the same HTTP connection as the requested action. Don’t be distracted by doomed hacks like storing access_tokens in lead fields. Even when they sort-of-sometimes work, they’re adding statefulness. You can’t expect two invocations of the same webhook, even for the same record in your database, to know anything about each other.

 

Note this doesn’t mean the service itself can’t call stateful APIs from the back end: we do this all the time! But the first-hop connection, from the webhook to the service, is stateless. Any additional network hops are hidden from the webhook.

 

[3] The “requested action” means the single GET or POST from the Marketo-like app. But that doesn’t mean all related actions on the other side are completed in that same short period!

 

Take an SMS webhook: its requested action is enqueueing the outgoing SMS message within the provider’s infrastructure, not delivering the message to the recipient’s handset. (Let alone listening for 2-way responses, which is way outside of webhook-land.)

 

Similarly, a service that inserts rows into a remote database need not have finished committing and/or replicating data (making it readable by other apps) by the time it returns 200 OK to Marketo. It might complete the insert a few seconds (or even minutes) afterward. What’s important is that the payload is eventually stored, not that it’s stored in real-time.

 

On the other hand: when the service offers data enrichment, field calculation, or remote lookup tables, the requested action does mean finishing everything before sending the HTTP response (typically JSON or XML). So some enrichment apps, particularly those that try to combine data from multiple back-end services (with each of those next-hop requests possibly requiring multiple connections) can end up being unusable via webhooks: they may have the other requirements down, but not the performance.

 

[4] I mentioned a while ago that Twilio’s Lookup API can be used by a Marketo webhook because it supports Basic Auth (username/password) credentials carried along with a lookup request. So it’s compatible, even though it’s not advertised with the word “webhook”.

 

A basic HTML form’s action URL is typically webhook-compatible as well. By definition, that URL expects x-form-www-urlencoded keys and values in a single GET or POST. So as long as the webhook-enabled app supports Form/URL encoding (Marketo does) then you can post from the server side as easily as from the client. That’s why you can use a webhook to call Marketo’s scriptless forms endpoint  /index.php/leadCapture/save to do some cool cross-lead stuff.

 

(Yes, CSRF tokens break this compatibility.)

 

[5] In some cases, a database has a native HTTP/S endpoint so there wouldn’t technically be a different tier for code. SQL Server used to have an XML service inside it, for example, but that’s been removed.

 

Document databases typically have an HTTP API built in. But if they require multi-request OAuth authentication, that endpoint would end up incompatible with webhooks. Similarly, OData services are always web-compatible but not necessarily webhook-compatible.

 

[6] Where supported, you can ask for more than one range, but for simplicity let’s assume it’s only a single contiguous range and a standard (non-multipart) HTTP 206 response.

 

[7] Simple ASCII to avoid confusion about the length of a UTF-8 file with or without the 2-byte BOM.

 

[8] True, there’s no general reason why a webhook response can’t be treated as binary and then chopped down further, 8 individual bits instead of 1 ASCII character (and in turn mapped to 8 Boolean fields).

 

But specifically within Marketo, this isn’t supported. In order for Marketo to treat B as 01000010 and then map 01000010 back to this...

 

Lead.Field 1 = false
Lead.Field 2 = true
Lead.Field 3 = false
Lead.Field 4 = false
Lead.Field 5 = false
Lead.Field 6 = false
Lead.Field 7 = true
Lead.Field 8 = false

 

... you’d have to pass the original B response to a whole other webhook, and that webhook would need to have real code behind it (albeit simple) as opposed to a static file. So it would defeat the purpose of today’s no-code experiment.

 

[9] Using Range: against dynamically generated resources (not static files) can be sketchy and cost more in server resources than it saves in bandwidth. But that’s a different situation from this post. Imagine you have a service that dynamically creates image files with certain transforms (sepia, bunny face, whatever). It’s likely impossible to know what the 999th through 1000th bytes will contain in advance. So it needs to render the transformed image to disk or memory, then jump to the 999th byte. If the whole image might only be 1K, that’s really wasteful and it would be better if it didn’t accept Range requests at all.

Say you have Badges Earned Custom Object ($BadgesEarned_cList in Velocity) to store a lead’s community achievements. Values are like so:

 

{description=Onboarding, points=200}
{description=Influencer, points=500}
{description=Helpful, points=200}
{description=Evangelist, points=350}

 

And you want to display the lead’s badges in a table with N alternating background colors. For example, with 3 alternating colors (leave aside the garish color scheme, that's not the point!):

 


This is is an old-school task that just about any template language can handle.[1] To start, we’ll stick with generic methods; later, we’ll see how Velocity’s Alternator helper class can save 1 or 2 lines of code.

 

As in other languages, alternation means a loop plus a modulo function (or native modulo operator[2]) which in Velocity is MathTool.mod:

 

#set( $rowColors = ["#ff4400", "#ccff00", "#0099cc"] )
#set( $numRowColors = $rowColors.size() )
#if( !$BadgesEarned_cList.isEmpty() )
<table style="border-top:10px solid #aaa;">
<tr bgcolor="#fff">
<td>Activity</td>
<td>Points Earned</td>
</tr>
#foreach( $badge in $BadgesEarned_cList )
#set( $rowColor = $rowColors[$math.mod($foreach.index, $numRowColors)] )
<tr bgcolor="${rowColor}" style="color:#333;">
<td>${badge.description}</td>
<td>${badge.points}</td>
</tr>
#end
</table>
#end

 

So I first set up an ArrayList, $rowColors, with the colors I want to alternate (in order from the top).

 

Then on every loop I get the modulo N of the loop index where N is the number of items in $rowColors.  (The list happens to have 3 items now, but the code dynamically adjusts if you add/remove colors.)

 

  • The first time through the loop, the loop index is 0.  0 modulo 3 is 0, so that means I use index 0 of $rowColors ($rowColors[0]) in turn.
  • Next time, the loop index is 1. 1 modulo 3 is 1. So $rowColors[1].
  • Next loop index is 2. 2 modulo 3 is 2: $rowColors[2].
  • Now the fun begins. Next time through the loop, the loop index is 3. 3 modulo 3 is 0. So we use $rowColors[0] again.
  • And that’s how alternating colors are done!

 

The HTML output is like so:

 

<tablestyle="border-top: 10px solid #aaa;">
<tr bgcolor="#fff">
<td>Activity</td>
<td>Points Earned</td>
</tr>
<tr bgcolor="#ff4400"style="color:#333;">
<td>Onboarding</td>
<td>200</td>
</tr>
<tr bgcolor="#ccff00"style="color:#333;">
<td>Influencer</td>
<td>500</td>
</tr>
<tr bgcolor="#0099cc"style="color:#333;">
<td>Helpful</td>
<td>200</td>
</tr>
<tr bgcolor="#ff4400"style="color:#333;">
<td>Evangelist</td>
<td>350</td>
</tr>
</table>

 

Simplifying a bit with AlternatorTool

You’ve seen above that without any “alternator-aware” code you can get exactly the output you want.

 

Velocity does offer a cool tool that abstracts away the modulo stuff. But as you can see in the Alternator source, it uses exactly the same method, just as compiled Java:

 

 

An Alternator might be infinitesimally faster because it’s compiled, but you’d never notice this in reality. The reason to use Alternators is to save lines of code, and every line does count in a language as verbose as VTL. Here’s how to get the same output using an Alternator:

 

#set( $rowColors = $alternator.manual(["#ff4400", "#ccff00", "#0099cc"]) )
#if( !$BadgesEarned_cList.isEmpty() )
<table style="border-top:10px solid #aaa;">
<tr bgcolor="#fff">
<td>Activity</td>
<td>Points Earned</td>
</tr>
#foreach( $badge in $BadgesEarned_cList )
#set( $rowColor = $rowColors.getNext() )
<tr bgcolor="${rowColor}" style="color:#333;">
<td>${badge.description}</td>
<td>${badge.points}</td>
</tr>
#end
</table>
#end

 

16 lines instead of 17: yay! And a little easier to read, maybe.

 

Create an Alternator object by passing a List to $alternator.manual. Velocity then handles the modulo-based loop internally, whenever you call getNext().

 

(If you’re confused about the difference between auto and manual, I don’t blame you, but trust me that manual + getNext() is what you always want, especially because of the more advanced application we’re going to do next.)

 

Alternating between complex objects

Alternating between Strings (single hex colors like "#ff4400") is the simplest task.

 

But let’s say you want to vary the background and foreground (text) colors for optimal contrast:

 


Now, you’ve got a set of 3 “color schemes” and each scheme has 2 characteristics (background and foreground).  You should already be thinking: an array of objects!

 

And that’s exactly what I do here, passing an [] of {}s – an ArrayList of LinkedHashMaps, technically – to AlternatorTool:

 

#set( $rowColorSchemes = $alternator.manual([
{
"bg" : "#ff4400",
"fg" : "#fee"
},
{
"bg" : "#ccff00",
"fg" : "#333"
},
{
"bg" : "#0099cc",
"fg" : "#ccff00"
}
]) )
#if( !$BadgesEarned_cList.isEmpty() )
<table style="border-top:10px solid #aaa;">
<tr bgcolor="#fff">
<td>Activity</td>
<td>Points Earned</td>
</tr>
#foreach( $badge in $BadgesEarned_cList )
#set( $rowColorScheme = $rowColorSchemes.getNext() )
<tr bgcolor="${rowColorScheme.bg}" style="color:${rowColorScheme.fg};">
<td>${badge.description}</td>
<td>${badge.points}</td>
</tr>
#end
</table>
#end

 

Each HashMap has two keys, bg and fg. Notice I only call getNext() once per iteration to advance to the next object in the List.

 

The generated HTML:

 

<tablestyle="border-top:10px solid #aaa;">
<tr bgcolor="#fff">
<td>Activity</td>
<td>Points Earned</td>
</tr>
<tr bgcolor="#ff4400"style="color:#fee;">
<td>Onboarding</td>
<td>200</td>
</tr>
<tr bgcolor="#ccff00"style="color:#333;">
<td>Influencer</td>
<td>500</td>
</tr>
<tr bgcolor="#0099cc"style="color:#ccff00;">
<td>Helpful</td>
<td>200</td>
</tr>
<tr bgcolor="#ff4400"style="color:#fee;">
<td>Evangelist</td>
<td>350</td>
</tr>
</table>

 

 

That’s it for Alternators for today! But I have another post ready to go on how to combine Iterators and Alternators for some advanced fun. Stay tuned.

 

 

 

Notes

[1] Many template systems have macros for N = 2, like isOdd and isEven, built-in. Few offer anything as flexible as AlternatorTool.

 

[2] Indeed, Java operators like % are semi-supported in Velocity as well. But the VTL parser is much stricter than Java’s, leading to hard-to-debug problems. I always use the MathTool methods instead.

The post title is a bit of a mouthful, but if you've been bitten by a certain feature gap you'll know what I mean.

 

One of the first things you learn about Wait steps is they don't have a literal Add Choice option.

nelson pointing at marketo

 

This can be frustrating when you want to vary the Wait delay based on  runtime conditions (that is, conditions you can’t know until the person has qualified and entered the flow) importantly including no delay at all.

 

But with a tiny bit of work, you can simulate Wait step choices.

 

It’s a matter of managing a Date/DateTime field, earlier in the same flow, using Marketo’s simple plus/minus support.

 

Here’s such a field:

 

field mgmt

 

And here’s a flow that uses that field to manage a subsequent Wait step:

 

flow steps

 

This approach works because of 3 convenient truths:

 

  • Change Data Value is synchronous within a single flow[1]
  • Date tokens understand a few math operators
  • Wait steps using a Date token will be skipped if the Date token is empty

 

Truth be told, I don't always endorse this tack over multiple Smart Campaigns. Whatever’s making you want drastically variable Wait periods (other than implicitly variable periods like wait-until-anniversary) may mean the lifecycle is going to differ in other ways as well, in which case discrete SCs help you keep your sanity. But it's there if you want it.

 

P.S. Yes, you can also use a Number {{my.token}} for the delay itself, a setup that might be almost too cool to follow! Or a Text {{my.token}} (don’t know why you’d choose this over Number, though) as long as you don’t include the unit  (“days”, “hours”) in the value, keep that hard-coded in the Change Data Value box.

 




Notes
[1] “Synchronous” meaning the New Value is guaranteed to be readable in the next flow step. Contrast this with, for example, webhook-based updates, which are asynchronous (background) value changes.

Despite instructing a Community member to “search my posts” the other day, I ran a search myself and there wasn’t a one-stop explanation of what Do Not Track (DNT) means in Marketo (on a deeper technical level than you get on the official doc page). So here goes.

 

As you probably know already, there are 2 DNT options, Ignore and Support:

 

 

We won’t worry about Ignore.

 

But what does it really mean to choose Support? On a technical level, it means one specific thing:

 

If a user’s browser sends the DNT: 1 HTTP request header along with a Munchkin-logged pageview or link click, Marketo will not save the activity to the Activity Log database.

 

So here are some things Do Not Track = Support does not do:

  • it does not stop gathering Clicked Email stats: email clicks are still tracked unless you separately turn off link tracking
  • it does not stop Munchkin JS libraries from loading
  • it does not stop Munchkin from initializing and setting its _mkto_trk cookie
  • it does not stop Munchkin from sending a Visit Web Page (assuming you're using the default configuration which always sends a VWP on startup)
  • it does not stop Munchkin from sending a Clicked Link for <a> links on the page

 

But again, here's the very important thing it does do:

  • it stops the Marketo platform from storing the Visit Web Page and Clicked Link hits sent by Munchkin

 

 

Why not stop Munchkin completely?

It's not that Marketo would not like to be more proactive on the browser side, I'm sure. But the weirdest thing about DNT is there's no programmatic (let alone cross-browser) way to know if the user has set a preference! Ergo, you cannot know if the person would've wanted you to turn off Munchkin downloading/initialization/hit logging. You have to dumbly send the hit in all cases, then the server will discard it if it's accompanied by the “please ignore me” header.

 

The privacy appeal of having the DNT setting be unreadable in the browser is clear — it's the equivalent of an HTTP-only cookie that can't be seen from JavaScript — but it certainly creates confusion. For example, someone with DNT enabled who’s also running Ghostery or similar will still see that the Munchkin tracking JS was blocked, which is suboptimal: ideally, it wouldn’t show up at all. You might seem like you’re being worse corporate citizens than you actually are. (A link on your Privacy Policy confirming that you honor Do Not Track is useful.)

 

 

The browser's-eye view

The browser sending the DNT: 1 header is a prerequisite, of course. Privacy-oriented browsers do this by default; other browsers do it in Private/Incognito/InPrivate mode only; the the rest do it for all pages/tabs/windows when selected. Here's the setting in an older version of Chrome, for one of a zillion examples, which will send DNT: 1 for all pages viewed in this user profile:

 

 

And here’s a screenshot of the HTTP request for the main document, showing the header:

 

 

And Munchkin’s Visit Web Page XMLHttpRequest, showing the same HTTP request header and its acknowledgment in the response:

 

Update 2019-11-07: This tip must be paired with the HTML and JS hints in this newer post to fully avoid accidental blocking by Tracking Protection.

 

Buried in a bunch of my Nation responses is this ginormously important guideline: whenever possible, use your Marketo LP domain in your form embed code instead of the default //app-something.marketo.com.

 

That is, if the embed code in the Marketo UI is:

 

<script src="//app-sj01.marketo.com/js/forms2/js/forms2.min.js"></script>
<form id="mktoForm_999"></form>
<script>MktoForms2.loadForm("//app-sj01.marketo.com", "123-ABC-456", 999);</script>

 

and your primary Marketo LP Domain (or a Domain Alias) is:

 

https://pages.example.com

 

then edit the embed code (after pasting on your external site) to be:

 

<script src="//pages.example.com/js/forms2/js/forms2.min.js"></script>
<form id="mktoForm_999"></form>
<script>MktoForms2.loadForm("//pages.example.com", "123-ABC-456", 999);</script>

 

The only and I mean only reason to avoid this change is if your external site requires SSL (https:) but your Marketo subscription does not yet include Marketo's SSL add-on. (Built-in browser security won't allow forms to load if you're in this unfortunate situation.)

 

In all other cases, you can and should switch over. Like, yesterday. That includes (1) when your external domain and Marketo LP domain both run over SSL (best practice in 2019); (2) when neither uses SSL (eh, it works); and (3) when the external site doesn't, but the Marketo domain does use SSL (strange but possible).

 

What's this about?

It's about tracking protection. If someone browses your site using Firefox with TP turned on, or with Ghostery or a similar plugin, they will not be able to load forms from app-*.marketo.com, because they can't load anything from domains matching *.marketo.*.

 

It makes sense that Munchkin (from munchkin.marketo.net) would be blocked, of course. That's what anti-tracking features/plugins are designed to do. But forms can be thrown out with the bathwater, if you will.

 

Yes, it's not really fair for all that matters! because form submissions require deliberate user action, and they don't inherently “track” anything but the Filled Out Form activity itself (assuming Munchkin cookies are blocked and all existing cookies were deleted).

 

But it's something we have to live with: Munchkin is fairly described as a tracker, Munchkin comes from the domain marketo.something and the major marketo.{tld} domains are of course all owned by Marketo. So, fair or not, privacy wins out... even if that means forms leave a blank space on your page for some end users.

 

By loading from your Marketo LP domain instead, you fully comply with the anti-tracking plugin (since you aren't dropping any new cookies or logging any more pageviews/clicks with Munchkin blocked) but also allow forms to be seen and be filled out. So do it!

 

 

Why isn't it the default?

Because of the SSL exception described above. Apparently, the Embed Code textbox in the UI and the underlying domain setup can't communicate. So the LP domain can't safely be the default, as not everyone can use it.

 

 

We're in the process of getting SSL on our Marketo LPs, but not sure if it's ready

If you have a pending order with Marketo, you can quick-check the state of affairs (for the purposes of this blog post) by loading the Forms 2.0 forms2.min.js in your browser. Here's Firefox's way of saying your custom cert isn't installed yet:

 

A clear symptom of broken custom JS is when you see form fields, in URL-encoded format, in the location bar after clicking Submit:

 

5c886b174c52a500bf5f9c28_fields_in_url_url.png

 

You should get to know the cause on sight: an uncaught error has been thrown in a custom Forms API onSubmit function.

 

To replicate this, just set up an otherwise empty page with your form embed and this purposely bad JS:

 

MktoForms2.whenReady(function(form){
form.onSubmit(function(form){
form.oohlala(); // the property `form.oohlala` will never exist
});
});

 

This code won't throw any early errors on page load, because it doesn't have a syntax error. There's no way for the JS engine to know that form.oohlala will end up being a nonexistent method at runtime, it just adds the onSubmit listener happily onto the form.

 

However, when the form is actually being submitted, Marketo runs your custom onSubmit function. Then the browser gets serious and throws a hard TypeError, as you can see in the Dev Tools console (make sure to check off Preserve Log or you'll miss it people don't realize this behavior is always, without exception, a JavaScript error because they fail to prepare their debugging environment ahead of time).

 

Here's the runtime error:

 

5c886b174c52a500bf5f9c28_fields_in_url_console_error.png

 

But why does a JS error result in fields in the URL?

It's simple. Marketo forms use a true HTML <form> tag (this is A Very Good Thing™) and standard <input>/<select>/etc. elements under the hood.

 

The Forms API, among many other things, is responsible for transforming the standard W3C submit event into an elegant cross-domain POST to your Marketo instance.

 

In an error-free environment, the standard submit is swallowed by the Forms API's robust set of handlers. The event is triggered, but its default action that is, sending fields to the form's action (destination URL) using its method (HTTP method) is turned off. The API's non-default actions take over.

 

But when there's an error thrown within the Forms API listener stack, the form reverts to standard HTML form behavior as the default action doesn't get a chance to be turned off.

 

The Marketo <form> element doesn't have a specific action or method[1], i.e. it looks basically like so:

 

<form>
<input name="Something">
<input name="SomethingElse">
<button type="submit">Submit</button>
</form>

 

The default value for method is GET, and the default value for action is the empty string which represents the current URL. So the effective markup is:

 

<form
method="GET"
action="https://whatever.example.com/the_current_page.html">
<input name="Something">
<input name="SomethingElse">
<button type="submit">Submit</button>
</form>

 

When the button is clicked with this markup, and there's a JS error, the browser reverts to a standard GET of the current URL with form fields in the query string:

 

https://whatever.example.com/the_current_page.html?Something=a%20value&SomethingElse=another%20value

 

Whenever you see this behavior in your browser, realize your forms aren't working at all (even though the form seemingly "posts" somewhere). So fix em!

 

 

 


Notes

[1] Nor does the <button> have the newfangled formaction or formmethod attributes, which would be used if present.

Printing fallback content when a Lead field is blank is a basic Velocity task. You can do it in a few lines of clunky code... but that's a few lines too many!

 

Seeking a one-liner, you might reach for Velocity's built-in $display.alt. But that won't fill the bill in Marketo-land. You see, $display.alt($field, $fallback) outputs the fallback if the field is null. But null isn't the same as the empty String that Marketo uses for unfilled Lead fields.

 

Therefore, the code

 

Dear ${display.alt($lead.FirstName,"Friend")},

 

will only ever output

 

Dear Joe,

 

or

 

Dear ,

 

It will never fall back to

 

Dear Friend,

 

because Marketo ensures $lead.FirstName is always a String of some kind, never null.[1]

 

So without any other tools at your disposal, you're left with a typically wordy #if block:

 

Dear ##

#if( $lead.FirstName.isEmpty() )

Friend,##

#else

$lead.FirstName,##

#end

 

This will work fine, but as your scripts get cluttered with repeats of this same structure, you start to go a little crazy.[2]

 

The good news is there's a short Velocimacro you can include globally that greatly lightens the load. Once you set up the #displayIfFilled macro, you can reduce the logic to one easy-to-read line:

 

Dear #displayIfFilled($lead.FirstName, "Friend"),

 

Building #displayIfFilled

Here's the macro definition:

 

#macro ( displayIfFilled $checkValue $fallbackValue )
#if( !($checkValue.isEmpty()) && !($checkValue == $display.get("0")) )
$!checkValue##
#else
$!fallbackValue##
#end
#end

 

As you can see, #displayIfFilled takes 2 self-explanatory arguments: the field to check for filled-ness, and the fallback value.

 

#displayIfFilled is designed to treat null the same as the empty String. It thus covers a superset of the cases covered by $display.alt, so you may never be tempted by the latter function again.

 

Step further out: #displayIf

We can abstract the functionality of #displayIfFilled into a more general #displayIf:

 

#macro ( displayIf $truePredicate $trueValue $falseValue )
#if( $truePredicate )
$!trueValue##
#else
$!falseValue##
#end
#end

 

#displayIf takes 3 arguments: a Boolean, the value to output if the Boolean is true, and the output if the Boolean is false.

 

To emulate #displayIfFilled, pass an isEmpty() check as the first arg (the $truePredicate):

 

Dear #displayIf($lead.FirstName.isEmpty(), "Friend", $lead.FirstName),

 

Lots of tricks up your sleeve with #displayIf. Say you want to switch output based on a specific non-empty value

 

Your #displayIf($lead.trialType.equals("Other"), "VIP", ${lead.trialType}) trial is almost over!

 

or output based on a date/time property

 

Good #displayIf($calNow.get($calFields.AM_PM).equals($calFields.AM), "mornin'", "aft'noon")!

 

or anything that can be expressed as if-then-else!

 

Of course, #displayIf can be overused; past a certain point of complexity, you should be using #if-#else on separate lines. But where a one-liner doesn't hurt readability, I say use it. #displayIf is critical to my sanity (if I have any left) as an avid Velocity coder.

 

Stay functional

There's another detail that you'd eventually learn on your own, but I'll spoil it to save you time.

 

When passing macro arguments in parentheses, you can include any chain of function calls but not syntactical expressions. For example, though Boolean operators and expressions are valid in other parts of Velocity, you can't do:

 

#displayIf(!$lead.FirstName.isEmpty(), "${lead.FirstName}'s", "Your") special offer is ready!

 

That won't compile because of the !. Velocity's parser doesn't accept operators in that place (nor would it accept <, > or == operators there).

 

Instead, either chain with the equals() function:

 

#displayIf($lead.FirstName.isEmpty().equals(false), "${lead.FirstName}'s", "Your") special offer is ready!

 

or as some programming style guides suggest anyway don't rely on negated Booleans and instead put your true case first:

 

#displayIf($lead.FirstName.isEmpty(), "Your", "${lead.FirstName}'s") special offer is ready!

 

To be clear, this doesn't mean you can't use all manner of operators and expressions to construct Boolean values, you just can't use them directly inside the parentheses when calling a macro. This will work fine:

 

#set( $hasCompany = !$lead.Company.isEmpty() && !$lead.Company.equals("N/A") )

Is #displayIf($hasCompany, $lead.Company, "your family") in the mood for pizza?

 

Disrupts the dream of a one-line solution, though.

 

What's with $display.get("0")?

Ah, yes. I don't want to overwhelm you earlier with the details of null checking in Velocity.

 

$display.get("0") (up above in the first #displayIfFilled macro) gets the value of a reference that is guaranteed to not exist, i.e. guaranteed to be null, in any Velocity context.

 

Why is it guaranteed to not exist? Because neither Java nor Velocity variable names are allowed to begin with a number.

 

Why not compare to the literal null? Because and this reality sneaks up in other important places in Velocity there is no four-letter keyword null! The null-ability of injected data is honored, even favored, in Velocity, like by $display.alt as noted above. But it doesn't have a keyword to create new null values easily.

 

So you have to find a roundabout way of getting a reference to a null value. Elsewhere on the net, people say "just use a variable you didn't #set anywhere else, like $abcdefg, as that will naturally be null." The flaw in this reasoning is nothing actually stops someone else (either a future coder or a current collaborator) from using $abcdefg for something else. So I prefer to use a reference that cannot exist even by coincidence.

 

 


 

 

Notes

[1] In the Marketo Lead/Person world, you aren't gonna run into literal null values, but rather empty strings. This is true despite non-filled fields being represented as [null] or NULL in parts of the Marketo UI. You can and will encounter null with Custom Objects, though. That's the stuff of another post.

 

[2] Yes, you could smush the VTL into one line. But if you think Dear #if($lead.FirstName.isEmpty())Friend#else${lead.FirstName}#end, is sufficiently readable, you're made of stronger stuff than me.

Filter Blog

By date: By tag: