Base64-ing a String in Velocity without using Reflection

SanfordWhiteman
Level 10 - Community Moderator
Level 10 - Community Moderator

The removal of Reflection-based features from Marketo’s Velocity environment in June 2019 was a disappointment — however inevitable — to those of us who push email scripting to the limit.

 

Requirements like encoding, hashing, encryption, flexible rounding and randomization, XPath queries, and advanced filters/sorts on Custom Objects could no longer be met with Marketo alone. (Webhooks can fulfill a subset of those needs, but with nowhere near the performance and simplicity of Velocity.)

 

At the prodding of Community user TM, I set out to see if Base64 encoding, one of the Java methods that used to be callable from Velocity, could be written with “pure” Velocity and public Java methods, without relying on the Reflection API.

 

I was indeed able to do it... though unlike most of my Velocity work, this one wouldn’t File Under Fun!

 

You can call the #userlandBase64_v3 macro below in 4 ways:

 

#1: Basic usage

#userlandBase64_v3("some string")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The result uses the standard RFC4648 Base64 character set (A-Za-z0-9+/). (That is, the characters used/expected by browsers’ native btoa()/atob(), not a “URL-safe” variant.)

 

#2: Automatic URL encoding

#userlandBase64_v3("some string",true)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This option URL-encodes the input string (using $esc.url) before Base64-ing it, which is critical if (a) you have non-Latin-1 characters in the string and (b) you want to use the browser’s native atob() to decode the result.[1]

 

#3: Basic + URL-safe characters

#userlandBase64_v3("some string",false,"RFC4648URLSafePadding")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This result is always safe to embed in URLs directly, even in the path, because it uses - and _ in place of the more sensitive + and /.

 

#4: URL encoding + URL-safe characters: The works

#userlandBase64_v3("some string",true,"RFC4648URLSafePadding")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The safest route, as it accounts for both non-Latin-1 in the input string and for possible URL conflicts.[2]

 

Get the code

I don’t want to digress into how the code works in this post, but perhaps there’ll be a Code Anatomy follow up... my version of the code has lots of comments!

 

Enjoy...

#**
 * Base64 encoding in Velocity without Reflection
 * @version v3 2010-10-11
 * @author Sanford Whiteman, TEKNKL
 * @license MIT
 * @param $inputString String String to transform
 * @param $alsoURIEncode Boolean [default false] URI-encode the String first (support browser-native atob())
 * @param $charset String ["RFC4648" | "RFC4648URLSafePadding" | default "RFC4648"] 64-character set 
 *#
#macro( userlandBase64_v3 $inputString $alsoURIEncode $charset)
#set( $Base64PaddingChar = "=" )
#set( $Base64ByteChunkSize = 3 )
#set( $Base64BitGroupSize = 6 )
#set( $Base64Charsets = {
  "RFC4648" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
  "RFC4648URLSafePadding" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
} )
#set( $Base64Charset = $display.alt($Base64Charsets[$charset],$Base64Charsets["RFC4648"]) )
#set( $uriEncode = $display.alt($alsoURIEncode, false) )
#set( $Integer = 0 )
#set( $Long = $field.in($Integer).MAX_VALUE + 1 )
#set( $BigInteger = $field.in($Long).MAX_VALUE + 1 )
#set( $String = "" )
#set( $Base64Final = [] )
#if( $uriEncode )
#set( $inputString = $esc.url($inputString) )
#end
#set( $inputBytes = $inputString.getBytes() )
#set( $inputList = [] )
#set( $void = $inputList.addAll( $inputBytes.subList(0,$inputBytes.size())) )
#set( $inputSizeB = $inputList.size() )
#set( $padding =  $math.mod($inputSizeB, $Base64ByteChunkSize) )
#if( $padding > 0 )
#foreach( $pad in [1..$math.sub($Base64ByteChunkSize,$padding)] )
#set( $void = $inputList.add($null) )
#end
#end
#foreach( $byteGroup in [0..$math.sub($math.idiv($inputList.size(),$Base64ByteChunkSize),1)])
#set( $startOffset = $math.mul($byteGroup,$Base64ByteChunkSize) )
#set( $endOffset = $math.add($startOffset,$Base64ByteChunkSize) )
#set( $bytes = $inputList.subList($startOffset,$endOffset) )
#set( $integerList = [] )
#foreach( $byte in $bytes )
#if( $byte )
#set( $currentInteger = $convert.toInteger($byte) )
#set( $void = $integerList.add($convert.toInteger($BigInteger.valueOf($currentInteger).and($BigInteger.valueOf(255))) ))
#else
#set( $void = $integerList.add($null) )
#end
#end
#set( $binStrings = [] )
#foreach( $currentInteger in $integerList )
#set( $void = $binStrings.add($String.format("%8s",$Integer.toBinaryString($currentInteger)).replace(" ","0")) )
#end
#set( $allBinStrings = $display.list($binStrings,"") )
#set( $bitGroups = [] )
#foreach( $bitGroup in [0..$math.sub($math.idiv($allBinStrings.length(),$Base64BitGroupSize),1)] )
#set( $startOffset = $math.mul($bitGroup,$Base64BitGroupSize) )
#set( $endOffset = $math.add($startOffset,$Base64BitGroupSize) )
#set( $void = $bitGroups.add($allBinStrings.substring($startOffset,$endOffset)) )
#end
#foreach( $group in $bitGroups )
#if( !$group.contains("null") )
#set( $Base64CharsetPos = $Integer.parseInt($group,2) )
#set( $Base64Char = $Base64Charset.charAt($Base64CharsetPos) )
#else
#set( $Base64Char = $Base64PaddingChar )
#end
#set( $void = $Base64Final.add($Base64Char) )
#end
#end
$display.list($Base64Final,"")
#end‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

 

 


Notes

[1] Related browser-side decoding:

decodeURIComponent(
  atob(myEncodedString).replace("+"," ")
);‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

 

[2] Browser-side hint for this format:

decodeURIComponent(
  atob(
    myEncodedString.replace("-","+").replace("_","/")
  ).replace("+"," ")
);‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

 

5145
6
6 Comments
Alex_Munro1
Level 1

This needs way more likes 😁

akhandoker_tm
Level 2

Hi @SanfordWhiteman , I'm attempting to encode an email address into a URL and then decode later for a form.

 

The script itself works wonderfully, so thank you for that. An issue arises, though, when adding the encoded string into a URL query string. It adds literally thousands of "%20" values (space-character) before the encoded string. 1,592 to be exact.

 

I saw in another post from you that space-characters and line breaks can sometimes be explicitly added to a script's output but, when I count those, it results to around 260.

Is there something I'm doing wrong that results in this? Would love any instructions on how to correct this. See below for my script:

#**
 * Base64 encoding in Velocity without Reflection
 * @version v3 2010-10-11
 * @author Sanford Whiteman, TEKNKL
 * @license MIT
 * @param $inputString String String to transform
 * @param $alsoURIEncode Boolean [default false] URI-encode the String first (support browser-native atob())
 * @param $charset String ["RFC4648" | "RFC4648URLSafePadding" | default "RFC4648"] 64-character set 
 *#
#macro( userlandBase64_v3 $inputString $alsoURIEncode $charset)
#set( $Base64PaddingChar = "=" )
#set( $Base64ByteChunkSize = 3 )
#set( $Base64BitGroupSize = 6 )
#set( $Base64Charsets = {
  "RFC4648" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",
  "RFC4648URLSafePadding" : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
} )
#set( $Base64Charset = $display.alt($Base64Charsets[$charset],$Base64Charsets["RFC4648"]) )
#set( $uriEncode = $display.alt($alsoURIEncode, false) )
#set( $Integer = 0 )
#set( $Long = $field.in($Integer).MAX_VALUE + 1 )
#set( $BigInteger = $field.in($Long).MAX_VALUE + 1 )
#set( $String = "" )
#set( $Base64Final = [] )
#if( $uriEncode )
#set( $inputString = $esc.url($inputString) )
#end
#set( $inputBytes = $inputString.getBytes() )
#set( $inputList = [] )
#set( $void = $inputList.addAll( $inputBytes.subList(0,$inputBytes.size())) )
#set( $inputSizeB = $inputList.size() )
#set( $padding =  $math.mod($inputSizeB, $Base64ByteChunkSize) )
#if( $padding > 0 )
#foreach( $pad in [1..$math.sub($Base64ByteChunkSize,$padding)] )
#set( $void = $inputList.add($null) )
#end
#end
#foreach( $byteGroup in [0..$math.sub($math.idiv($inputList.size(),$Base64ByteChunkSize),1)])
#set( $startOffset = $math.mul($byteGroup,$Base64ByteChunkSize) )
#set( $endOffset = $math.add($startOffset,$Base64ByteChunkSize) )
#set( $bytes = $inputList.subList($startOffset,$endOffset) )
#set( $integerList = [] )
#foreach( $byte in $bytes )
#if( $byte )
#set( $currentInteger = $convert.toInteger($byte) )
#set( $void = $integerList.add($convert.toInteger($BigInteger.valueOf($currentInteger).and($BigInteger.valueOf(255))) ))
#else
#set( $void = $integerList.add($null) )
#end
#end
#set( $binStrings = [] )
#foreach( $currentInteger in $integerList )
#set( $void = $binStrings.add($String.format("%8s",$Integer.toBinaryString($currentInteger)).replace(" ","0")) )
#end
#set( $allBinStrings = $display.list($binStrings,"") )
#set( $bitGroups = [] )
#foreach( $bitGroup in [0..$math.sub($math.idiv($allBinStrings.length(),$Base64BitGroupSize),1)] )
#set( $startOffset = $math.mul($bitGroup,$Base64BitGroupSize) )
#set( $endOffset = $math.add($startOffset,$Base64BitGroupSize) )
#set( $void = $bitGroups.add($allBinStrings.substring($startOffset,$endOffset)) )
#end
#foreach( $group in $bitGroups )
#if( !$group.contains("null") )
#set( $Base64CharsetPos = $Integer.parseInt($group,2) )
#set( $Base64Char = $Base64Charset.charAt($Base64CharsetPos) )
#else
#set( $Base64Char = $Base64PaddingChar )
#end
#set( $void = $Base64Final.add($Base64Char) )
#end
#end
$display.list($Base64Final,"")
#end
#userlandBase64_v3($lead.Email,false,"RFC4648URLSafePadding")

 

SanfordWhiteman
Level 10 - Community Moderator

You‘re setting

alsoURIEncode 

to false, so it wouldn’t be URI-encoded at all in Velocity.

 

Please provide the sample input and output you’re seeing. And copy it from the source of a received email (i.e. the raw message).

akhandoker_tm
Level 2

New finding: it seems to add an endless string of "%20" for any string added via Velocity token into the href value within a link. It doesn't resolve anything but, it happens even if the Velocity token is as simple as passing an unmodified token through (eg. "$lead.Email").


When sending a sample email, I'm faced with this error:

An error occurred when procesing the email Rendered_Email_Velocity_Error_Area_?!

Invocation of method 'subList' in class java.util.ArrayList threw exception java.lang.IndexOutOfBoundsException: toIndex = 3 near

?

 

 

When approving the email, I'm faced with this error instead:

Validation Error approving 21-Q4-NABU-US-ALL-TME-New Template Testing.Email-1_Invite-1 —  <div>An error occurred when procesing the email Rendered_Email_Velocity_Error_Area_?! </div> <p>Invocation of method 'subList' in class java.util.ArrayList threw exception java.lang.IndexOutOfBoundsException: toIndex = 3 near</p> <div><pre >?</pre></div>

Michael_Knephe1
Level 1

Hi @SanfordWhiteman I have a project this macro was tailor-made for, but I'm having similar issues as the prior comment with the output pre-pending tons of %20 characters to the token. I've tried using the different flag options, but am getting the same result each time. I copied and pasted from the code block above, so I'm not sure if there's a stray character that's not playing nice when pasting into a code editor/marketo's token editor? Any help would be greatly appreciated.

SanfordWhiteman
Level 10 - Community Moderator

Make sure there aren’t any trailing control characters. I think the blog post might have some extra junk injected by the forum software.

 

Here‘s the (correct) result of

#userlandBase64_v3($lead.FirstName,true,"RFC4648URLSafePadding")

for the First Name value

Sandy-ford

SanfordWhiteman_1-1730417502243.png