Velocity: Escape unicode

[Markus_Bianchi5]

What would be the best / most efficient way in velocity to automatically convert (ideally all or at least some) special characters to the equivalent unicode escape code, e.g.

Ä -> \u00c4,

ä -> \u00e4

Ö -> \u00d6

ö -> \u00f6

Ü -> \u00dc

ü -> \u00fc

ß -> \u00df

I haven't found some built-in function that can automatically convert these characters and a simple

$str.replace("ä", "\u00c4")

doesn't seem to work in some initial tests and would also mean that I have to replace each character separately (unless there is some way to replace multiple characters in one go).

Sanford Whiteman maybe?

Many thanks,
Markus

[SanfordWhiteman]

What are you trying to accomplish exactly?

Velocity itself understands UTF16/UTF8.

There are tens of thousands of special characters if you mean non-ASCII7.

Tell me more about your project!

[Markus_Bianchi5]

We have to send a CSV file every time someone submits a form (so ideally as an attachment to an alert email). Since we can't attach files to emails in Marketo, we could

... either use a web hook or similar to generate and store a CSV file externally and respond with a link to that CSV file that we could then use in the alert email (most likely the better solution but not feasible as this is only a temporary workaround and would be too time consuming)

... or "simply" send a CSV string in the alert email ("simply" is in quotes as there are quite a lot of things to look out for in CSV files... but again only temporary). Fortunately, we are sending the alert email to an external tool that can get easily get the CSV string from the email and work with it (tested and working just fine). Generating the CSV string in Velocity also works (there are probably some cases where this will not generate a properly formatted CSV string but we can handle these cases either manually for the time being or adjust the velocity script accordingly). The "bad" part is that according to what I was told/seen the tool doesn't play well with special characters (probably uses some incorrect encoding) and requires that these characters are transferred to the equivalent unicode escape code. I still have to investigate if there is any way to teach this tool to use a proper encoding (ideal solution) but already wanted to double check (just in case) if there is an easy way in velocity to transfer these characters automatically. If there isn't (at this point I'm pretty sure there is not?), then it would probably be easier to adjust this tool accordingly as manually replacing all special characters is almost impossible and only replacing a few of them not ideal (could result in a lot of manual work to fix these issues afterwards).

Also, it has to be a CSV file for various reasons... Otherwise, we could directly use a webhook and send the data as e.g. JSON.

[SanfordWhiteman · Accepted solution]

if there is an easy way in velocity to transfer these characters automatically.

Oh, there is, I just cringe at the need. But I live in a perpetual state of cringe at what people need to put up with from their connected apps.

EscapeTool's propertyValue will do it:

#set( $nativeToAscii = $esc.propertyValue($variable) )

$esc.propertyValue is almost like $esc.java but with an important difference: java escapes quotation marks, which can be very disruptive (you can re-replace them afterwards, but propertyValue does it in one shot).

propertyValue takes its name from Java's rules for .properties files (like .ini files on Windows, .conf files on Unix/Linux, etc.). When read from disk they can only contain ASCII, a surprising but lasting restriction.

See if that works for you.

[Markus_Bianchi5]

Thanks a lot for the help and explanation, that works just fine!

Oh, there is, I just cringe at the need.

Me too