Send SMS with an emoji using Twilio webhook, comes through as plain text only?

[KevinTruong]

Hi all

Does anyone use Twilio for sending SMS messages from Marketo? I have an issue that's stumping me.

We want to send SMS messages that have emojis in them eg 🌼 https://www.compart.com/en/unicode/U+1F33C

We send these messages from Marketo -> Twilio via a webhook. In this webhook are Marketo tokens which is where we configure the content of the message.

This webhook passes the payload to Twilio in plain text, which strips out the emojis. If I try to add them in unicode or HTML formats ( U+1F33C or 🌼), Twilio still treats them as plain text and fails to send them as emojis. So eg an SMS saying "Here is a flower 🌼" turns into "Here is a flower U+1F33C". 

I can't tell if this is an issue with how Marketo sends the information or how Twilio is receiving it. I've lodged tickets with both platforms but hoping someone has come across this before?

[SanfordWhiteman · Accepted solution]

The format “U+1F33C” is not an encoding. It’s the way we discuss Unicode codepoints specifically independent of any encoding.

HTML encoding doesn’t make sense here because you aren’t sending HTML.

Twilio’s Message resource expects a URL-encoded (i.e. percent-encoded) payload.

[KevinTruong]

Thanks @SanfordWhiteman. I'm not familiar with percent encoding, what do I need to do to send an emoji in a Marketo token to Twilio using their webhook?

[SanfordWhiteman]

If you set the webhook to use Form/Url request encoding, Marketo automatically URL-encodes any tokens.

Please show the config of your webhook and the tokens on the Tokens tab.

[KevinTruong]

Request encoding is already set to Form/URL:

We then enter the content of the SMS into the token called "SMS Message Body 1" which is just a normal text token:

We then send the SMS using the Call Webhook flow step. 

[SanfordWhiteman]

You obfuscated the text of the token, which is part of what we need to see.

[KevinTruong]

The exact text of the token we're using isn't suitable for public but imagine it is something like this:

Hello, here is a flower 🌼

It is a normal text token so the text characters work fine, however Marketo will delete the emoji from the token when I try to paste it in as is. Marketo support say I need to escape the emoji characters in some way that Twilio will accept, but Twilio support can't give details on what that is supposed to be.

[SanfordWhiteman · Accepted solution]

I mean you can easily put up a test program where the token can be anything you want. Much easier to get help if you don’t redact the important part.

Twilio doesn’t need to tell you anything because they expect URL-encoded. They’re fine.

Marketo is removing the text because the platform doesn’t support emoji literals (or any unencoded surrogate pairs, actually — I’ve blogged about this recently) in string fields.

So you need to pre-encode the value and then set Request Token Encoding to None.

[KevinTruong]

How does this work in the context of an emoji in a sentence?

So say the end result is we want is to send an SMS saying "Hello, here is a flower 🌼"

Our webhook is currently configured to deliver this payload:

From=[phonenumber]&MessagingServiceSid=[account]={{lead.Mobile Phone Number:default=edit me}}&Body= Hi {{lead.First Name}}, {{my.SMS Message Body1}}

If I'm understanding you correctly, I have to create a separate token for the emoji only, so the webhook would need to look like this?

From=[phonenumber]&MessagingServiceSid=[account]={{lead.Mobile Phone Number:default=edit me}}&Body= Hi {{lead.First Name}}, {{my.SMS Message Body1}} {{flower emoji}}

And then if we wanted to have multiple emojis throughout the text eg "Hello, here is a flower 🌼 and a heart 💛 and a present 🎁" the webhook would have to look like:

From=[phonenumber]&MessagingServiceSid=[account]={{lead.Mobile Phone Number:default=edit me}}&Body= Hi {{lead.First Name}}, {{my.SMS Message Body1}} {{flower emoji}} {{my.SMS Message Body2}} {{heart emoji}} {{my.SMS Message Body3}} {{gift emoji}}

Is that correct? What would the request encoding be if the text portions require Form/URL and the emojis need none?

[SanfordWhiteman]

You need to have the whole payload pre-URL-encoded. It doesn’t matter if you use one or many tokens, they all need to be URL-encoded.

Unfortunately, this means your {{lead.First Name}} token will break if there are any reserved characters that require URL-encoding — or more properly, IRI-encoded, as I’m pretty sure Twilio doesn’t mind a part-encoded string. For example, if the First Name contains a literal plus sign, equals sign, or ampersand.