Issues with Unicode uploads

[SanfordWhiteman]

For my usual nitpicky reasons I'm gonna point out that Unicode isn't a file format (despite the label): it's still UTF-16 LE, which isn't any more legit Unicode than UTF-8.

There's clearly a regression, of course. It may be in Marketo's endianness (expecting BE as opposed to LE).

[Anonymous]

Anyone have any luck yet? Marketo Support is saying that it should work as UTF-8, but I am not getting any different results.

[Dan_Stevens_]

Nothing yet.  Support is still telling me they're looking into it. We have a bunch of lists we need to upload, but can't.

[SanfordWhiteman]

Hey Dan,

Convert Excel's UTF16LE file to UTF8 and it'll work.

To do this I used the classic ​iconv​ which you can get from LibIconv for Windows

Command line:

iconv -f UTF-16 -t UTF-8 "E:\My Documents\unicode_export_utf16lebom.txt">"E:\Unicode_tests\unicode_export_utf8nobom.txt"

Screenshot showing success:

[Dan_Stevens_]

Thanks Sandy, but these are already in UTF-8 format.  Even if this workaround is a solution for whatever reason, we can't expect our marketers in every country to download this software (which will have to go through a security review) and process every list using the command line approach.  The fact is this was such an easy process for us for years - take Excel file, save as Unicode text file, import into Marketo.  Done.  And now, since the Winter release, this no longer works.

[SanfordWhiteman]

Excel "Unicode Text" is UTF-16LE, not UTF-8.  UTF-16LE w/BOM (which Marketo is misleadingly calling "binary") is no longer supported, apparently, but UTF-8 w/no BOM is ok.

Thought you said you had lists waiting to be imported, so this would be how to get 'em done.

(FWIW you could write an Excel macro to do the conversion, too -- I just didn't feel like it as iconv was out there, and it's been a standard Linux/Mac OS tool for many years.)

[Dan_Stevens_]

So even when saving a Unicode text file as follows, it's still not really UTF-8?

As far as it not being supported any more, the last I heard from support is the following:

This issue is affecting multiple customers, and Engineering is working diligently to see this resolved. I will update you further as progress is made.

So hopefully it's still supported and just considered a bug that's going to eventually be resolved.

[SanfordWhiteman]

That dialog only applies to saving as HTML! The TSV generated is still ISO-8859 (or maybe Windows-1252) not UTF-8.

Note (since there seems to be confusion above and from Marketo) Excel UTF-16LE is still tab-delimited text, not "binary"... there's no magical format that avoids the need for a delimiter w/variable-length data!

You could have UTF-16LE that's comma-delimited as well. The question of delimiter collisions (i.e. commas within data) is a good one, but quoting should automatically take care of that. Also, using tabs instead of commas just shifts the delimiter collision over to tabs, it doesn't eliminate it. You can have tab characters within a table cell/column, too.

[SanfordWhiteman]

Danny Tran​ Gary Verster​ Dan Stevens​ I packaged the encoding conversion as a simple Excel VBA macro.

You can get it from here: https://s3.amazonaws.com/blog-images-teknkl-com/marketo_import_template_v1.xlsm

When you run the macro ExportAsUTF8 you get a Save As dialog:

The filename you choose is stored as the standard Excel "Unicode Text" (UTF16LE as mentioned above), and also saved to a separate file as UTF8 (in the same directory) that's suitable for Marketo importing. The latter is suffixed "_MarketoReady_utf8".

Give it a shot and let me know how it goes. The sheet was Authored in Excel 2007/Windows and tested in Excel 2013/Windows as well.

[Dan_Stevens_]

Thanks Sandy - that worked (using Excel 2016).  Really appreciate you taking the time to develop this interim solution for us while the engineers at Marketo figure out a way to fix this bug.

Latest response from Marketo Support:

We are continuing to investigate the full scope of this issue, prior to making an immediate changes. I will update you further as progress is made.