34 Replies Latest reply on Jul 12, 2017 2:07 PM by Dan Stevens.

    Issues with Unicode uploads

    Danny Tran

      Has anyone else noticed a change when uploading unicode files? I tried several different methods of uploading a file (including this Import a Non-Latin Characters List - Marketo Docs - Product Docs), but I always seem to have to retype the headers in the upload process:

      I also noticed that the bottom of the form says that zero fields are ignored, but that's definitely not the case.

       

      Any one have insight or want to vent about the same issue?

        • Re: Issues with Unicode uploads
          Addy Sharma

          I noticed it once when I used an incorrect header format in the csv file after I chnaged the header name it worked fine.

            • Re: Issues with Unicode uploads
              Danny Tran

              Hey Addy - Thanks for the note. I haven't noticed any issues with Tab Delimited or CSV files myself. Do you import lists containing Asian characters using CSV?

                • Re: Issues with Unicode uploads
                  Dan Stevens.

                  CSV should never be used for advanced/alternative character sets - always use Unicode Text.  It's also worth mentioning that you should always define the file format and never leave this field as "auto-detect" (I wish Marketo would just remove that option) as this often causes issues.  When importing Unicode Text, be sure to select "Tab Delimited".

                   

                   

                  It also looks like you have some invalid characters in your header row:

                   

                    • Re: Issues with Unicode uploads
                      Danny Tran

                      Hey Dan,

                       

                      Thanks for your response. The form is automagically changing the header after I start the upload process. I've already tried copying and pasting the entire spreadsheet into a different file and saving it like that. No luck. I tried rewriting all of my column headers and saving the file like that. Same thing. I also never choose the "Auto-Detect" option and totally agree with you that it should be eradicated from the menu. I opened  a ticket with Mkto Support in hopes to figure this out. This issue is becoming a timesuck for the leads that can't be updated/created via API.

                       

                      Thanks again,

                       

                      Danny T.

                        • Re: Issues with Unicode uploads
                          Gary Verster

                          Hi everyone,

                           

                          We're experiencing the same issue. We've been uploading Unicode .txt files from a standard upload template for the last 18 months with no problems, always using the tab delimited option, and then suddenly this week we started havng the same problems that Danny is having. We haven't made any changes to the header or the columns and yet Marketo is picking up the strange characters before the Email Address field and then not mapping any of the columns to field names. Also worth noting that manually mapping the fields does NOT work... the record is imported, but the values don't map properly.

                           

                          Something is definitely broken... support case also opened. Fingers crossed...

                           

                          Gary

                  • Re: Issues with Unicode uploads
                    Dan Stevens.

                    This issue exists for us as well.  After selecting the list and setting the options, we see this:

                     

                     

                    After manually selecting the proper fields to map, the import results in new leads with no data!

                     

                     

                    This is a huge issue for us as we are a global company.  We import program members weekly (and sometimes daily) containing leads with special characters.

                      • Re: Issues with Unicode uploads
                        Dan Stevens.

                        Marketo Support came back with the following (even though I told them that Unicode text file imports have worked for years in Marketo):

                         

                        We are only supporting UTF-8, UTF-16, Shift-JIS and EUC-JP as per documentation. Could you provide a list of an import that used to work so I can compare? As a workaround, could you in excel do a save as and save it as a tab delimited format? That should save it as a UTF-8 format and you should be good to import it.

                        So I decided to try this alternate approach.  When I saved the file as "tab-delimited" (vs. Unicode Text), the data in that file contained the special characters.  So that was encouraging.  But unfortunately, Marketo replaces all of those characters with question marks after it has been imported:

                         

                         

                        I'm not quite sure why Marketo Support isn't recognizing this as a bug - that just surfaced this week with multiple customers - rather than say "we don't support the file type you're trying to import"

                        • Re: Issues with Unicode uploads
                          Dan Stevens.

                          Update from Marketo Support:

                           

                          We have found that the problem is related to changes between php 5.3 and 5.6 (which was upgraded around the same time as release). It is not able to process a special character present in an existing file.  We are currently working on building a patch for this. I will continue to update you on that as progress is made.
                          • Re: Issues with Unicode uploads
                            Gary Verster

                            Hi Said,

                             

                            Yeah, the comma-separated UTF-8 upload is always a fallback, but this could introduce extra columns into the data (which is why there is the 'Open in Wordpad/Notepad' step in the solution). The chances of getting a comma in an address field is not exactly low, so what should be a relatively clean, quick and painless upload of a tab-delimited file could easily become a much longer, painful one. And I fear for what would happen to those people who are less adept with data cleaning...

                             

                            Needs to be fixed... that's all there is to it.

                             

                            G.

                          • Re: Issues with Unicode uploads
                            Danny Tran

                            Anyone have any luck yet? Marketo Support is saying that it should work as UTF-8, but I am not getting any different results.

                              • Re: Issues with Unicode uploads
                                Dan Stevens.

                                Nothing yet.  Support is still telling me they're looking into it. We have a bunch of lists we need to upload, but can't.

                                  • Re: Issues with Unicode uploads
                                    Sanford Whiteman

                                    Hey Dan,

                                     

                                    Convert Excel's UTF16LE file to UTF8 and it'll work.

                                     

                                    To do this I used the classic iconv which you can get from LibIconv for Windows

                                     

                                    Command line:

                                     

                                    iconv -f UTF-16 -t UTF-8 "E:\My Documents\unicode_export_utf16lebom.txt">"E:\Unicode_tests\unicode_export_utf8nobom.txt"

                                     

                                    Screenshot showing success:

                                     

                                      • Re: Issues with Unicode uploads
                                        Dan Stevens.

                                        Thanks Sandy, but these are already in UTF-8 format.  Even if this workaround is a solution for whatever reason, we can't expect our marketers in every country to download this software (which will have to go through a security review) and process every list using the command line approach.  The fact is this was such an easy process for us for years - take Excel file, save as Unicode text file, import into Marketo.  Done.  And now, since the Winter release, this no longer works.

                                          • Re: Issues with Unicode uploads
                                            Sanford Whiteman

                                            Excel "Unicode Text" is UTF-16LE, not UTF-8.  UTF-16LE w/BOM (which Marketo is misleadingly calling "binary") is no longer supported, apparently, but UTF-8 w/no BOM is ok.

                                             

                                            Thought you said you had lists waiting to be imported, so this would be how to get 'em done.

                                             

                                            (FWIW you could write an Excel macro to do the conversion, too -- I just didn't feel like it as iconv was out there, and it's been a standard Linux/Mac OS tool for many years.)

                                              • Re: Issues with Unicode uploads
                                                Dan Stevens.

                                                So even when saving a Unicode text file as follows, it's still not really UTF-8?

                                                 

                                                 

                                                As far as it not being supported any more, the last I heard from support is the following:

                                                 

                                                This issue is affecting multiple customers, and Engineering is working diligently to see this resolved. I will update you further as progress is made.

                                                 

                                                So hopefully it's still supported and just considered a bug that's going to eventually be resolved.

                                                  • Re: Issues with Unicode uploads
                                                    Sanford Whiteman

                                                    That dialog only applies to saving as HTML! The TSV generated is still ISO-8859 (or maybe Windows-1252) not UTF-8.

                                                     

                                                    Note (since there seems to be confusion above and from Marketo) Excel UTF-16LE is still tab-delimited text, not "binary"... there's no magical format that avoids the need for a delimiter w/variable-length data!

                                                     

                                                    You could have UTF-16LE that's comma-delimited as well. The question of delimiter collisions (i.e. commas within data) is a good one, but quoting should automatically take care of that. Also, using tabs instead of commas just shifts the delimiter collision over to tabs, it doesn't eliminate it. You can have tab characters within a table cell/column, too.

                                        • Re: Issues with Unicode uploads
                                          Dan Stevens.

                                          On Monday (3/27), engineering pushed a patch into production to address this.  I did hear back from my team and they confirmed that the Unicode import is now working as expected.

                                          2 of 2 people found this helpful