Import list issues with LARGE file

[Anonymous]

I have a 378 MB file of approximately 1.8 million leads with about 20 rows of data per lead.

Started the process of importing the list -- browsed for file, selected file format, trusted source, default import mode, click Next. The window for Import List has now been spinning for almost 2 hours, and I'm not even to the point of seeing the fields to match them. How long should this take to show up for a file of this size?

(The "Uploading 0%..." to 100% progress did complete in around 5+ minutes, but it was stuck spinning after that.)

And once I complete the matching of fields, how long will the actual import take?

[Edward_Unthank_]

That is going to be an absolute monster of a list upload. I would recommend breaking into smaller lists of around 100k. The problem with doing a single list of 1.8m is that you won't know until 24 hours later whether it worked or not.

I've done a ~1m list upload and the spinning wheel doesn't differentiate between "it's working—slowly" and "something is wrong," so breaking it down at least provided some sense of a progress slider bar.

Also, are you uploading net-new leads, or updating existing leads? If you're updating leads, trim your CSV to only the absolutely-necessary fields. Email address and then any updated fields. The amount of processing needed is (oversimplified) # rows * # columns, so if you're talking 1.8 rows, every row counts.

(Another option is uploading to SFDC and having it sync over to Marketo from there, if you keep running into problems trying to upload that volume of leads to Marketo directly.)

Edward Unthank | Founder, Etumos

[Anonymous]

Yes, it is a monster, and yes, all new leads -- a data dump from previous email platform into Marketo.

Unfortunately, we have no way of breaking it down into smaller chunks, as it is too large of a file to open in Excel. (I'm sure there are other ways, but no one internally is aware/has experience.)

Could you elaborate on the SFDC method? Not familiar.

[Edward_Unthank_]

The SFDC sync is assuming that you're going to be doing this as a one-to-one sync between records in Marketo and records in SFDC anyway, which is probably a big assumption in this case.

I haven't done it myself, but I know people who swear by DemandTools as a great tool for SFDC, a feature of which (MassEffect, I think?) can do mass upload of data. Then you'll just be waiting for the Marketo/SFDC sync to bring the leads back over from SFDC. When you're doing this much volume of a sync, you'll want to call up Salesforce support and have them temporarily (for one week) boost the number of API calls you get to make this happen. They'll do that free of charge when it's a mass data upload, in my experience. You'll still be waiting about a week for that number of leads to sync between Marketo and SFDC.

Here's a good article on some common gotchas with data upload in SFDC.

If that doesn't work, I would personally (1) look into ways to split that CSV into reasonable chunks and (2) call up Marketo Support and see if they can assist you with the massive upload, like if you provide the CSV to them and they might have a back-end tool to upload.

Edward

[Josh_Hill13]

I agree with Ed's approach.,...but I might do this instead:

For this situation, I would call Support and ask them to:

• see if they will upload on their side
• see if they can temporarily open up the max # of SFDC sync channels for you to speed it up.
• check if you have the right # of SFDC API calls per day. You can do this in Admin on SFDC.

[Anonymous]

Hi Matt

I agree with Josh and Ed. It was nightmare for my user. We had similar scenario which we were seeing very frequently. Huge lists and monthly uploads. I have come with Java program, which split files into multiple chunks and then web services API call import the leads. This complete process runs in loop to do automated import. This reduced huge amount of time.

If this is for just one time, you can manually do this instead going with programming approach.

[Anonymous · Accepted solution]

Solution we arrived at was the locate a file splitting program (csvsplitter) and divide the 1.8m into 18 chunks of 100K, which took some time but did import properly.

I did have to keep track of each import for errors and warning that resulted, download each of those files from Marketo for each import, and then scrub that data manually for the issues.

All in all, it probably amounted to a day's worth of work.