Summary
If you're using the Bulk Custom Object Import API to push records to a Marketo custom object (and you're importing separate files that have duplicates across the files) -- the records will not be deduped on the custom object if multiple imports are pushed at the same time.
Issue
If you are using a
Bulk Custom Object Import API third party service to push records to a custom object (and multiple files, containing duplicates across those files are being imported to the Marketo custom object in a similar time window) -- the records will not be deduped on the custom object, even if the dedupe field value is matched.
This is because two concurrent imports can be processed at the same time if they are pushed close together. More information can be found
here.
"A maximum of 2 jobs are processed at the same time. A maximum of 10 jobs are allowed in the queue at any given time (including the 2 currently being processed)."
Solution
In this scenario, the third-party App/API service provider that is making the bulk import calls must be consulted to ensure that separate files containing duplicate records aren't being imported to a Custom Object at the same time (or close together times).
Once the calls are spaced out adequately so there aren't concurrent imports running containing duplicate records in separate import files, the custom object will dedupe as expected based on the designated dedupe fields. More on dedupe fields can be found
here:
"Dedupe fields can be used to retrieve, update, or delete custom objects. Every custom object definition must contain at least one (and no more than three) dedupe fields."
In Summary, while Marketo does deduplicate on custom object records at import based on the dedupe field(s), it cannot dedupe when concurrent imports are being processed at the same time with duplicates in the separate files (for example: file 1 has dupe A while file 2 has dupe B).
If all the duplicates are in the same file, however, Marketo can dedupe within that file as its imported to the custom object.
Root Cause
Concurrent Custom Object imports running at the same time: when this occurs, the two files being imported concurrently don't have time to check against each other to dedupe based on the configured dedupe field.
.png)