Duplicate Leads are got when separating a large time span to several smaller ones to multi-thread data extracts

[Anonymous]

Hi all,

When I get data by LeadUpdateAtSelecter on a long duration, to get records batch by batch would take too long.
So I separate the duration to five smaller ones, and do the extract actions using multi-thread.
After getting all data, I find that, in the five parts, there are several Leads appear in not only one part. So when joining them together, I get several duplicate records.
Any suggestions about this?

Thanks,
Biao

[Anonymous]

Have you checked to see if you have duplicates of those leads in your Marketo db to ensure it is a reporting error? 

[Anonymous]

Hi Jeffrey,

I don't think setting a new value to one Attribute of a Lead record would cause duplicates in db.

In my test, I changed the Email Address values of some Leads including whose ID is 20 from 2014-02-18T03:26:00Z to 2014-02-18T03:36:00Z. And then around 2014-02-18T04:30:00Z, I changed the Lead Status values of some Leads also including whose ID is 20.
I tried to get the updated Leads hour by hour. I find that ID 20 is in time span 2014-02-18T03:00:00Z ~ 2014-02-18T04:00:00Z (as TimeSpan1), and also in time span 2014-02-18T04:00:00Z ~ 2014-02-18T05:00:00Z (as TimeSpan2), and the Attribute values of the two downloaded records are all the same. And when I get the Leads updated in the whole day 02-18, ID 20 appears only once.

In my mind, I have done a new updating after 04:00 and the UpdateAt timestamp has changed too, then I would not get the record when I querying data updated before 04:00.
Any suggestions?

Thanks,
Biao