Duplicate Leads are got when separating a large time span to several smaller ones to multi-thread data extracts
When I get data by LeadUpdateAtSelecter on a long duration, to get records batch by batch would take too long. So I separate the duration to five smaller ones, and do the extract actions using multi-thread. After getting all data, I find that, in the five parts, there are several Leads appear in not only one part. So when joining them together, I get several duplicate records. Any suggestions about this?
Re: Duplicate Leads are got when separating a large time span to several smaller ones to multi-thread data extracts
I don't think setting a new value to one Attribute of a Lead record would cause duplicates in db.
In my test, I changed the Email Address values of some Leads including whose ID is 20 from 2014-02-18T03:26:00Z to 2014-02-18T03:36:00Z. And then around 2014-02-18T04:30:00Z, I changed the Lead Status values of some Leads also including whose ID is 20. I tried to get the updated Leads hour by hour. I find that ID 20 is in time span 2014-02-18T03:00:00Z ~ 2014-02-18T04:00:00Z (as TimeSpan1), and also in time span 2014-02-18T04:00:00Z ~ 2014-02-18T05:00:00Z (as TimeSpan2), and the Attribute values of the two downloaded records are all the same. And when I get the Leads updated in the whole day 02-18, ID 20 appears only once.
In my mind, I have done a new updating after 04:00 and the UpdateAt timestamp has changed too, then I would not get the record when I querying data updated before 04:00. Any suggestions?