SyncLeads sometimes doesn't send back LeadId

[Yann_Lair]

Hi,

We are using the REST api in our instance to sync lead with their munchkin ID. We have a problem with the "SyncLeads" endpoint. When we are creating leads, for unknown reason, the request comes back "Successful" but the leadId for that user has no value, which causes us problems because we deal with that situation as an error when actually the lead is still being created in the background! Now when we tried to update that lead based on the mail for other purposes, the same email comes back duplcated:  One with the munchkin id and one with without. 

So I actually have 2 questions regarding this situation:

- Why does syncLeads doesn't give us back the lead id everytime, even while it seems to be creating it? I expect an API call such as this one to be idempotent. 

-  How can we avoid to have duplicates with the same email? 

[SanfordWhiteman]

- Why does syncLeads doesn't give us back the lead id everytime, even while it seems to be creating it? I expect an API call such as this one to be idempotent.

Well... technically idempotence has nothing to do with the response (a GET is idempotent even if the response varies, as long as the server state is not changed by repeated requests). 

An HTTP request is still idempotent even if the request is carried out asynchronously (that's one of the benefits of idempotence!). It's like if you throw a bunch of identical database updates into a queue: they're going to leave the database in the same state, regardless of the order in which they execute and/or whether you know when they're done executing.

I agree that this is frustrating. But there isn't a way around it unless you look up the ID by email first and then associate with the ID. (A race condition still exists, but you can in theory build locking into your client app to work w/that.)

-  How can we avoid to have duplicates with the same email? 

Marketo doesn't enforce email as a primary or unique key. So it's up to you, as the API consumer, to make sure you use email as the lookup key everywhere you can (but there are still areas you do not control such as CRM sync).

[Yann_Lair]

Thank you for your quick answer. 

I understand your point about the request being idempotent, I might have not express myself clearly. What I was meaning to say is that I would expect the same call in the same situation to give me the same answer. In this situation, you make a first call, its returns nothing, but if I do the same call 5 minutes after, it gives me a different result.

For 2 different leads, one will return the created object, the other doesn't. For the same lead, the first one returns me an empty object, the second returns me the created object.

It's weird to receive a "Success" with the returned object not "fully" created and nothing else. I would expect to get at least a warning message saying "Object not yet created, retry" etc.

I still have some questions:

- Could you please indicates what's happening on Marketo side that explains sometimes we don't receive the created object? 

- When I receive a success with an empty object, is it correct to assess that the item hasn't been fully created yet but not error so far?

- What's the timeout for syncLeads?

- Should we put in place a retry mechanism to wait until we get the actual object?

Thank you  very much for your time.

[SanfordWhiteman · Accepted solution]

I understand your point about the request being idempotent, I might have not express myself clearly. What I was meaning to say is that I would expect the same call in the same situation to give me the same answer. In this situation, you make a first call, its returns nothing, but if I do the same call 5 minutes after, it gives me a different result.

Ah, but it's not actually the same situation. Because the back end is being updated asynchronously.

This the way Munchkin association works from the browser, too. When an association is in progress, you won't see (for example) lead tokens filled in. But if you were to request the same LP a mere 1 millisecond later, you might see the tokens! And from that point forward the requests would have the same result. That's the thing about a distributed system.

- Could you please indicates what's happening on Marketo side that explains sometimes we don't receive the created object?

Asychronous/queued db update.

- When I receive a success with an empty object, is it correct to assess that the item hasn't been fully created yet but not error so far?

Yes.

- What's the timeout for syncLeads?

No idea!

 Thank you  very much for your time.

Sure, you could start polling for the ID (looking up via email) in the event that you get an uncertain result.

We do the same kind of thing on the client side, polling an endpoint for up to 10s to see if an association is complete.

[Yann_Lair]

Thank you for your time, it is much appreciated. I understand better the constraints linked to the marketo architecture and how to handle it properly forward. 

Thanks again for the quick answer!

[SanfordWhiteman]

Great, maybe mark one of my above answers as Correct?

[Yann_Lair]

Done!

Is this async architecture mentioned in the API doco? I've read it and I can't remember seeing this. If I missed it, other will too. Can you point where we can read more about that in the doco? That'll probably help future people to get that info. 

Thanks mate.

[SanfordWhiteman]

It's not in the API docs, but rather something you learn through lots and lots of testing.

My first experience w/Marketo was reverse-engineering aspects of Munchkin so it's pretty top-of-mind for me.

[Yann_Lair]

That feels like that it should be written down in there doc! I would have saves days of work trying to figure out what was wrong on our side when it was just the marketo architecture working as intended but without anybody knowing. I'm pretty  sure I won't be the only one bamboozled by the successful request with expected object not returned. 

Any idea who we can contact to pass down that feedback?