Replicate Marketo Database

[Anonymous]

Hi,

I need to build an analytics application in top of the data available in Marketo.

For that I want to replicate the Marketo data into a local database (Sql Server/Oracle, etc.) so I can use that info in a simpler way.

Any suggestions on how to accomplish this.

I was planning in using the REST API for getting all the new activities and activity attributes and I can't foreseen an issue there, however when getting the lead data, getting the updated leads will not do the job I think because I'll still keep delete/merged leads.

Any suggestion on how to overcome this or on how to implement a different replication mechanism.

Regards.

[Anonymous]

Hi Carlos,

We have bumped into the same issue as you had before while we are trying to replicate data to SQL server database to build some analytic reports, but it turned out to be hard to achieve with Marketo standard  API.

Therefore, we have developed our own data retrieving tool called Data Loader for Marketo(DLM) which allows you to sync Marketo data to the database without any coding. We invite you to visit our website to have a free trial of DLM and see if this could solve your problem.

Data Loader for Marketo:  https://dlm.trend.org

Feel free to contact us for further information or any feedback are very welcome!

Sincerely,

Vivian

[SanfordWhiteman]

The product appears to require old Marketo SOAP credentials -- i.e. full create/read/update/delete access to Marketo -- in addition to REST API access. Is that correct?

[Anonymous]

Hi Sanford,

We use both SOAP and REST APIs in our product, but if you have any concern about SOAP credentials, welcome to contact us to try REST-only version.

The reason why we use both SOAP and REST APIs is that there is some limitation in REST API Marketo provided.

First, Marketo only supports Opportunity REST API for those who use Marketo without integration with CRM tool. Therefore, we won’t be able to include Opportunity REST in our product since we use Marketo and Salesforce. Second, although Program can be retrieved with REST API, it might easily exceed API limitation since it returns only 1 record in 1 API call.

Per the limitation of REST API-only might have, we will still provide current version which has the full function (SOAP+REST). In the same time, REST API-only version will also be provided but with limited functions.

Btw, although SOAP API has the delete function, we didn't use it at all. You're welcome to give it a test.

Thank you again for your suggestions, all the opinions are valuable for other people to have a better product to use.

Best Regards,

Vivian

[SanfordWhiteman]

I've since been in contact with your developer and outlined the security problems with the SOAP approach, which he was not aware of. Granting read, (over)write, and delete access to every object in a Marketo instance -- especially, but not only, when software is in beta -- isn't something a security-conscious organization should authorize. It's not a matter of your code mailiciously taking advantage of its root access to an instance, but the undeniable risk involved. One misstep in parsing the WSDL and everything is gone.

I also noticed that the software does not enforce a cap on its daily API calls, so simply testing the software can break all other running integrations. And, as an experienced Marketo dev I feel you're not accurately describing the technical problems with the.REST API, though indded there are are many.

Nevertheless, I'm looking forward to testing the REST-only version, which I downloaded the other day as soon as it was published!

[Anonymous]

Thanks for your interest in our product and advice. We know you might concern for SOAP security, so we investigated this issue and released the REST-only version.

Regarding to API calls, we will allow users to view current API usage in new version, which plans to release in late October. Besides, customized daily API call limitation is on our roadmap. We are working on many new features based on users feedback.

Thank you again for downloading the free trial, any feedback is welcome!

Best Regards,

Vivian

[Grégoire_Miche2]

Hi Carlos,

Mind the REST API limitations. If your database is big, you may exhaust them pretty quickly. From the doc:

Latest API Version: v1

Default Daily API Quota: 10,000 API calls (counter resets daily at 12:00 AM CST)

For higher volume integration scenarios, please contact your account representative to increase your daily API quota.  The quota applies to SOAP and REST API calls only.

Rate Limit: 100 API calls in a 20 second window

-Greg

[Anonymous]

Yes I know.

I've a 10.000 quota daily. I'dd say that for now, based on the data I have, those would be enough to cope with that.

I was wondering if there was a better way of doing this.

To me is kind of odd to find out that I have to go through all this trouble with REST API's to be able to extract analytics from Marketo.

[Grégoire_Miche2 · Accepted solution]

Hi Carlos,

The API is really needed to extract activities. But for the lead themselves, you may just use the data export feature.

-Greg

[Anonymous]

First of all thank you for your responses.

Data export? That would create a manual process or are we talking about something that can be automated?

I want be able to update lead changes every day.

[Grégoire_Miche2]

Hi again Carlos,

Unfortunately, data export can only be manual

-Greg

[Anonymous]

Bottom line.

If I want an automatic system that each day it updates with the changed data, APIs are my only option?

That's really disappointing to be honest.

Regards.

[Grégoire_Miche2]

Hi Carlos,

Yes. I do not understand why you feel it's disapointing. Any other system would do the same.

Now, if what bothers you is the development, you can use a data movement tool such as bedrock data (Bi-directional Marketo Integrations – Marketo LaunchPoint )

-Greg

[Anonymous]

The disappointment comes both from effort I've to put to get data on my side in a easy way to access (SQL) so I can build my analytics application and also because the alternative is based on API calls in which I have a quota.

Still, thank you very much for you help. I'll take a look into the tool.

Regards.

[SanfordWhiteman]

If you were expecting old-school flat-file exports, I understand that the JSON-based API might seem complex.  But like Greg said, the Marketo API style is pretty normal.  I'm surprised when people still offer CSVs (for my own apps, I refuse to provide them anymore, since even if data can be flattened now it could well become too complex later).

As for the API quota, it's definitely frustrating (even much more so if you have multiple apps sucking up calls with no individual caps) and I warn people off using the API in any singleton context.  But you're working firmly in the bulk context and, assuming yours is the only app using the API, you do get to download 3,000,0000 somethings per day.  That's a lot of data to take in, period, and I wouldn't be surprised if your infrastructure + ETL overhead + API endpoint performance + network overhead end up being the operative bottlenecks, not the API call limit.

[Anonymous]

There is an API call for get deleted leads as well.  Not sure why you would want to keep merged leads as by definition they are the same lead so I would think you would want the data to be merged from reporting purposes as well

[Anonymous]

I don't want to keep them. That's why I need to get the delete leads so I can act accordingly in my local replica.

Thanks for the response. In that point that's exactly what I needed. Not quite sure how I missed that.