Export data from Marketo to a data lake using the minimal amount of API Calls

[Tamir_Belzer_TB]

Hi, does anyone know if we extract data from Marketo to an external database using an API connection once or twice a day, will it reduce the number. of API calls used per day? Alternatively, we're not exporting all the tables and data from Marketo today but still use between 10 and 15K API calls per day. Is there a way we can reduce the daily usage, even if it means to send the data in bulk once or twice a day? 

Thank you. 

[Darshil_Shah1]

Yes, you can use the bulk extract API to extract data (Leads, Activities, Program Members, Custom Objects) from Marketo in bulk. Check out the documentation here: https://experienceleague.adobe.com/en/docs/marketo-developer/marketo/rest/bulk-extract/bulk-extract

Ideally, you should use bulk APIs (and not regular APIs) to import and export data in bulk for such high-performance data extraction, which is particularly useful when retrieving historical or large-scale data for analysis, reporting, or migration. Also, make sure you use the correct startAt and endAt timestamp watermarks while making these API calls so you don't end up extracting the same set of data again—unnecessarily consuming your export/import data limits.

Hope this helps. Please let us know if you have questions.

[Miray]

I agree with Darshil. The largest download typically occurs the first time you download all the data, which can use up your entire Bulk API credit for the day. However, once you have that initial download, subsequent updates should fit within your daily limit, depending on the size of your data. If you find that you need more credits, you might want to consider talking to support about increasing your Bulk API limit. You can purchase a one-time increase and use the credits to pull all the data you need.

[SanfordWhiteman]

One note about date ranges and activities: you actually can’t just extract web activities one time for a given day, or you’ll lose some data.

This is because web activities are automatically backdated when someone’s Munchkin session is promoted from anonymous to known. So a proper activity export always checks back a certain # of days. You might do this reconciliation only once per month, constraining by a Smart List. But if you don’t do it the data lake won’t match what’s in Marketo.