Use of Bulk API Export vs REST API

[Jess_Ramos]

Hello! 

We are currently using the BULK API export to fetch update lead data for our Tableau dashboard.  We have ~600K lead records and 300 fields. 

We are using the python library 'marketorestpython' to make calls to the marketo API.  The python script fetches all the leads starting from June 2017 up until now and gets all the fields that are present in Marketo.

The script runs on weekends and we get the 500 mb limit exceeded error when the script has fetched all the leads ranging from July 2017 to about August 2020. We want to capture any changes and deleted records in this refresh. Not all fields or leads will have updates though. 

Marketo support has directed us to use the REST API instead of the BULK API for this process. Our developer is concerned that doing so will be too slow and create delays in refreshing our lead data in Tableau.  Does anyone have an opinion or similar experience that could help guide us in our best path forward here?  

Thank you!!!

[SanfordWhiteman]

If this is a one-time backfill — not representative of your ongoing weekly volume — why not continue to use Bulk Extract, but do it in parts?

You can also purchase more daily BE transfer volume, even on a temporary basis. That'll be more efficient than rewriting to use Bulk Export.

[Jess_Ramos]

Thanks for the reply Sanford.  It's not a one time back-fill. It's an ongoing weekly refresh we do of our lead data table for our Tableau Dashboard. We are consistently hitting the bulk API limit and Marketo recommends we use REST API for these ongoing updates -- we are told by our developers it might be a very long update process. I'm not sure if that is a valid concern and not getting a clear answer from support. 

[SanfordWhiteman]

Why do you have to go back to 2017 every week? That doesn't make sense.

[Jess_Ramos]

We are checking all the leads every week because any of them can be updated (worked) regardless of when they were created (2017 or 2020)... we are checking for updates to them each week. 

[Katja_Keesom]

The API does have capabilities to only pull records that have changed since your last check. I would suggest to look into these, as that would save you a whole lot of data processing every week.

[SanfordWhiteman]

Unfortunately, the updatedAt filter isn't available on all subscriptions.

[Jess_Ramos]

Thanks Sanford! Any other thoughts for us? I feel stuck... 

[SanfordWhiteman]

Did you try using that filter in your instance?

[Jess_Ramos]

Thank you Katja! We will inquire if we have this filter.