kdutta
Level 1
Contact Me
Online Status
Offline
Date Last Visited
Monday

Re: Building a data pipeline for Snowflake

by in Product Discussions
Saturday
Saturday
Why do you need more than 31 days in a single job?  - I needed data beyond the 31-day limit because I was trying to download huge volumes of data on Aug 30th. I did not want to miss out on any data and bulk export would require me to put in multiple requests for the activities data I was hoping to download.   Why can't you use the Bulk Extract for Leads, exactly? - I take your recommendation for the 'leads' database and the activities. And thank you so much for your response on the API / data limit. It makes sense to go with the bulk export. 👍   🎩 ... View more

Re: Building a data pipeline for Snowflake

by in Product Discussions
a week ago
a week ago
Thank you for such a quick response!!! 👍 I had to use the paginated export method because I wanted to download data beyond the 31-day limit. In a bulk export, I did not know how the output will be since each activity type ID will have a different set of 'attributes'. I will have to test the bulk extract API approach for lead activities as well. And for the leads data, I have no choice but to use the bulk export and the output is much simpler too, so its working out for me.   I am a bit confused about how marketo api measures and if byte size takes precedence over #of API calls... Thank you! ... View more

Building a data pipeline for Snowflake

by in Product Discussions
a week ago
a week ago
We are trying to build a data pipeline using Marketo REST API, AWS lambda and Snowflake. We exceeded the API limits recently (API limit was 10K / day... now it is changed to 50K), and we want to avoid such problems in the future.   I am trying to understand the API query limits better. I found this blog post  explaining the limits, but I am hoping to get some more details.  If we use the REST API with dynamic date range for lead activities data (for 10 activity type ids) with next page token, how much data(in MBs) can be consumed for a date range of one week and how many API calls could one request / response consume? Up to 200k+ / month of lead activity data (approximately) How many api requests would we end up consuming if we download all leads from the marketo leads database? oldest record is from September of 2019 Bulk export for a 31-day period would result in 3k - 5k records (approximately) Thank you in advance! ... View more
Latest Tags
No tags yet
Public Statistics
Date Registered a week ago
Date Last Visited Monday
Total Messages Posted 3
Likes given to
Home