Re: Marketo Replication to Data Warehouse

Anonymous
Not applicable

Marketo Replication to Data Warehouse

Hi All,

We are trying to setup replication of Marketo objects to our data warehouse. Is there an out of box solution that can help us achieve this is an automated manner. Thank you!

10 REPLIES 10
Josh_Hill13
Level 10 - Champion Alumni

Re: Marketo Replication to Data Warehouse

did you check launchpoint?

What tools and db do you have?

What's your MDW going to do for you?

From what I've seen, it is all custom. You select a DB and setup API connections to pull the data you want in the way you want. You can use SOAP, REST, or Bulk API with Marketo. there are limits and hiccups, so you should get an experienced developer to help.

SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo Replication to Data Warehouse

+ 1 to Josh.

Ashish, this is not something that can be asked and answered in one line: you haven't even told us anything about the platform you're using for your data warehouse.

You can connect to any database, using either commercial connectors built into integration platforms (Boomi, Talend, CData/SSIS) or custom connectors that use Marketo the REST API.

There will be shortcomings and surprises (but, hopefully, overall success) with any approach. But it would be reckless for anyone to tell you there's an out-of-box solution when we don't know the actual "problem."

Anonymous
Not applicable

Re: Marketo Replication to Data Warehouse

Thank you for your reponse Sanford & Josh. I should have provided a brief summary of my use case. We are moving from an old instance of Marketo to a new instance and want to archive Activity data from the old instance to a DW. We are also trying to setup a DW to replicate data from the new instance to facilitate custom reporting for Marketo. DW is Postgress DB.

Josh_Hill13
Level 10 - Champion Alumni

Re: Marketo Replication to Data Warehouse

Bulk API somehow.

Anonymous
Not applicable

Re: Marketo Replication to Data Warehouse

Josh:

We've integrated the Marketo data into our application, which is basically a data warehouse, using the Marketo API and Talend.  There are some workflows that are a little tricky, but, it was not that difficult.  We did not use the Bulk API, but, just created processes to pull and push data.

Thanks,

David

SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo Replication to Data Warehouse

If it's a one-time occurrence, use the Bulk Activity Extract API to get a CSV (which is actually nested text, JSON, and serialized PHP data, but it's all there), unwrap the nested fields, and import into your Postgres table(s).

Obvs. there's no single, universal mapping schema, because it depends on how you want to store the data for reporting purposes.

You could also just import the CSV as-is to a single table for now, then unwrap it into other tables later when you decide on your schema.

For periodic -- but not daily -- replication, say at the end of each of month, you can use the same Extract API as it's extremely efficient with large amounts of data. On the other hand, if you need daily imports it may make more sense to use the Activities API as that's more "interactive" (synchronous) and supported by more ETL tools.

Jep_Castelein2
Level 10

Re: Marketo Replication to Data Warehouse

Not sure if you've already engaged with Marketo Professional Services, but we are able to extract activities from your old instance and push them into your new Marketo instance as custom activities. If desired, we can also provide you with a backup of the data. By default, we use the SQLite database format. You can then write a script that takes the data out of SQLite and pushes it into PostgreSQL.

In addition to migrating person/activity data, we can also restore List Membership, Program Membership and - if desired - migrate assets like Snippets and Images/Files.

Anonymous
Not applicable

Re: Marketo Replication to Data Warehouse

Thank you for your response everyone.

Jep Castelein​ we are working with the professional service team but this is not part of the scope right now. We do have Talend and are looking to utilize that as an integration and automation platform. Will update this post one we finalize the solution.

Neil_Robertson6
Level 5

Re: Marketo Replication to Data Warehouse

We use this : SSIS Integration Toolkit for Marketo   It is one of the gave seen/used.  We looked at Cdata as well as rolling our own via Talend but the performance of the Kingswaysoft connector as well as their customer service has been brilliant. We asked for several features to be built into the product (such as a limit on the number of API calls made) and within a week or so we had it!

if you are not restricted by API call limits or have a large set - then just set it up, and deploy to an SSIS catalog.