Marketo Replication to Data Warehouse

[Anonymous]

Hi All,

We are trying to setup replication of Marketo objects to our data warehouse. Is there an out of box solution that can help us achieve this is an automated manner. Thank you!

[Josh_Hill13]

did you check launchpoint?

What tools and db do you have?

What's your MDW going to do for you?

From what I've seen, it is all custom. You select a DB and setup API connections to pull the data you want in the way you want. You can use SOAP, REST, or Bulk API with Marketo. there are limits and hiccups, so you should get an experienced developer to help.

[SanfordWhiteman]

+ 1 to Josh.

Ashish, this is not something that can be asked and answered in one line: you haven't even told us anything about the platform you're using for your data warehouse.

You can connect to any database, using either commercial connectors built into integration platforms (Boomi, Talend, CData/SSIS) or custom connectors that use Marketo the REST API.

There will be shortcomings and surprises (but, hopefully, overall success) with any approach. But it would be reckless for anyone to tell you there's an out-of-box solution when we don't know the actual "problem."

[Anonymous]

Thank you for your reponse Sanford & Josh. I should have provided a brief summary of my use case. We are moving from an old instance of Marketo to a new instance and want to archive Activity data from the old instance to a DW. We are also trying to setup a DW to replicate data from the new instance to facilitate custom reporting for Marketo. DW is Postgress DB.

[Josh_Hill13]

Bulk API somehow.

[Anonymous]

Josh:

We've integrated the Marketo data into our application, which is basically a data warehouse, using the Marketo API and Talend.  There are some workflows that are a little tricky, but, it was not that difficult.  We did not use the Bulk API, but, just created processes to pull and push data.

Thanks,

David

[SanfordWhiteman]

If it's a one-time occurrence, use the Bulk Activity Extract API to get a CSV (which is actually nested text, JSON, and serialized PHP data, but it's all there), unwrap the nested fields, and import into your Postgres table(s).

Obvs. there's no single, universal mapping schema, because it depends on how you want to store the data for reporting purposes.

You could also just import the CSV as-is to a single table for now, then unwrap it into other tables later when you decide on your schema.

For periodic -- but not daily -- replication, say at the end of each of month, you can use the same Extract API as it's extremely efficient with large amounts of data. On the other hand, if you need daily imports it may make more sense to use the Activities API as that's more "interactive" (synchronous) and supported by more ETL tools.

[Jep_Castelein2]

Not sure if you've already engaged with Marketo Professional Services, but we are able to extract activities from your old instance and push them into your new Marketo instance as custom activities. If desired, we can also provide you with a backup of the data. By default, we use the SQLite database format. You can then write a script that takes the data out of SQLite and pushes it into PostgreSQL.

In addition to migrating person/activity data, we can also restore List Membership, Program Membership and - if desired - migrate assets like Snippets and Images/Files.

[Anonymous]

Thank you for your response everyone.

Jep Castelein​ we are working with the professional service team but this is not part of the scope right now. We do have Talend and are looking to utilize that as an integration and automation platform. Will update this post one we finalize the solution.

[Neil_Robertson6]

We use this : SSIS Integration Toolkit for Marketo   It is one of the gave seen/used.  We looked at Cdata as well as rolling our own via Talend but the performance of the Kingswaysoft connector as well as their customer service has been brilliant. We asked for several features to be built into the product (such as a limit on the number of API calls made) and within a week or so we had it!

if you are not restricted by API call limits or have a large set - then just set it up, and deploy to an SSIS catalog.