10 Replies Latest reply on Sep 24, 2017 1:02 PM by Ed King

    Marketo Replication to Data Warehouse

    Ashish Yadav

      Hi All,

       

      We are trying to setup replication of Marketo objects to our data warehouse. Is there an out of box solution that can help us achieve this is an automated manner. Thank you!

        • Re: Marketo Replication to Data Warehouse
          Josh Hill

          did you check launchpoint?

           

          What tools and db do you have?

           

          What's your MDW going to do for you?

           

          From what I've seen, it is all custom. You select a DB and setup API connections to pull the data you want in the way you want. You can use SOAP, REST, or Bulk API with Marketo. there are limits and hiccups, so you should get an experienced developer to help.

          • Re: Marketo Replication to Data Warehouse
            Sanford Whiteman

            + 1 to Josh.

             

            Ashish, this is not something that can be asked and answered in one line: you haven't even told us anything about the platform you're using for your data warehouse.

             

            You can connect to any database, using either commercial connectors built into integration platforms (Boomi, Talend, CData/SSIS) or custom connectors that use Marketo the REST API.

             

            There will be shortcomings and surprises (but, hopefully, overall success) with any approach. But it would be reckless for anyone to tell you there's an out-of-box solution when we don't know the actual "problem."

              • Re: Marketo Replication to Data Warehouse
                Ashish Yadav

                Thank you for your reponse Sanford & Josh. I should have provided a brief summary of my use case. We are moving from an old instance of Marketo to a new instance and want to archive Activity data from the old instance to a DW. We are also trying to setup a DW to replicate data from the new instance to facilitate custom reporting for Marketo. DW is Postgress DB.

                  • Re: Marketo Replication to Data Warehouse
                    Josh Hill

                    Bulk API somehow.

                      • Re: Marketo Replication to Data Warehouse
                        k winters

                        Josh:

                        We've integrated the Marketo data into our application, which is basically a data warehouse, using the Marketo API and Talend.  There are some workflows that are a little tricky, but, it was not that difficult.  We did not use the Bulk API, but, just created processes to pull and push data.

                         

                        Thanks,

                        David

                      • Re: Marketo Replication to Data Warehouse
                        Sanford Whiteman

                        If it's a one-time occurrence, use the Bulk Activity Extract API to get a CSV (which is actually nested text, JSON, and serialized PHP data, but it's all there), unwrap the nested fields, and import into your Postgres table(s).

                         

                        Obvs. there's no single, universal mapping schema, because it depends on how you want to store the data for reporting purposes.

                         

                        You could also just import the CSV as-is to a single table for now, then unwrap it into other tables later when you decide on your schema.

                         

                        For periodic -- but not daily -- replication, say at the end of each of month, you can use the same Extract API as it's extremely efficient with large amounts of data. On the other hand, if you need daily imports it may make more sense to use the Activities API as that's more "interactive" (synchronous) and supported by more ETL tools.

                        • Re: Marketo Replication to Data Warehouse
                          Jep Castelein

                          Not sure if you've already engaged with Marketo Professional Services, but we are able to extract activities from your old instance and push them into your new Marketo instance as custom activities. If desired, we can also provide you with a backup of the data. By default, we use the SQLite database format. You can then write a script that takes the data out of SQLite and pushes it into PostgreSQL.

                           

                          In addition to migrating person/activity data, we can also restore List Membership, Program Membership and - if desired - migrate assets like Snippets and Images/Files.

                      • Re: Marketo Replication to Data Warehouse
                        Ashish Yadav

                        Thank you for your response everyone.

                         

                        Jep Castelein we are working with the professional service team but this is not part of the scope right now. We do have Talend and are looking to utilize that as an integration and automation platform. Will update this post one we finalize the solution.

                        • Re: Marketo Replication to Data Warehouse
                          Neil Robertson-Ravo

                          We use this : SSIS Integration Toolkit for Marketo   It is one of the gave seen/used.  We looked at Cdata as well as rolling our own via Talend but the performance of the Kingswaysoft connector as well as their customer service has been brilliant. We asked for several features to be built into the product (such as a limit on the number of API calls made) and within a week or so we had it!

                           

                          if you are not restricted by API call limits or have a large set - then just set it up, and deploy to an SSIS catalog.

                          • Re: Marketo Replication to Data Warehouse
                            Ed King

                            Ashish,

                             

                            Our product Openprise can do this out-of-the-box without you writing code or use a system integrator, but we need to find out more about the exact data you want to archive in addition to activities and where is your Posgress located, on-prem or in the Cloud. Let me know if you wish to discuss and we can take it off line. You can learn more about our product and company at www.openprisetech.com

                             

                            Ed