Re: Marketo DataBase Architecture

abhishek_c
Level 2

Marketo DataBase Architecture

Hi All,
I want to get more details about 
Marketo Database Architecture as it would be more helpful for me to understand and which is the best Database to store for Marketo Data after extracting from API's as we need to perform some operation after extracting the data from Marketo.

12 REPLIES 12
SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo DataBase Architecture

The best database to use — to reuse the saying about programming languages— is the database you know.

 

Well... at least that's mostly true in this case.

 

If you have mastery of MySQL/MariaDB, or Postgres, or SQL Server, those will be OK. But you will have to tune queries for your workload (and we have no idea what that is).

 

I prefer to store Activity Logs in a timeseries database like InfluxDB, not in a standard RDBMS. Recently, for one project that concentrates on Custom Objects, I chose CouchDB, since JSON objects/documents fit the CO model. A wide and deep Marketing Activities tree structure should be stored in a hierarchical database for best performance, or use the MSSQL HIERARCHY type.

 

There isn't a single correct answer here.

abhishek_c
Level 2

Re: Marketo DataBase Architecture

I want to understand how we are storing Data in Marketo Database is it in Table column like MySql or other?

Jo_Pitts1
Level 9

Re: Marketo DataBase Architecture

@abhishek_c , to a large extent, how data is stored in Marketo is immaterial.  It is what it is what it is.

 

I think you need to define your requirements a lot more clearly. 

 

Is it to get all new/changed leads out on a regular basis, along with their activities, and all ancillary campaign details, and store these in a DB for reporting purposes? 

 

If so, pick a DB you know well, and use it.

 

If the question is 'How do I get this data out of Marketo' then it is a longer question.  Many of us have built code to do this already in a variety of languages.

 

Cheers

Jo

 

abhishek_c
Level 2

Re: Marketo DataBase Architecture

@Jo_Pitts1 ,
I'm asking it because I need to understand Marketo. As I need to Crawl/PRofile complete Marketo Data and store it in local Database. So, need to understand all dependencies and sync flow to be followed during API's calls like to get All Leads then Activities. I need to store this in MySql using Java. Can you please help me with the link for built code.

SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo DataBase Architecture


I need to store this in MySql using Java. Can you please help me with the link for built code.

There are some Java code samples here. They're quite old and can't vouch for their quality, but they'll point you in the right direction. Of course there are tons of Java HTTP libraries/abstractions to choose from, all built on the Java core networking stack. You should find the one that fits the best.

 

CData and maybe one other company offers a so-called Marketo JDBC driver, but it's never been flexible or stable enough in my experience and it's better to roll your own REST-to-relational logic.

SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo DataBase Architecture


I want to understand how we are storing Data in Marketo Database is it in Table column like MySql or other?

You asked this in a separate thread...

 

https://nation.marketo.com/t5/product-discussions/what-database-is-used-for-marketo-storage/td-p/313...

 

Again, please do us a favor by not duplicating questions in different threads. Thanks!

abhishek_c
Level 2

Re: Marketo DataBase Architecture

@SanfordWhiteman,
It's because No one is answering if I kept it in reply. So, I created a separate thread for that. Why would I ask duplicating questions if anyone had replied in same thread and even there is no option to delete that reply.

SanfordWhiteman
Level 10 - Community Moderator

Re: Marketo DataBase Architecture


It's because No one is answering if I kept it in reply. So, I created a separate thread for that. Why would I ask duplicating questions if anyone had replied in same thread

You're not giving people sufficient time to reply. Waiting only few hours before duplicating content, with a community of volunteers working in different time zones, isn't reasonable.

Jo_Pitts1
Level 9

Re: Marketo DataBase Architecture

@abhishek_c ,

I think you need to step back a little.  If replies aren't forthcoming, maybe it is the nature of the question - which you can always expand on within your original thread (and I often do when I realise my original question didn't have enough detail, was unclear, or just poorly phrased).  FWIW - I am onto my third edit of this response.

 

I still don't have a real handle on why you care about Marketo's underlying data storage.  This is what APIs are all about - they abstract the data you can access from the underlying and internal way in which it is stored (not just Marketo's APIs.. but fundamentally ALL APIs).  This in turn allows you to access the data that is relevant to you, and store it in a manner that suits you the best. 

 

For example, my team has built routines in MS Azure/C# to extract leads, activities, and all ancillary data.  We've also built similar in python.  The language is largely irrelevant - the APIs are the same.  In one case, we are dumping to CSV files as that suits the client's needs best.  In another instance, we pump the data into staging tables, and then run a full ETL process into their data warehouse.

 

The documentation on the Marketo APIs is pretty solid, so that is where I would start.  From your perspective, for all intents and purposes, this IS how Marketo stores/retrieves data.

 

The dependencies you refer to above around leads and activities - from a retrieval perspective it won't matter if you are pulling universal deltas.  From a writing to a DB perspective, that will depend on your situation and how you set up foreign key constraints.  It would be nigh on impossible for any of us to comment on that without knowing how you are planning on building your local DB.

 

Cheers

Jo