Self Service Flow Steps Guide

Tyron_Pretorius

Self Service flow steps (SSFS) are a powerful way to integrate with third party applications and to build custom workflows that can't be achieved in Marketo alone. This blog post will give you a Replit repository that you can clone to get started and it will walk you through the code in the repository so that you can make it your own and bring SSFS into your Marketo instance.

Ok so we all know what a SSFS looks like inside of Marketo but what happens when the SSFS is executed in a smart campaign flow and the data is sent outside of Marketo? There is a server set up to catch all this information, process it, and then send the information back to Marketo to update the lead record.

Don't be intimidated by the word "server", this is just somewhere we have hosted our SSFS code so that it is constantly listening out for information sent from Marketo, ready to process it and send it back. This hosted code for the SSFS can be referred to as a "service".

I have seen a lot of documentation and posts about what a SSFS looks like inside of Marketo but I haven't yet seen anyone show what this looks like on a server. That's why I'm here to show you what's under the hood so that once I'm finished you can take the wheel and drive your Lambo off into the sunset!

Walkthrough Video

I recommend checking out the YouTube video below where I can go into much more detail than here in the blog. The aim of this blog and the associated walkthrough video is to:

Give you a Python template in Replit that you can clone to get started
Walk you through the code structure necessary to service multiple SSFS
Show you how you can deploy your code in Replit and install the SSFS in Marketo

Self Service Flow Steps Vs Webhooks

Josh Arrington does a great job at highlighting the advantages of using SSFS over Webhooks. Since the purpose of my blog is to walk through the technical build out of self service flow steps I will just summarize the advantages that Josh mentions:

Customization: The ability to have multiple fields in the configuration of a SSFS within a smart campaign flow (see images of SSFS below) allows a single SSFS to be used for different scenarios, which is much more scalable than webhooks which are limited to using tokens for dynamic input.
Ease of Use: SSFS are set up by Admins using a a graphical interface and users can modify what is sent to a SSFS in the flow of a smart campaign, which is much more user friendly and less prone to errors than having users or admins editing the JSON payload of a webhook.
Batch Operations: SSFS can be used in trigger, batch, and executable campaigns whereas webhooks can only be used from trigger campaigns. Addditionally, when a SSFS is used in a batch campaign it can send up to 1000 leads at once to the server which makes it much quicker to process large amounts of leads at once.
Reliability: The fact that the SSFS code on the server can execute in it's own time and then send the information back to Marketo when it is ready means that SSFS are more robust than webhooks and won't be subject to the 30 second timeout limitation. As mentioned in the FAQs, the smart campaign flow will wait for the SSFS to complete before progressing to the next flow step.
Data: Lead, program, smart campaign, program member, and subscription data can all be sent to SSFS and any updates to the lead will be shown in the activity log. The custom trigger and filter associated with the SSFS will allow you to trigger or run batch workflows based on this activity or create smart lists of people who have been processed by the SSFS.

Hosting Your Self Service Flow Step Code

I really like using Replit to host my code because it is relatively cheap and it makes it easy to deploy your code to production and handle scaling your capacity to handle increased volume from Marketo. To see how easy it is to work in Replit and deploy your service check out the video above.

At the moment the Replit Core package that allows you to deploy your service and make your service private costs $20 per month. Then all the computation your service uses when your code executes is billed monthly.

To give you an idea of how much usage will cost, it cost my company less than $1 to run 2 SSFS for 2 weeks so Replit is a cost effective solution for hosting your SSFS service.

Self Service Flow Step Code Overview

When I first opened up the self service flow step Github repo provided by Marketo and started scrolling through the ReadMe file, I didn't have the slightest clue about where to begin. I figured that I would need to host code externally which Marketo could interact with but after this I was lost.

In stepped my buddy ChatGPT. I used the O3 reasoning model and pasted in the link to the Github repo and explained that I wanted to build a service in Replit to host a Marketo SSFS. Then after much back and forth with ChatGPT and lots of experimenting and testing in my Marketo instance I finally was able to build a Replit repository to support multiple SSFS.

Now I want to save you all the blood, sweat, and tears that I had to endure by sharing a Replit template that you should be able to clone as a starting point.

I personally found the ReadMe file to be hard to navigate easily since it is missing hierarchy so I copied all this information to a Google doc to make it easier to navigate. So if you feel the same way then feel free to use this doc.

The Replit template contains code for the 3 self service flow steps described below.

GPT Completion Self Service Flow Step

This self service flow step allows a user to call the OpenAI chat endpoint. All that is needed to get this SSFS working for your instance is to put your OpenAI API key into the secrets in Replit. You can follow this link to get an API key.

SSFS to create a GPT chat

Send SMS Self Service Flow Step

This self service flow step using the Telnyx SMS API to send an SMS message.

Follow these docs to sign up for the Telnyx portal, buy an SMS capable phone number, and assign this number to a messaging profile.
The code in the Repl is already configured to send SMS but if you want to learn more about the Telnyx SMS API then you can check out these docs.
Once you are logged into your account you can follow this link to get your Telnyx API key and then all you need to do to get this SSFS working is to add the API key to the secrets in Replit.

SSFS to send an SMS

Calculate Formula Self Service Flow Step

This self service flow step allows you to compute Excel formulas using the formulas Python library. If you want to see all the functions that are available then you can check out the docs here.

SSFS to calculate an Excel formula

Known Bugs with Self Service Flow Steps

Josh Arrington and I have collected our notes on the bugs we have noticed with SSFS and Josh has also put together a wishlist of features he would like to see. So when you are testing out the SSFS provided in the template and things are not working as expected give this document a look first because these could be bugs of SSFS overall and have nothing to do with the code in Replit.

Replit Template Structure

Replit code repository

The structure of the Replit template consists of:

A "main.py" which is a simple script that is just responsible for forwarding the incoming SSFS information to the correct "routes.py" file.
Folders for each SSFS containing a "routes.py" file and a "swagger.json" file
- I'll go more into the "routes.py" file structure down below but it houses all the endpoints Marketo needs to interact with and it contains the code needed to do the processing of the information sent from Marketo and to send information back to Marketo.
- The "swagger.json" file contains a list of every endpoint available on the service, the data and data types expected by each endpoint, and the shape of what the service will send back. This allows Marketo to know how to talk to the service.
A "workflowpro_icon.png" file which is referenced by the "/brandIcon" and "/serviceIcon" endpoints meaning this is the logo you will see in the Admin > Service Providers section or in the flow of a smart campaign when configuring the SSFS.
A "googlesheets_functions.py" file which contains the code used to log the incoming data along with the output data to a Google sheet.
An "inbound-footing-xxx.json" file which contains the credentials used by the "googlesheets_functions.py" for authentication.

Since this blog post will be long enough already I will show in a future post how to get the "inbound-footing" file and how to set up the Google API in the Google Admin console. If you want to figure this out now then you can ask ChatGPT, this is how I found out 😉

Example Payloads Sent By Marketo

Before diving into the /submitAsyncAction endpoint below I wanted to share an example of the information Marketo sends to your code.

The "token", "apiCallBackKey", and "callbackUrl" are used by the /submitAsyncAction code to send information back to Marketo.
The "objectData" contains all the lead and flow step information sent by Marketo. This can contain the information for up to 1000 leads.
The "objectContext" contains all the information about the lead. The "id" field is included by default, if you want to send more fields to the service then you will need to add these fields to the Invocation Payload Definition (see below) so that the admin can add these fields upon install.
The "flowStepContext" contains all the information that was placed in the SSFS by a Marketo user in the smart campaign flow.

The request below is what it looks like when a single lead is sent by Marketo. If you want to see what the payloads look like for multiple leads and for each of the 3 SSFS in the template then you can check out this sheet.

{
"token": "xxx", 
"apiCallBackKey": "123abc", 
"campaignId": 42504, 
"callbackUrl": "https://mkto-cfa.adobe.io/customflowaction/submitCustomFlowAction", 
"context": {"admin": {}}, 
"objectData": [
   {
     "objectType": "lead", 

     "objectContext": {"email": "tyron@theworkflowpro.com", "sms_subscription": "true", "id": 7315607}, 

     "flowStepContext": {"to_phone": "+17126258027", "from_phone": "+18005713186", "message": "Hello World"}

   }
]
}

Other Self Service Flow Step Endpoints

the /serviceIcon endpoint is where you reference the image you want to appear in the Admin console and in smart campaigns for this SSFS.
the /brandIcon endpoint is where you set the brand icon for the SSFS which will only be visible in the "Admin > Service Provider" section.
the /status endpoint is pinged by Marketo daily to check that the service is still active.
the /install endpoint points Marketo to the "swagger.json" file when you are installing the service in the Admin section in Marketo.
the /getPicklist endpoint is optional and only needed if your SSFS will have picklist values offered (see the calcFormula "routes.py" file for an example). This endpoint:
- Retrieves the name of the SSFS field for which the picklist values are being sought
- Uses conditional logic to return the choices for that field name

Brand and Service icons in the Admin section

/submitAsyncAction Endpoint

This is the where the magic happens in your SSFS service. Whenever a SSFS is called from within Marketo, Marketo will send the lead and flow step information to this endpoint where code will then:

Loop through each lead present (remember Marketo can send leads in batches of up to 1000)
- Extract the lead and flow information sent by Marketo
- Do the custom logic that the SSFS was designed to carry out
- Add the output information to a callback object list
- Add the input and output information to a list to later store in a Google sheet
Send a single callback response to Marketo containing the list of callback objects for all the leads
Log the input and output information for each lead to a "Leads" Google sheet
Log the request sent by Marketo and the callback response to a "Batches" Google sheet

callbackObject Structure

The callback object consists of two parts:

the leadData part which is where you include all the fields that you want the SSFS to update (the "id" field is always included so Marketo knows which lead to update).
the activityData part which contains all the attributes you want to be listed in the activity detail for the SSFS when viewed in a person's activity log.

callback_objects.append({
                    "leadData": {
                        "id": lead_id,
                        "quality_score": q_score,
                        "quality_score_detail": q_detail
                    },
                    "activityData": {
                        "company_employee_number": emp_num,
                        "company_industry": industry,
                        "person_job_title": job_title
                        "quality_score": q_score,
                        "quality_score_detail": q_detail,
                        "success": True
                    }
                })

/getServiceDefinition Endpoint

This endpoint contains all the information Marketo needs to know to configure the SSFS upon install.

This section defines the:

The name of the SSFS along with the associated trigger and filter names.
The name of the primary attribute which will always be present when configuring the SSFS in the flow of a smart campaign.
The "Invocation Payload Definition", which is where you tell Marketo what information it needs to send your service.
The "Callback Payload Definition", which is where you tell Marketo what information your service will send back and how it will be mapped to different lead fields.

Invocation Payload Definition

Within the Invocation Payload Definition there are 3 things that need to be defined:

What fields you will be requesting from the configuration of the SSFS in the smart campaign flow i.e. the flowAttributes parameter.
What additional fields you want Marketo to always send along with the flow attributes i.e. the fields parameter.
the userDrivenMapping parameter which can either be True or False. I will explain the implications of this setting in more detail below.

As mentioned in the docs it is possible to send program, my token, subscription, trigger, campaign, and programMember token context along with Global attributes in the Invocation payload.

userDrivenMapping set to False

If userDrivenMapping is set to False then when you install the flow step the "Map Outgoing Fields" section will be populated with all the fields you listed in the fields parameter and then the admin must activate the fields they wish to send and match these fields to the corresponding Marketo fields.

Then in your code when you want to retrieve these field values from the objectContext you will use the "External Field" names shown in the screenshot below (versus the Marketo field API names when userDrivenMapping is set to True).

title = context.get("person_job_title", "")

If userDrivenMapping is set to False and your fields parameter is empty then the "Map Outgoing Fields" section will be skipped during install.

userDrivenMapping set to False

userDrivenMapping set to True

If userDrivenMapping is set to True then even if you have populated the fields parameter it will be ignored. Instead during install the admin will get to choose all the fields that will be sent every time the SSFS is called.

Then in your code when you want to retrieve these field values from the objectContext you will use the Marketo field API names.

phone = context.get("Form_Phone__c", "")

userDrivenMapping set to True

Callback Payload Definition

Within the Callback Payload Definition there are 3 things that need to be defined:

What fields that you want the SSFS to update i.e. the fields parameter.
The fields you want to appear in the activity detail for the SSFS when viewed in a person's activity log i.e. the attributes parameter. Note that if you specify the fields in the callback object as shown above but you don't define them here then they won't show up in activity detail.
the userDrivenMapping parameter which can either be True or False. I will explain the implications of this setting in more detail below.

Note that you do not need to (and won't be able to) declare the Primary attribute in the attributes parameter in order for it to show up in the activity log.

userDrivenMapping set to False

If userDrivenMapping is set to False then when you install the flow step the "Map Incoming Fields" section will be populated with all the fields you listed in the fields parameter and then the admin must activate and map the fields that they want the SSFS to be able to update.

Then in order to actually update these fields with the SSFS you must include them in the leadData section of the callback object and use the "External Field" names shown in the screenshot below.

callback_objects.append({
                    "leadData": {
                        "id": lead_id,
                        "quality_score": q_score,
                        "quality_score_detail": q_detail
                    },

userDrivenMapping set to False

userDrivenMapping set to True

If userDrivenMapping is set to True then even if you have populated the fields parameter it will be ignored. Instead during install the admin will get to choose all the fields that they the SSFS to be able to update.

Then in order to actually update these fields with the SSFS you must include them in the leadData section of the callback object and use the Marketo API field names.

callback_objects.append({
                    "leadData": {
                        "id": lead_id,
                        "AE_SDR_Notes__c": ai_summary
                    },

userDrivenMapping set to True

What's Next After Self Service Flow Steps?

If you are already comfortable with programming and APIs then you should check out these 2 blogs which show you how to get started with the Marketo API and link out to other blogs showing plenty of useful use cases for revenue operations.

Self Service Flow Step FAQs

Here are some answers to frequently asked questions. If you have any more then please get in touch!

What Campaigns Can Use SSFS?

Trigger Campaigns, Batch Campaigns, and Executable Campaigns are all able to call self service flow steps.

Does the Smart Campaign Flow Wait For SSFS to Complete?

Yes, SSFS behave similar to Executable campaigns in this sense that the flow of the smart campaign will wait for the SSFS to complete before continuing on to carry out the subsequent flow step actions.

When do SSFS send leads in batches?

If the SSFS is used in the flow of a trigger campaign then it will only send the single lead that triggered the campaign to your service. If the SSFS is used in the flow of a batch campaign then it will send multiple leads in the batches it sends to your service.