Bulk Program Member Import - Not All Members are uploading

[wbarnes0113]

Utilizing the Marketo API - I have been attempting to upload program members via the bulk method. This seems to fail 30% of the time with not all members uploading to a specific program (randomly only 10/4000 people will upload). This made me think that I should probably add in logic to make sure the task job has a status of "Completed" before proceeding to the next upload. However, I am not sure what the syntax should look like and am at a loss.

Below is my code:

---------------------------------------------------------------------------------------------------------------------------------------------

def import_program_members(program_id, host, endpoint, program_status, access_token, file, filepath, log_id):

    try:

        args = {
          'access_token': access_token,
          'format': 'csv',
          'programId': str(program_id),
          'programMemberStatus': program_status,
          'programStatus': 'completed'
          }
        with open(filepath, 'rb') as oFile:

            files = {'file': (file, oFile, 'text/path')}
            resp = requests.post(host + endpoint, params=args, files=files)
            result = resp.json()
            batchid = result['result'][0]['batchId']
            status = result['result'][0]['status']
            print(batchid)
            while status == 'Queued':
                endpoint1 = '/bulk/v1/program/members/import/' + str(batchid) + '/status.json'
                args = {
                    'access_token': access_token,
                    'batchId': str(batchid)
                    }
                resp = requests.get(host + endpoint1, params=args)
                print(resp)

---------------------------------------------------------------------------------------------------------------------------

The arguments required in the documentation suggest it only needs batchId but that does not seem to work....nor does solely utilizing the endpoint1 (in the code).

Note: the host is utilized in other portions of the code without fail.

[wbarnes0113]

Upload does not show any errors

[wbarnes0113]

Ignore my code being "wrong". Running on new dad energy 😑

[Jo_Pitts1]

@wbarnes0113,

there is a status between Queued and Completed - Importing.

I kind of have to worry about your code being wrong as it's at the crux of your challenge here.

You first need to maintain a holding pattern until it's queued.  it SHOULD be instant.. .but from memory, it pays to check.  You then need to maintain a holding pattern until it has completed importing.  If you prefer, this can be a single loop checking for a status that isn't Complete.

Also:

• You are smashing the API with no wait between checking.  Put a time.sleep(30) in there.
• You aren't setting the value of status in your while loop.

In terms of determining failed rows and/or warnings upon completion, you need to refer to this: https://developers.marketo.com/rest-api/bulk-import/bulk-program-member-import/#creating_a_job:~:tex...

And while it doesn't cover the use case above, check this out.  It doesn't provide 100% coverage, and has some quirks (I must create a fork one day), but it's super helpful https://github.com/jepcastelein/marketo-rest-python

BTW - under the burger menu at the top right of your post, you can edit.  It is valuable.  I just had to use it to correct some weird typos that was still a valid word, but turned a sentence above into nonsense.

Cheers

Jo