Java SOAP API - GetMultipleLeads hanging

[Anonymous]

I am trying to retrive leads information using Java SOPA API.

When I start the program it goes well untill about 30-40K leads are fetched and then hangs.

ps shows it is in interruptible sleep.

what can cause this? any way I can avoid this issue?

[Anonymous]

Kiryl,
The batchsize and number of leads you pull have an impact on the throughput of this call.  Here are some recommendations to use this function effectively -

Here are our recommendations -

1. If you know which lead attributes you need to execute your business logic, you can specifying only those attributes you wish to be returned with each lead.  This helps in two very big ways.  The performance is much faster and you will also reduce the response payload.  This will give you the biggest performance boost with the least implementation change.
 
                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="http://www.marketo.com/mktows/">
           <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
              <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
               <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt>
           </leadSelector>
           <batchSize>200</batchSize>
           <includeAttributes>
               <stringItem>FirstName</stringItem>
              <stringItem>AnonymousIP</stringItem>
               <stringItem>Company</stringItem>
           </includeAttributes>
       </ns2:paramsGetMultipleLeads>
 
 
2. Use the <leadSelector> field to specify dates instead of <lastUpdatedAt>.  (This alone won't help performance – however <lastUpdatedAt> is deprecated)
                                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="http://www.marketo.com/mktows/">
           <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
              <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
               <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt> <!-- The timestamp value you were specifying in lastUpdatedAt -->
           </leadSelector>
           <batchSize>200</batchSize>
       </ns2:paramsGetMultipleLeads>
 
3. If you want to really speed things up now that you are using leadSelector is to specify a date range.  <oldestUpdatedAt/> is the since time stamp and <latestUpdatedAt/> is the until time stamp.  With this approach you could thread multiple calls for different time ranges.  If you choose this approach I highly recommend you implement your SOAP client using persistent SOAP connections.  This is a bit more complex, but the rewards can be significant for processing larger data sets.
 

[Anonymous]

Can you elaborate a bit on "persistent SOAP connections". Is there some reference about how to do this?

[Anonymous]

Persistent SOAP connection means you connect once and reuse the connection to fetch all subsequent batches.  See Java sample code below for details -

import java.net.URL;

import java.text.DateFormat;

import java.text.SimpleDateFormat;

import java.util.Date;

import javax.crypto.Mac;

import javax.crypto.spec.SecretKeySpec;

import javax.xml.bind.JAXBContext;

import javax.xml.bind.JAXBElement;

import javax.xml.bind.Marshaller;

import javax.xml.namespace.QName;

import org.apache.commons.codec.binary.Hex;

import com.marketo.mktows.ArrayOfString;

import com.marketo.mktows.AuthenticationHeader;

import com.marketo.mktows.MktMktowsApiService;

import com.marketo.mktows.MktowsPort;

import com.marketo.mktows.ObjectFactory;

import com.marketo.mktows.ParamsGetMultipleLeads;

import com.marketo.mktows.StaticListSelector;

import com.marketo.mktows.StreamPosition;

import com.marketo.mktows.SuccessGetLead;

import com.marketo.mktows.SuccessGetMultipleLeads;

public class GetMultipleLeads {

public static void main(String[] args) {

System.out.println("Executing GetMultipleLeads");

try {

URL marketoSoapEndPoint = new URL(

"https://100-AEK-913.mktoapi.com/soap/mktows/2_2" + "?WSDL");

String marketoUserId = "demo17_1_809934544BFABAE58E5D27";

String marketoSecretKey = "27272727aa";

QName serviceName = new QName("http://www.marketo.com/mktows/",

"MktMktowsApiService");

MktMktowsApiService service = new MktMktowsApiService(

marketoSoapEndPoint, serviceName);

MktowsPort port = service.getMktowsApiSoapPort();

// Create Signature

DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");

String text = df.format(new Date());

String requestTimestamp = text.substring(0, 22) + ":"

+ text.substring(22);

String encryptString = requestTimestamp + marketoUserId;

SecretKeySpec secretKey = new SecretKeySpec(

marketoSecretKey.getBytes(), "HmacSHA1");

Mac mac = Mac.getInstance("HmacSHA1");

mac.init(secretKey);

byte[] rawHmac = mac.doFinal(encryptString.getBytes());

char[] hexChars = Hex.encodeHex(rawHmac);

String signature = new String(hexChars);

// Set Authentication Header

AuthenticationHeader header = new AuthenticationHeader();

header.setMktowsUserId(marketoUserId);

header.setRequestTimestamp(requestTimestamp);

header.setRequestSignature(signature);

// Create Request

ParamsGetMultipleLeads request = new ParamsGetMultipleLeads();

// Request Using LeadKey Selector

// //////////////////////////////////////////////////////

// LeadKeySelector keySelector = new LeadKeySelector();

// keySelector.setKeyType(LeadKeyRef.EMAIL);

//

// ArrayOfString aos = new ArrayOfString();

// aos.getStringItems().add("formtest1@marketo.com");

// aos.getStringItems().add("joe@marketo.com");

// keySelector.setKeyValues(aos);

// request.setLeadSelector(keySelector);

/*

* // Request Using LastUpdateAtSelector

* ////////////////////////////////////////////////////////

* LastUpdateAtSelector leadSelector = new LastUpdateAtSelector();

* 

* GregorianCalendar gc = new GregorianCalendar();

* gc.setTimeInMillis(new Date().getTime()); gc.add(

* GregorianCalendar.DAY_OF_YEAR, -2);

* 

* DatatypeFactory factory = DatatypeFactory.newInstance();

* 

* ObjectFactory objectFactory = new ObjectFactory();

* JAXBElement<XMLGregorianCalendar> until

* =objectFactory.createLastUpdateAtSelectorLatestUpdatedAt

* (factory.newXMLGregorianCalendar(gc));

* 

* GregorianCalendar since = new GregorianCalendar();

* since.setTimeInMillis(new Date().getTime()); since.add(

* GregorianCalendar.DAY_OF_YEAR, -5);

* 

* leadSelector.setOldestUpdatedAt(factory.newXMLGregorianCalendar(since

* )); leadSelector.setLatestUpdatedAt(until);

* 

* request.setLeadSelector(leadSelector);

*/

// Request Using StaticList Selector

// //////////////////////////////////////////////////////

StaticListSelector staticListSelector = new StaticListSelector();

// staticListSelector.setStaticListId(value)

ObjectFactory objectFactory = new ObjectFactory();

JAXBElement<String> listName = objectFactory

.createStaticListSelectorStaticListName("CustomFormula.listOfLeads");

staticListSelector.setStaticListName(listName);

// JAXBElement<Integer> listId =

// objectFactory.createStaticListSelectorStaticListId(6926);

// staticListSelector.setStaticListId(listId);

request.setLeadSelector(staticListSelector);

ArrayOfString attributes = new ArrayOfString();

attributes.getStringItems().add("FirstName");

attributes.getStringItems().add("AnonymousIP");

attributes.getStringItems().add("Company");

request.setIncludeAttributes(attributes);

JAXBElement<Integer> batchSize = new ObjectFactory()

.createParamsGetMultipleLeadsBatchSize(150);

request.setBatchSize(batchSize);

SuccessGetMultipleLeads result = null;

int count = 0;

do {

if (count > 0) {

String pos = result.getResult().getNewStreamPosition();

JAXBElement<String> streamPos = new ObjectFactory()

.createParamsGetMultipleLeadsStreamPosition(pos);

request.setStreamPosition(streamPos);

}

result = port.getMultipleLeads(request, header);

JAXBContext context = JAXBContext

.newInstance(SuccessGetLead.class);

Marshaller m = context.createMarshaller();

m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);

m.marshal(request, System.out);

m.marshal(result, System.out);

count += result.getResult().getReturnCount();

} while (result.getResult().getRemainingCount() > 0);

System.out.println("Returned a total of :" + count + " items");

} catch (Exception e) {

e.printStackTrace();

}

}

}

[Anonymous]

that is precisely how I am doing it. with few additional things:

- my batch size is 1000
- I get all the columns

Basically I need to sync the data from Marketo with another system that we are setting up. First I need to pull all the "historical" data and then set incremental daily or hourly updates.

The issue comes retrieving historical data, my program starts hanging after few thousand of records retrieved (out of 250K)..

Any ideas on how to solve this?

Thanks,
Kiryl