I have a custom object linked to contact object in salesforce ( NtoN relation)
How it works in Marketo if I sync this custom object ( can I do smart list based on criteria in this custom object )
Yes, you can run Smart Lists on Custom Objects. Search for a filter that starts with 'Has'.
You can then add the constraints so you are able to find the right record within that Custom Object. All the results will be for this particular lead.
You won't be able to trigger off changes to the Custom Object directly. The best way around this is to create a datetime field that your CRM changes when a record is added/updated to the Custom Object. You can then trigger off this field and search for updates to a Custom Object.