Yes the custom entity wouldnt show up on the left tree however if you create a relationship between your custom entity and contact or lead or account entity and sync down the field to marketo, you will see the field as an attribute under the person or company. From which you can filter your leads accordingly.
Custom entities work a little different than regular person or company fields. You say you can see the filter "Has Custom Entity" and that is exactly what you need. You add this filter to your smart list and with the "Add Constraint" in the top right corner you can access the fields in the custom entity to your filter and set criteria for then.