I'm not 100% sure, but the more constraints on the range of activities will likely speed up any database pull. Of course, if you make that smart list complex, then it takes longer. A lot of AND/ORs with NOT WAS SENT or something wouldn't help you.
Keep in mind that the smart list in that report is looking at the Emails Sent in a Particular Period and then the subset of leads who meet the smart list criteria.
So if you choose a date range for which none of the leads had an email activity for that set of Emails, you will see 0, even if other leads were sent those emails.