Sounds like there is something off in your smart list.
Account Owner and Lead Owner may not be the same.
Here is my smart list logic:
Lead Owner Last Name is 'Person X, Person Y, Person Z' (We're trying to remove these former reps that own these leads).
Account Owner Last Name is not 'Person X, Person Y, Person Z' (We do not want these leads or contacts that are named to show up in the list above).