Maybe dont use a smart list and use a basic custom list with just the names of the people who you want to receive the email. Very simple to make, time however may depend on how many you have on it, but you could just copy them from your smart list over and remove the ones who you dont want to use.....Quick and simple!
Hopefully this helps, we tested a list exactly like it when we were testing some email prior to going out love to our clients.
Barry