A Smart List using a "Member of Program" or Member of Engagement Program" filter is qualifying people who are not members of the specified program, or not qualifying people who are members of the program.
Check if you have the filter set to "Member of Program/Engagement Program = true, Program is not [program name]". If this is the case, the logic works a little differently than you may think - what this means is that the person can be a member of any program, as long as it's not the specified program. How you may want to set up the filter would be "Member of Program/Engagement Program = FALSE, Program IS [program name]". That indicates that you want people who are not a member of the specified program.