That is true. However, I'm thinking about it from a reporting standpoint. It would be nice to right on the homepage of your nurture program to look at the counts of leads in each stream and have it be an accurate representation of who is actually receiving content in that stream.
The reason I don't think I'd want to remove them from the program is then you lose any stats associated with that person. I'm trying to keep a bird's eye view of the program so over the long-run I can look at success metrics, and I feel like looking at the quantity (and type) of people that unsubscribe is valuable. Plus doing any sort of success % would require you to use an accurate number of leads in the program.