Rob,
How is the smart list constructed? I usually only use 4 Not Was Filters: visits web page, clicked, opened, and filled out form. Often a good 70-80% of the database will qualify the first day and a few from there on in, but it is taxing on the system to run this so I only do so as a batch, nightly, and to qualify every 30 days.
I'd call support if you see this breaking, they may have ideas on how to help.
And yes, inactivity scoring is a good idea if you rely on scoring heavily. An alternative is to only score key behaviors and just not worry about inactivity.