The "correct" count you should rely on is the one that is in the email performance report (it also matches the summary page of the smart campaign). The reason the counts will differ is because business logic is added on top of the email performance report. The smart list is pulling a raw list of leads for which that activity occured. For example, many times a “soft bounce” can occur after a delivered event. This means that the delivered count could initially look higher, then slowly decrease (it may decrease steadily over time) as we receive bounces from mail servers. It's a weird concept to grasp, but that is the order in which these soft bounces are reported to the sender. As we process all of these activities, we update the email performance report accordingly. However, if you were to run a smart list of "was delivered email," it would still include the folks that had a "delivered" event logged to them (even if there was a bounce shortly after).
This is just one example of why the counts will be different...there are many more nuances similar to the one I just mentioned. I recommend using data from the email performance report if you're trying to understand your mailings.