Hello folks,
I figured it out. We have an email unsubscribe form which in the past was not working properly and updating leads to unsubcribed. There were leftover leads who had filled out the form but were not marked as such. Fixed this now with a smart campaign.
Thank you for your suggestions!