If these actions do not occur as above, we may not attribute the unsubscribe to the email asset.
Why are some being tracked? Those that are being tracked are almost always because of the record unsubscribing via the email host provided link (ex. GMail hosted emails)
Are records unsubscribed even if it does not attribute to the email? Yes, as long as the field Unsubscribed is true on the record, we will not send the record a non-Operational email.
If you would like a one click unsubscribe solution for your records, this is still possible through the use of the Forms 2.0 API and submitting the form in the background.