Yes, you can do this by connecting your product to Marketo using the API.
I would avoid using Marketo for security emails like reset pw. Do not use it for PCI compliance either.
Not a good idea to respond to uncontrolled user action (password reset forms are a particular magnet for hackers) by consuming a Marketo API call. I advise strongly against it.
Sending transactional emails in response to trusted user action (i.e. logged-in, paying users) can be okay, in contrast.
But even in the latter case, you have to consider that you're sending critical content through a server known to send promotional content (no matter how much *we* love Marketo, them's the facts!). For best deliverability, use your own server here. Don't know what's "not working well," but better to fix that than to go down this road.