RTP offers a decent amount of customization but we're looking to take it further (ie. what the recommendation bar looks like when it is minimized, etc). Is there any advanced customization that can be done via CSS? And if so, is there any documentation on customization?
You can now use rich media to completely customize the recommendation bar, to my knowledge.
You may want to read through this to see if this will meet your needs: Enabling the Rich Media Recommendation Engine - Marketo Docs - Product Docs
Kristen Carmean is right on here. You have move compelling visual options with the Rich Content Recommendation Engine, but the options for the standard (classic?) CRE are limited, and described here Enabling the Content Recommendation Bar - Marketo Docs - Product Docs.
This is definitely something on the road map, and I highly recommend adding an idea about this.