How to judge a Subject Line Test Result: Opens vs Click to Opens?

[Anonymous]

We just ran a subject line test on 5 subject lines. The results have a clear leader for the most opens on Subject Variant "C".  However, Subject Variant "E" had a lower open rate, but a drastically higher Click to Opens rate.

Any advice on which is the better metric to based a final decision on which variant we should send?

Results here:

[Dory_Viscoglio]

The only thing I'd add here, is that sometimes a subject line could be much more indicative of what someone will receive in the email which could lead to the higher click rate... I could promise people a million dollars for opening which would result in a high number of opens, but obviously a low number of clicks. On an email where you got a substantially higher number of clicks, I'd ask what was so different about the subject line before declaring a winner based on opens alone.

[Kristen_Malkov1]

Yeah you could also change up the CTA with the subject line. I tend to test both conurrently, and have them be somewhat related. A good site to check out is "which test won" if you're not familiar. They do regular testing of landing pages and email CTAs, might give you some good ideas for your next test.

[Anonymous]

Is it even possible to do a subject line test and provide different CTA options? I thought that a Subject Line test only allows for subject lines to change, but the exact same email.  

Good point on the ratio. Open rate seems the right way to go. 

[Kristen_Malkov1 · Accepted solution]

Also keep in mind the click to open ration is simply the number of people that clicked on that email that opened the email first. Since E was in the lowest for opens, it would make sense that the ratio of click to open would be higher. 

[Kristen_Malkov1]

Did you provide a different CTA for varient E? If the CTA was the same in all emails, I'd say that you should stick with the highest number of opens. I'm not personally a fan of the Click to Open ratio, but a lot of it is based on personal preference. Also, it might be better to do an additional test with a higher number of leads.