False open rate results?

[Anonymous]

I ran just ran a split test with my winner critera/CTA as a call in, with the body content of the email was different. There was no difference in the number of call-ins but I did notice that the Challenger provided a 3% increase in open rate. This is of course a good thing, but I'm stumped as to how or why I got these results, as the From name, From Email, Subject line, and Preheader were all identical.

One difference is that the Challenger had an image in the body content removed. I understand that opens are tracked when someone downloads images. Is it possible that the lack of images in the body just encouraged more people to download images, creating a false lift in my open rate? Or any other ideas on how this could happen is appreciated!

[Anonymous · Accepted solution]

Hey Manisha > 

Perhaps 3% more of people in the Challenger group just opened the email? I don't think there's any conclusion I'd draw from the set of data you put forth. I'd test this image idea another 10 times before I'd draw any conclusions.

Changes in the body of an email don't change open rates (**unless it effect deliverability**)

[Anonymous]

I'm new to testing - how often do identical assets get varied results like that? Out of curiousity I plugged it into a statistical significance calculator, and it's 99% confident that the challenger won in terms of Opens. Deliverability doesn't seem to be an issue either, as delivery rates are similar as well.

You're right - I will definitely test more before applying any sort of changes. But the thought that I could get a false increase in opens due to number of images is interesting to me, and was wondering if this was a "thing" I should be thinking about.

[Josh_Hill13]

It may be possible that your image change modified what was seen in Preview of the email reader, and encouraged more image downloads. I wouldnt use open rates for anything as they are unreliable due to the image blocking software.

[SanfordWhiteman]

As @Adam W said, if content changes end up affecting deliverability, they will in turn affect openability.

A low ratio of text to image area is considered spammy (error-prone or not, this relationship is coded into major anti-spam software, based on the practice of hiding text within large images).  So if by reducing the # of images, you change the text:image ratio from, say, 2:1 to 6:1, you will likely see more deliveries.