Does changing font sizes impact email performance?

Below is a test run by GetUp that found no conclusive evidence of larger fonts improving email performance, however there may be a case for further testing. For more information talk to ben@getup.org.au 

Experiment: Increasing font sizes for emails

Following on from tests conducted by a number of organisations which found a positive impact from increasing the size of fonts in emails, GetUp decided to try this over the last few weeks.

While there was early evidence of a strong positive effect, overall these tests have been inconclusive.

We suspect that this might be related to how we constructed the test, and will be looking at re-running the test using different code.

Structure of experiment

Initially, we structured the experiment with two variations. Fonts were either set at 13px or 16px, which was done using HTML code in two different versions of the same email.

We decided to go with 13px as our default because Gmail uses 12.8px as its default font size.

After running the first two experiments, we realized that we needed to include a control to reflect our current practices. At the moment, we don’t specify a font size at all, so we included an option where we didn’t specify a font size for the remaining three tests.

Results of experiments

Test one – Corporate Tax send to four states

Test two – Corporate Tax send to Victoria and Western Australia

Test three – Better Power campaign

Test four – People’s Climate March recruit send to Melbourne area

Test five – People’s Climate March recruit send to Sydney area

Analysis of results

The first test appeared to produce a statistically significant result in favour of 16px font size, but the exclusion of a control makes it difficult to draw conclusions from this test, in particular because the following tests did not confirm this result.

For the second test, which had similar content and was sent to a similar list in different states, there is practically no difference between 13px and 16px.

For the third test, 16px again beat 13px (although it didn’t quite meet the threshold for significance) but there was no advantage for 16px over our current practice.

For the fourth and fifth test, the control won in both cases. For reasons we cannot explain, the click rate for the control for the fifth test is much higher than the others – we haven’t found any explanation for this, with the samples looking very similar.

Next steps

These experiments didn’t demonstrate any improvement over our existing practice of not specifying font sizes. It’s possible that this reflects that there is no benefit to be gained from changing font sizes, or that we designed the experiment poorly.

We are concerned that those people who prefer a larger font size may already have changed their settings to have larger fonts, and our test would over-ride that and possibly reduce their font size. Thus imposing a 13px font size actually shrinks their font, and for some even a 16px would shrink. It makes it impossible to distinguish those who actually experienced a change in their font size.

We will do some testing using Litmus to see how our proposed font sizes look on different devices and in different programs, before running some additional tests.