Testing is one of the most effective, if not the most effective method for optimisation in email marketing. But how do you get meaningful results when testing different mail variants? What are meaningful results anyway?
The right size of the test group
How large does your distributor or test groups have to be to achieve meaningful results? One test group should account for at least 10 percent of the total distribution (or entire segment), but at least 1,000 subjects per test group. The deeper you move in the funnel the more people you need.
Significant results
You have performed a test and the result is available. At what point are results meaningful? It is difficult to give a general answer. For example, if your opening rates are usually between 31 percent and 32 percent, Subject A with an opening rate of 31.8 percent does not necessarily have to be better than Subject B with 31.3 percent. If Subject A had 38 percent instead, the result would be more significant. It is important that you make hypotheses (e.g. “a difference of more than two percent in the opening rate is significant”) and check them again and again.
Additional tip: Exclude customers, who always react to your emails anyway, from your tests. You no longer need to optimize for these customers.
Segment the right way
Imagine you have an email distribution list with 100,000 customers and you are testing an email with two different subject lines. Subject line 1 has a five percent higher opening rate than subject line 2. A clear result, isn’t it? Unfortunately it is not that simple. Because maybe there are different segments “hidden” in these 100,000 customers and not all of them have subject line 1 working better. Maybe it even achieves worse results for some of them. Segmentation is therefore very important for testing in email marketing. First use segments that already exist in your marketing and check whether they react differently to different test variants.
Use testing to verify hypotheses about your customers and possibly form new segments. Take a close look at whether certain test results correlate with certain customer characteristics.
With artegic‘s ELAINE technology you can combine detection and optimization of segments that have been undetected so far. By means of artificial intelligence, ELAINE automatically recognizes correlations between factor values and segment characteristics during test operation and automatically forms segments that are as granular as possible depending on which segment characteristics work best with which factor values. Detection and optimization works while the campaign is still running, i.e. immediately and in realtime. Content is adapted in realtime for the newly identified segments or even for individual customers. If a customer opens an email one hour after sending, he may receive different content than if he had opened the same email five minutes after sending.