I worked at Airship (formerly Urban Airship), a SaaS company that provides mobile marketing tools to large enterprise teams. The main product focus while I was there was sending push notifications, collecting data from mobile interactions and providing that data in a large event stream to customers.

As a Senior Product Designer, I worked on the web app that customers used compose and send messages, create AB tests and consume analytics around message performance.

In 2015, I designed and built the HTML and CSS for the Multivariate Testing feature for Airship. We internally called it A to Z testing. Think AB tests but you can have up to 26 variants and a control. At the time, our competitors where lauding their A to E tests - with them you could test up to 5 variants and a control. We wanted to raise the stakes and decided to provide 26 variants and a control. We also ended up calling it ABN testing

Process

At Airship, we followed a process called product modeling which is a mixture of design thinking ideation methods. I facilitated ideation sessions with product manager and lead engineer on the project. Together, we defined the Target Audience, Problems (Customer & Business), Success Metrics, Goals & whiteboarded potential solutions.

Data / Research provided

Competitive Analysis
VP of Product recognized that many of our competitors had either an AB test or AE Test feature. They felt we needed to build the feature in order to be competitive within the market.

Qualitative Feedback
Multiple high priority customers requested the feature.

Target Audience

Problems

Marketers' problems

Business Problems

Success Metrics & Goals

Goal: To launch a feature unsing newly released composer that allows a customer to send a test message to up to 26 variants.

Designs / Screen Flow

Default state

The default state starts with choosing the audience. The user can choose "All Devices" or "Target Specific Users". The latter allows them to segment by platform, user generated or system generated predefined segments.

First screen of AB testing feature

Audience Setup

In this screen, the user chooses "All Devices" and we see that they only have iOS app set up with Airship so that is the only platform to show up.

Test setup

Test set up allows the user to pick which percentage of the audience they want to test and how many variants they want to include. Notice the Summary panel in the Right sidebar. This populates as the user chooses their selections.

Default test setup

In the default state, the user selects percentage of audience and variants. Since they select less than 100%, there is automatically a control group.

Test with no control

If the user selects 100% with 2 variants, the control goes away and A will be sent to 50% and B will be sent to 50%.

One Variant at 30% with 70% control group

Two variants at 15% each with 70% control group

Content

Composer for Variant A

Composer for Variant B

Report

The Report shows how each variant and the control group performed. A and B are links in the Variant column to view the message. The user is also able to track events place on CTAs within the message. This can show which group performed the best. Here you can see a whiteboarding session with the product manager for us to figure out how to show all appropriate data.