I worked at Airship (formerly Urban Airship), a SaaS company that provides mobile marketing tools to large enterprise teams. The main product focus while I was there was sending push notifications, collecting data from mobile interactions and providing that data in a large event stream to customers.
As a Senior Product Designer, I worked on the web app that customers used compose and send messages, create AB tests and consume analytics around message performance.
In 2015, I designed and built the HTML and CSS for the Multivariate Testing feature for Airship. We internally called it A to Z testing. Think AB tests but you can have up to 26 variants and a control. At the time, our competitors where lauding their A to E tests - with them you could test up to 5 variants and a control. We wanted to raise the stakes and decided to provide 26 variants and a control. We also ended up calling it ABN testing
At Airship, we followed a process called product modeling which is a mixture of design thinking ideation methods. I facilitated ideation sessions with product manager and lead engineer on the project. Together, we defined the Target Audience, Problems (Customer & Business), Success Metrics, Goals & whiteboarded potential solutions.
VP of Product recognized that many of our competitors had either an AB test or AE Test feature. They felt we needed to build the feature in order to be competitive within the market.
Multiple high priority customers requested the feature.
Goal: To launch a feature unsing newly released composer that allows a customer to send a test message to up to 26 variants.
The default state starts with choosing the audience. The user can choose "All Devices" or "Target Specific Users". The latter allows them to segment by platform, user generated or system generated predefined segments.
In this screen, the user chooses "All Devices" and we see that they only have iOS app set up with Airship so that is the only platform to show up.
Test set up allows the user to pick which percentage of the audience they want to test and how many variants they want to include. Notice the Summary panel in the Right sidebar. This populates as the user chooses their selections.
In the default state, the user selects percentage of audience and variants. Since they select less than 100%, there is automatically a control group.
If the user selects 100% with 2 variants, the control goes away and A will be sent to 50% and B will be sent to 50%.
The Report shows how each variant and the control group performed. A and B are links in the Variant column to view the message. The user is also able to track events place on CTAs within the message. This can show which group performed the best. Here you can see a whiteboarding session with the product manager for us to figure out how to show all appropriate data.