Babbel’s VP Product Design explains why the north-star metric alone is not enough for user tests

The holy grail is A/B-testing. „We do this at every opportunity,“ says Scott Weiss. But to do this effectively, he and his team shifted away from just aiming for the north-star metric, which, at Babbel, is weekly active paying users. By adding supporting metrics they were able to get a much better grip of what user behavior can actually impact.

The VP Product Design of the language learning app came to our DL Summit 2019 to talk about the challenges of a product that the users would love to use more. If only it weren’t for their weaker self that prevents them from doing the next Spanish lesson or any other of the twelve languages, you could learn with Babbel. So the key challenge for Scott and his team is to understand the user’s behavior and how they can impact that effectively.

Scaling starts on the US market

Scott compares this to a membership at the local gym. Many people carry the membership card in their wallet. But few go there every week. Keeping users engaged and motivated is one of the most critical challenges for Scott and his team of UX and product designers.

Though Babbel started in Berlin back in 2007 – and still has its headquarters there – the focus has shifted away from the German market. Scott says: „We firmly believe that if we can create a successful product in the US, we can scale to the rest of the world. We have been very successful with that.“

Adding to the north-star metric

The user journey at Babbel follows three stages: impression, engagement, and learning. „The product design is organized according to those areas,“ explains Scott in his fireside chat with host Till Ohrmann. In this process, his job is „to participate in constructing and developing the product strategy and translate it into messages that help my team.“

Each area has its metrics that guide the teams. With impression, it is „visit to lead“ and „visit to sale“ to make sure „that we are engaging with early users in a way that has results.“ Engagement and learning are led by weekly active paying users (WAPU). However, „we recently added another metric, which is a seven-day return to learning,“ says Scott. While WAPU is the north-star metric for the subscription-based service, Scott and his team found the seven-day return to learning more effective for A/B-testing.

„It is more straightforward. WAPU is an aggregated metric, and it’s hard to tell what we can impact. With the seven day return, we are able to test the results of a given experiment more effectively and rapidly.“ Scott gives an example of a successful experiment that even influenced how new users engaged with the product.

Patience and commitment

A new feature was designed to rate how confident users feel in specific conversations, for example, ordering a coffee. The goal was to align new language lessons according to this rating, thus improving the learning experience. To the surprise of the team, this affected the seven-day return positively, but it also drove visits to lead and visits to sales in an unexpected way.

So, how does Babbel come up with new features regularly? To Scott, patience and commitment are essential factors in the user research and testing: „We need to establish minimum viable iterations and have committed ourselves to at least three or four iterations before we ditch something. Because it would be a real shame if one more iteration would have made a big impact.“

About Scott

Having started with an internship at Apple in the late 1980s, Scott Weiss has seen many design trends come and go. He went on to work for Microsoft, SwiftKey to running his own startup for eleven years. From the first PCs over early handheld devices to mobile phones, one thing has never changed for Scott: „Every company says, how important design is. However, few live by the most important standard: Always put the user first.”

LinkedIn: Scott Weiss, Till Ohrmann

Read Next: How to iterate fast with GitHub’s Senior Director of Product Design