My ride as a product manager to improve cabify ratings

I would like to start in Medium with a post related to my current role in Cabify. It is not a post on how to make a product, but how it worked for me to make a product in a specific case. I hope you enjoy the reading.

It was December 18’, I worked as an Associate PM in the Rider Experience team with two months of experience in the Product Management field so far. By that time, I faced my first big challenge: I had to define the team mission and KPIs.

We concluded that our mission was to create a pleasant, consistent and easy to use experience for our riders. One of the KPIs that would measure the success of our experience was the ‘% of rated journeys’. We understood this would be a good indicator of how good the experience was, directly obtained from our riders.

At that time, <25% of the journeys were rated. Clearly, we had a lot of room for improvement. Therefore, we decided to get down to work.

Understand the status quo

The first step was to understand how riders perceive our rating and identify its pros and cons. And, for having such information, we had the help of the User Research discipline.

Lærke and Itsaso did a large explorative research not only with riders but also with drivers. We not only had to understand what riders thought about the rating but also if the information gathered was useful for drivers. The research was carried out in three different countries where we operate that had different percentages of rated journeys. The objective was to understand:

• How the current rating system was used.

• How we could increase the usage of the rating system.

• How the rating system could work better.

After some weeks, we had a complete and valuable report with all the information gathered. The report was presented and shared with all the stakeholders to provide feedback about the points mentioned above.

Thanks to the User Research report we were able to identify, directly from our users, what was working with the current ratings system and the areas that should be improved. I list some of the most important insights below.

Positive aspects

• We already had a rating system. Our riders were used to it. They understood the value of this feature as a way to improve the service.

• The rating system was fast and easy to use.

• The free text helped riders to explain in detail their experience.

Aspects to be improved

• We were asking our riders to evaluate their drivers instead of asking about the whole experience. (Marta, please rate your driver)

• The five stars system was inconsistent. The meaning of each star was interpreted differently by users. So, how could we use the information obtained? We would be doing it but based on our own rating perception.

• People in Cabify didn’t use that information too much. It was difficult to get the most out of it.

• Drivers were not able to know what they were doing well and what things had to be improved.

Define vision

Since I had little experience, I got help from Tiago and Ramón, the most senior PMs in Cabify at that moment. We had brainstorming sessions to set the vision and identify the key pieces of this feature (problems, goals, metrics, some product considerations…). All of them were collected in a product initiation document.

Our main objective, as Rider Experience team, was to offer the best experience to our riders. The rating was a powerful tool that would help us to achieve the goal. But, first, we needed riders to trust us, to feel safe when telling us about their experiences. We wanted to guarantee them that Cabify cares about their feedback. Then, we had to ensure that we would collect enough quality feedback so that it could be useful for employees and drivers.

If we achieved our objective, then we would be able to continuously evaluate the quality of our service. This way, we would identify the areas where we are doing well and, at the same time, the ones we need to improve.

Let design deal with the problem

Once problems, vision and goals were clearly defined, it was time for the design discipline to have more active role. They had enough information to understand the problem and the objectives we wanted to achieve, so it was their time to fully work on the solution.

Embrace all ideas

They had some time to sketch out some proposals. Later, we discussed them and got great feedback that allowed us to improve over the first wireframes. After some of these sessions, we came out with a new prototype. It was a complete redesign: we changed stars for emotional faces, that are a metaphor of the emotional state of the rider with our service. We had different tags to enrich the rating and give the option to leave free comments not only when the rider rated bad but when they rated well too.

Since it was a risky change and we were not sure about the result, we wanted to test it with our users and gather their feedback before moving forward.

Involve your users

With the support and guidance of User Research, we tested those prototypes with our users (riders, drivers and Cabify employees).

We wanted to see riders’ reactions to two different metaphors: stars (the old one) and emotions (the new one). We did both individual and group interviews. There, we found that riders identified better with emotions: also that showing fewer options to choose was positive.

We sat down with some drivers to show them a quick sketch of how it would appear on their App. They liked it not because it was a different metaphor but because we would display more information about areas going well and not so well. Now, they would have more information to understand and improve their service.

Finally, it was time to show it to our colleagues, who also liked the idea of having more valuable feedback. They also found interesting the idea of having two types of feedback: the one for improvement and the one to recognize the job well done.

Apply feedback and get the final solution

Design did a great work putting everything together and coming up with a final proposal. This time, every detail was taken into account.

Time for engineering

We had a kick-off session with engineers to show them the final proposal. Marta, our designer, did a wonderful job explaining the reasoning behind each design decision.

The team gave a lot of feedback, asked questions and challenged us. This session helped us to define the minimum viable product (MVP).

Define the MVP

Lots of changes were presented in the kick-off session. We had to decide how to reduce the effort but still give value to our users. This would help us to measure the impact and decide if more effort was needed or not.

We agreed that changes would be done only in Apps. We would change the metaphor, add more tags to each option and leave the opportunity to write free comments if riders considered it. We would gather all this information in Amplitude so we could analyze the results later on.

Let’s experiment!

After some weeks of development, we included it in the next App release and we opened it to a % of our riders. It was a matter of time before we had enough data to analyze.

Then, the data came: both Firebase and Amplitude indicated that there were >99,99% certainty that the treatment variant outperformed the base one. We had an improvement up to 30% over the baseline.

People were rating more, they were giving more details about the rating and they also used free comments combined with good feedback (something that did not happen before because the option didn’t exist). We also discovered that some of the rating tags were barely used, some free comments could be represented with new tags and still quite a lot of people proactively skipped the rating, among others.

We concluded that it was a great rating iteration and we could roll it out to everyone.

Communication

Communication was really important during the whole process: with other disciplines such as User Research or Design was key to understand the problem and think about solutions; with product stakeholders in order to let them know about the initiative and take into account their feedback; with business stakeholders so they had the chance to let us know their concerns and ideas; with all the product team during the experimentation phase and also when the results came.

We used different ways of communication: meetings to present User Research report, brainstorming sessions, kick-off meetings to present the design proposal, a specific Slack channel for this topic and announcements to the whole product team or company when needed.

Lessons learned

I’d like to share with you some of the things I learned during the whole process. I hope you find them useful.

Rating

• Metaphors are key when asking for feedback. People understand feelings, there’s no need for explanations.

• Reduce decision fatigue. It’s easier to decide if we show fewer options. It’s also less prone to confusion.

• Regarding rating sub-options, there’s a consistency among all of them, independently of the feeling/option. You’re judging the same thing but giving different adjectives (excellent, regular, bad).

• Free text is good. We can learn from riders’ feedback at the same time we offer a space for them to communicate with the driver to congratulate them on their work.

Process

• Metrics are key. It’s important to have an initial definition of the metrics we care about. They will guide us to identify where the opportunities are. They will also help us to understand if the solution proposed is working for our users or not.

• Involving all stakeholders (riders, drivers, internal ones) in the initiative guarantees its success. They will feel part of the initiative at the same time they will help you to anticipate mistakes.

• The sooner the engineers are involved, the better. If possible, make them participate in user testing too. It’s important for them to understand first hand who they are solving problems for. This way they will be closer to the product.

• Instrumentation is as important as the whole process. Thanks to it we will be able to analyze, learn and improve. We must pay extra attention when defining and implementing it in order to minimize mistakes.

• This type of process cannot be done with all initiatives. It’s slow but very valuable so you should decide when it is worth it and when it is not.

I had to face for the first time with different challenges during this process. It helped me grow and identify many important aspects related to product management to take into account from now on. If I should highlight only one it would be the communication. It’s very important that you continuously communicate with your team and the people interested in the topic. It’s important that they know you are working on it and what your plans are so everyone is aligned with the vision and objectives.

[Join the Cabify team to help us to improve our product.](https://cabify.careers/es/?utm_source=medium)