Collecting metrics and measuring the right things are hard.  In the open source community it often seems like those involved in the project or community at large view any sort of data collection and analysis of their projects and community as an afterthought at best, or at worst a complete waste of time.  Consider these two quotes I heard from people I know and respect in the community (protecting names for now): 

“We don’t really track metrics, our focus is on building the open source culture internally and building a great community.” - Recently told to me from an anonymous OSPO for a large fortune 500 company.   

“Metrics are overrated, we do pay attention to a few but it's more about establishing a healthy and engaging community” - Primary maintainer of a large project with over 5000 github stars, and a very large install base.  

This sentiment is held by many in the community.  People don’t get into open source to grow a community, they get into open source to scratch an itch, learn something new, solve a problem, make a difference, or change the world (amongst other reasons). So thinking about how to systematically grow a project or improve the experience for users or contributors is often not even on their radar so to speak.  If there are metrics being collected they are often first level metrics.  

First and second level function metrics -vs-  The “Why” metrics: the OSPO example 

When you are working on a specific problem or in a specific function you tend to focus on the activities, problems, and disruptions that directly impact your work.  If you are looking at metrics most people will focus on the metrics that directly impact their work.  These are first level metrics. 

Let's take the example of an Open Source Program Office (OSPO). If you're unfamiliar, an OSPO takes care of a variety of responsibilities, generally revolving around contributions to open source in enterprises. They ensure corporate responsibility, governance of open-sourced projects, compliance with software licenses, and contribution back to upstream projects. Essentially, they're the backbone of the backend administration and program management.

Most OSPOs I've talked to primarily focus on code contributor-level metrics, tracking project and upstream contributions, possibly even velocity if they're practicing inner sourcing.  Their direct focus is on improving the “contributor” experience and velocity.  

If you are meeting or exceeding the first level responsibilities of your role, you probably are seeing some additional benefits and outcomes from your activities. For instance, if you have more contributors, you should be getting more features shipped or maybe have the ability to answer questions faster.  Measuring these so-called second level metrics gives you more insight into your activities' impact.  

If your organization is only tracking first level metrics, you are missing out on beneficial insights and outcomes your activities are generating.  But even if you have 1st and 2nd level metrics tracking… these may not matter.  

This gets back to understanding your companies, projects, or foundations “Why” for doing these activities.  In our OSPO example, why is increasing contributions or contributors important?  Certainly more contributors is good, but that is not why you are investing in these activities.  If you are a for profit organization the missing link here is how the OSPO connects to the company's revenue or spending. Are we increasing revenue somehow, or decreasing spending?  The connected levels between the contributor metrics and the “Why” metrics may be 10 layers deep, but If this connection is vague, or not understood, an OSPO can become a luxury that executives might find hard to justify, especially during economic downturns.  The last six months we have seen OSPO’s hit hard by layoffs, and I have to wonder if these were better tied to the companies “why”, if these would have been better insulated. 

How do you do this?  One possible solution is to create metrics that tie the OSPO's work directly to the company's bottom line. This might involve some creativity, like calculating engineering resources saved by open-sourcing projects, or costs offset by contributing code and avoiding the purchase of new tools or support contracts. These metrics can not only prove the value of the OSPO but also enhance the sense of connection for those contributing to the open source base.

‍

Metrics for Open Source DevRel: Developer Adoption and Business Outcomes

It's true that metrics can be a bit daunting. When your hard work doesn't reflect in the numbers or your growth seems to stagnate, it can be quite discouraging. Moreover, metrics can sometimes give a skewed picture if they are not interpreted correctly, causing people to be misled.

This challenge isn't exclusive to OSPOs. Developer Relations (DevRel), a field focused on growing the business by building relationships with developers and aiding their success, faces a similar issue. Measuring the direct contribution of DevRel to revenue can be challenging, making it harder to justify its existence, especially when budgets tighten.

Typically DevRel teams focus on the first or second level metrics like # of slack users, GitHub stars, contributors, talks given, blog views, etc. However, bridging this gap between DevRel and revenue is crucial to get closer to the “why” DevRel exists in many companies. To do this, aligning business goals with the outcomes of DevRel initiatives is essential. DevRel should ideally form part of the growth engine, so its activities need to align with this growth. To measure DevRel, identify key indicators (or "North Stars") and concentrate on activities that drive those numbers upwards. For instance, you might focus on turning developers into evangelists for your project or tracking the conversion rate from users to customers.

‍

How to Measure and Foster Your Open-Source Community

While DevRel and OSPO’s may be associated with for profit companies, growth and metrics are equally valuable and challenging for open source projects and foundations.  The “why” for each project may vary greatly, but ensuring your measuring the first and second level metrics as well as connecting them to the why is still equally important.  The “why” of these projects can often come down to: grow a larger user base, grow the community as a whole, and reach some sort of sustainability.  In order to achieve these you should be focused not only on metrics like # of contributors, # of users, etc… but on the tactics (and metrics to verify they are working) to keep contributors engaged and have a healthy community. This might seem tangential to the goal of getting more people to use the software or donate money, but it's actually a vital part of the ecosystem.

A strong community will naturally bring in new users and sponsors as people share their experiences and advocate for the project. But a community won't just happen on its own, especially in the long term. It needs to be fostered and managed with intention, just like any other aspect of an open-source project.

Just as we've discussed before, you should identify what you want to achieve with your community. Is it to have more contributors? Is it to increase the frequency of contributions or the quality? Or is it to have a supportive environment where contributors feel like they can learn and grow? Once you know what you want, you can start to measure and track metrics that will show whether you're moving towards those goals.

For instance, you could measure the number of new contributors over time, or the rate at which new contributors become repeat contributors. You could also measure the amount of interaction in your community spaces, like your Slack channel or forums, to see if people are actively engaging with each other. These metrics can give you a good idea of whether your community is thriving or if there are areas you need to focus on.

‍

Conclusion:

Note: It is not enough to just measure things - you need to act on them too. If you notice that new contributors rarely become repeat contributors, it's worth investigating why. Is there a steep learning curve for contributing? Is the community not welcoming? If your community spaces aren't active, you might need to invest time in fostering discussions or organizing events to bring people together.

In conclusion, measuring an open-source project, whether it's from the perspective of an OSPO, a DevRel, or a project maintainer, is not an optional task. It's a necessary process that needs to be designed and implemented with clear goals in mind. It’s essential to keep in mind that your metrics are not just numbers, but they are a compass that guides you towards your end goal.

Remember, the key is to identify what you want to achieve. Find out how to measure first, second, and the why metrics.   Then track your progress, and adjust your actions based on what you learn. It might seem daunting at first, especially for smaller projects or foundations with limited resources. Still, in the end, this focus on metrics and measurement will help you achieve your goals more effectively and efficiently. And remember, the success of open-source is a collective effort. As the saying goes, "we rise by lifting others," so don’t hesitate to reach out and collaborate with others in the open-source ecosystem as you navigate this journey.

‍

If you haven't read the first part of this blog, you can do it here.