[Resource](/blog-category/resource/) May 30, 2022 # A Different Approach to Measuring Open Source Community Health Community is important to the success of open source software. The health of a community can be measured in different ways depending on the stakeholders involved. To understand and grow a community, project founders and maintainers need visibility into various technical, social, and even financial metrics. These metrics are important because they can help projects form data driven decisions. But what metrics should we be using? By Christina Gorton ![A Different Approach to Measuring Open Source Community Health](/imported/social-og/9f8619f0194816bd-5ca94c75.png) ![Supporting image 1 for Open Source Community Health Metrics](/imported/posts/452aeee044a1a8da.png) ## What is Community Health? Community is important to the success of open source software. The health of a community can be measured in different ways depending on the stakeholders involved. To understand and grow a community, project founders and maintainers need visibility into various technical, social, and even financial metrics. These metrics are important because they can help projects form data driven decisions. **But what metrics should we be using?** This post is inspired by several discussions I have seen around the failings of current open source metrics and the tools used to measure them. This post will discuss what metrics we currently use and what I believe we should be using instead. ## Current Metrics On LinkedIn’s publishing platform Pulse, William Tracz wrote about [Open Source vanity metrics](https://www.linkedin.com/pulse/open-source-vanity-metrics-william-tracz/) and the problem with using them to measure a project’s success. Tracz lamented they could not currently see a better way to gather valuable data about open source projects. **What are these current metrics?** Like Tracz mentions, many of the metrics open source maintainers and contributors rely on are considered “vanity metrics”. This term does not mean these metrics are not useful but they often do not give a full picture of a communities health. Examples of vanity metrics in open source are: - Number of GitHub stars - Number of active commits - Number of new and closed issues - Releases and more. Often these are metrics founders and maintainers care about as they provide a way to quickly show “success” but may not actually help when it comes to important business outcomes and decisions ## Tools You Can Use to See Metrics ### Orbit [Orbit](https://orbit-model.joshed.io/) is a tool you can integrate with GitHub to see metics like: - Pull request opened - Pull request merged - Issue opened - Issue comment created and more. As an open source community manager I used Orbit to quickly track new comments from contributors and maintainers so I could respond in a timely manner. I also used it to see when community members were no longer active. This helped me reach out and get feedback. ### CHAOSS Metrics [CHAOSS](https://chaoss.community/metrics/) is a Linux Foundation project focused on creating analytics and metrics to help define community health. ### Common Room [Common Room](https://www.commonroom.io/) is a growth platform that focuses on a business’ community. The platform helps companies connect their community engagement, product usage, and customer data through the use of artificial intelligence. ### Bitergia [Bitergia](https://bitergia.com/) is an analytics tool for software development teams, open source program offices, and InnerSource programs. Open source projects can use this tool to improve decision making and reporting by analyzing software development community, activity, and performance of open source projects. ### Merico [Merico](https://www.merico.dev/) provides tools to help engineering teams understand and optimize their productivity and code quality. ## What Metrics Should We Use? > “Tracking different types of metrics is essential for free and open source communities. Metrics give project insights into specific efforts and help get a feel of the community’s general perception. For that, tools that can pull data from various sources and develop a visualization of this data will help projects make informed decisions.” - Linux.com Editorial Staff, [Measuring the Health of Open Source Communities](https://www.linux.com/news/measuring-the-health-of-open-source-communities/) In the article quoted above and in William Tracz’s article the need for a tool that can pull more valuable data is apparent. But what data should projects be looking at and what tool could they use? ### Beyond Vanity Metrics What if you could see a project’s impact, identify potential customers, and understand/showcase growth without sacrificing your end-user’s privacy? [Scarf Gateway](https://docs.scarf.sh/gateway/) helps you understand how users are interacting with your open-source project at every step of their journey Instead of answering questions like “how many followers do we have”, Scarf’s data digs deeper in to a project’s community health and answers questions like: - _How many new companies tried our OSS this month?_ - _How many unique users did our OSS have this month compared to last month?_ - _Was the spike in downloads we saw last week genuine growth, or a thrashing CI pipeline from an existing user?_ - _Should I actually spend the time to fix the bug for that one Windows user?_ We can also have visibility into how OSS software is distributed. Instead of wondering who is using our software, Scarf’s data answers questions like: - _What percent of our user base has adopted our latest patch?_ - _What cloud providers are the most popular with our users?_ Scarf can also help uncover community metrics like your projects discoverability, usage, churn, and commercial reliance. Community Managers can use this data to proactively connect with users and grow a healthy and vibrant OSS community. This data compliments the metrics you can get from tools like Orbit and answers questions like: - What is the geographical breakdown of our user-base? - What percentage of our user-base is corporate usage vs hobby usage? - What time zones should we cater to with our online event planning? For a project’s documentation it can answer: - What percent of our user-base is on Windows vs Linux vs Mac? - Which section of our docs are users finding most confusing? ## A New Way This type of rich usage data moves beyond the typical vanity metrics the open source community has typically relied on. While those metrics can tell you a lot about a project, they do not give the full picture of a community’s health. With the addition of tools like Scarf Gateway, maintainers now have the power to make more informed decisions. When maintainers have hard data around who, where, and why companies and individuals are using their software they can better prioritize features, patches, releases and more. Related Resources [![](/imported/social-og/56df6684ceae399c-9c0647d5.png) Resource Sep 5, 2023 ### Beyond the Surface: How to Engage with the Quiet Members of your Open Source Community In the dynamic realm of community management, marketing, and developer relations, success depends upon more than just attracting attention. It's about fostering meaningful relationships, nurturing engagement, and amplifying your community's impact. ](/post/beyond-the-surface-how-to-engage-with-the-quiet-members-of-your-open-source-community/)[![](/imported/social-og/9277af8f0c81f6d2-3589135f.png) Resource Sep 1, 2023 ### Why Open Source Projects and Companies Should Adopt "Open Source Qualified Leads" The modern digital age, driven by the proliferation of open-source projects, is as much about community as it is about the code. This means understanding and engaging with potential adopters, contributors, and users becomes critical for the health and success of a project or product. Here enters the concept of "Open Source Qualified Leads" (OQL). ](/post/why-open-source-projects-and-companies-should-adopt-open-source-qualified-leads/)[![](/imported/social-og/dfcfee806089be6a-45bd3ca2.png) Resource Aug 3, 2023 ### Mastering Telemetry in Open Source: A Simple Guide to Building Lightweight Call Home Functionality Implementing a call-home functionality or telemetry within open-source software often raises privacy concerns within the community. Many parties, including enterprise security teams, customer advocates, and developers, express rightful apprehensions about the transmission, storage, and usage of data. ](/post/mastering-telemetry-in-open-source-a-simple-guide-to-building-lightweight-call-home-functionality/) Article Notes [**Browse all resources** Search, categories, and latest Scarf writing. ](/resources)[**See pricing** Understand how Scarf packages its offering. ](/pricing) On this page [What is Community Health?](#what-is-community-health)[Current Metrics](#current-metrics)[Tools You Can Use to See Metrics](#tools-you-can-use-to-see-metrics)[Orbit](#orbit)[CHAOSS Metrics](#chaoss-metrics)[Common Room](#common-room)[Bitergia](#bitergia)[Merico](#merico)[What Metrics Should We Use?](#what-metrics-should-we-use)[Beyond Vanity Metrics](#beyond-vanity-metrics)