Overview

In this playbook, you’ll learn how Apache Superset integrated Scarf. It details how the Preset team implemented Scarf in their Apache Superset project, as shared during our first-ever Scarf Summit on July 16th, 2024.

What you’ll need: 

• Scarf (sign up for free). You can sign up with a valid email address or your GitHub account.
• If you’re looking to track a container, it must be published to an existing public registry

Step 0: Understand the Benefits

For projects in the Apache Software Foundation (ASF), having detailed insights into how the project and its documentation are being used can significantly impact their development, maintenance, and community engagement strategies.

Maximizing the potential of an ASF project like Apache Superset involves understanding how the community interacts with it. This includes:

• Version Adoption: Knowing which versions are most in use helps in maintaining and supporting the right parts of the codebase.
• Documentation Usage: Insights into which parts of the documentation are being used (or not) may hint at parts of your project that are confusing or can be clarified.
• Community Engagement: Identifying organizations using the software can indicate an opportunity for landing support contracts or sponsorship to financially support your project.
• Security Management: Tracking version adoption helps to pinpoint which versions need security patches and which are safe to depreciate.

Why Scarf?

Scarf provides deep insights into your project’s usage, including:

• Adoption rates of different versions
• Engagement metrics
• Which companies are using your software
• Which parts of the documentation users are frequenting

In 2023, Scarf has formally committed to robust privacy guarantees for the ASF, ensuring member projects can use Scarf in compliance with ASF policies and GDPR requirements.

“The guarantees that Scarf provides around data management and privacy checked all the boxes for the ASF – and actors in the community.” Maxime Beauchemin, founder & CEO of Preset, and original creator of both Apache Superset and Apache Airflow.

Step 1: Develop a Proposal

Before integrating Scarf, anticipate and address common concerns from your community related to privacy, data protection and transparency.

Different projects and communities have their own consensus-seeking mechanisms as part of their governance models. In the case of Superset, they use a process called the Superset Improvement Proposal (SIP). This document defines what should be included in an SIP. They presented this Superset Improvement Proposal (SIP) to the community. 

Components of a Good Proposal:

• Introduction: Explain what Scarf is and the benefits it brings.
• Privacy: Detail Scarf's data practices, including how it processes and protects data, highlighting its approval for use in ASF projects.
• Implementation: Describe how Scarf will be implemented technically. For Preset, this included:some text
  • Scarf's React component in the product.
  • Scarf's tracking pixels in documentation and other resources.
• Opt-in/Opt-out Mechanism: Specify what will be your consent management system and how users can opt-in or opt-out depending on the case. The goal is to ensure an ethical approach while maximizing participation to get the most accurate data possible.
• Data Access: Clearly communicate who will have access to the data collected by Scarf. This can vary by project and may include public access, access upon request, or restricted access to specific maintainers or individuals within the organization or community.

In the case of Apache Superset, they added copious amounts of documentation to be as transparent as possible about Scarf's existence and how to disable it.

Before submitting your proposal, listen to your community. It’s a good idea to conduct one-on-one discussions, gather feedback on potential objections, and clarify any misconceptions as early in the process as possible. 

You can include in your proposal a link to our Privacy Page, where we list the most commonly asked questions regarding our data practices.

Submit the proposal formally and facilitate a voting process within the community.

Step 2: Implement Scarf

For this step, go to Scarf’s Getting Started Playbooks, and learn how to implement Scarf in your project.