Union.ai and Flyte: Privacy, Open Source, and Building a Commercial Business
Union is a powerful and flexible AI orchestration platform serving as the backbone of cutting-edge AI products. Founded by the co-creators of the hugely successful open-source project Flyte, Union.ai provides software users with the tools they need to move their ideas from concept to high-quality production. Union's aim is to organize the numerous processes required to train and grow the most sophisticated and dynamic machine learning and data products.
About Union.ai
Union is a powerful and flexible AI orchestration platform serving as the backbone of cutting-edge AI products. Founded by the co-creators of the hugely successful open-source project Flyte, Union.ai provides software users with the tools they need to move their ideas from concept to high-quality production. Union’s aim is to organize the numerous processes required to train and grow the most sophisticated and dynamic machine learning and data products.
Challenge
Flyte was experiencing a few challenges. For one, they had always been an open-source code base that promised backward compatibility for its users. In order to uphold that promise, they needed to maintain old versions of the code base forever. There was no way for them to know which, if any, of the old features and versions were even being used, so they couldn’t confidently deprecate them.
“We had one of the simple OSS problems where we didn’t know what versions people were actively using,” said Haytham Abuelfutuh, co-author of Flyte and CTO at Union.ai.
Union was founded by the creators of Flyte. As a commercial business adding value to the open-source project, there was also the issue of not having a clear or complete picture of the project usage.
“We were sort of flying blind. We’re hosting container images on GitHub. We get raw numbers of downloads a month; it didn’t quite tell us much. Zero visibility for our Helm charts,” added Haytham. “Only the most active users in the community had a say in what we did, including what versions or changes were important.”
Flyte is in the Linux Foundation and under the Apache 2 license. As open source advocates, the Flyte co-creators were sensitive to the community’s reaction to them collecting any data.
Solution
“Using data from Scarf, we are able to send messages to the community warning people about certain versions that have a bug or a security issue,” said Haytham. “Knowing that somebody is still using it very actively and being able to warn them is very powerful.”
“It’s actually hard to remember what we did before Scarf; it was one of the very first things we set up for tracking in Flyte,” said Haytham. “With Scarf, we know that people have stopped using version X, so we can safely deprecate it.”
The setup
Union.ai was a very early adopter of Scarf. The Flyte creators saw the value of usage analytics for the health of their community. For open source, they depend on Scarf for their analytics needs. They also have some basics for measuring web traffic and use Common Room to view all the signals together.
Result
Union is an open source first company. It uses Scarf to drive their DevRel strategy and improve their open source project. It also uses Scarf to power its consultative sales approach to help customers where it makes sense.
Union has been successfully leveraging Scarf funnel analysis to shape the product to better fit the market so that they can focus on ensuring that companies can get value from Flyte sooner.
With the help of Scarf, Union realized that thousands of customers are using Flyte in production, and hundreds of them from fortune 2000 companies.
Key outcomes
- Using open source analytics while maintaining users privacy and confidence
- Ability to identify and convert companies using Flyte that would benefit from the additional offerings from Union
- Knowledge of usage patterns identify when to discuss updates with community
- Full funnel analysis
- Maintaining backward compatibility while deprecating old, unused versions
You Can’t Build an Open Source Business Blind: Stirling PDF Case Study
StirlingPDF is one of the largest PDF platforms on GitHub, with an open-source core and an enterprise offering around it. Their platform includes:
Building a Predictable ICP: How Liquibase uses Scarf for GTM Operations
Liquibase is the open source standard for automating database change, with more than 100 million downloads and a community that has been growing for over a decade. Teams adopt Liquibase Community to keep database schema changes in lockstep with fast-moving application releases, then graduate to Liquibase Secure when they need governance, compliance, and control at scale.
From “Flying Blind” to Full Visibility: How Wherobots Uses Scarf to Guide GTM and DevRel
Wherobots is a Series A-stage startup building the Spatial Intelligence Cloud that makes it possible to build production-ready data products with data about the physical world up to 20X faster and at a fraction of the cost of existing approaches. Founded by the creators of Apache Sedona (used by more than 20,000 organizations), Wherobots brings the performance and governance of a modern lakehouse architecture to spatial data workloads through its optimized Sedna-compatible engine and SedonaDB, a spatial-first single-machine runtime. Teams move from complex, do-it-yourself pipelines to 5–20× faster processing without having to manage infrastructure.
