Announcing The Scarf Gateway
March 2, 2021
This article was originally posted onHackernoon
After half a year of hard work from everyone on our team, I'm thrilled to introduce a foundational piece of Scarf's toolchain for bringing observability to open-source software distribution.
The Scarf Gateway is a central access point to all of your containers and packages, regardless of where those artifacts are actually hosted. It provides in-depth insights into how your packages are being downloaded and used, and makes it simple to host your packages from your own domain without requiring any infrastructure. (I’ll use the term packages throughout this post as an umbrella term to refer to libraries, containers, or any bundled software artifact you might fetch from a registry.)
Rather than directing users to pull your packages straight from the registries, with Scarf you use your own domain (or we can provide one if you'd prefer) as a static point decoupled from the registries themselves. The Gateway serves as a thin, configurable redirection layer that sits in front of your current registry or registries. When an end-user downloads your container through Scarf, the Gateway redirects or proxies the request to the right place, and surfaces the associated data to you in your Maintainer Insights dashboard within the Scarf website.
By distributing software through your own domain rather than what's provided by your current registry, Scarf puts your software's distribution in your control, where it belongs.
I’ve written before about the growing misalignment in incentives between open-source maintainers and the package registries they use to distribute their software to their users. Established registries are incentivized to deal with competition by strengthening the lock-in effects of their platforms, rather than providing the best product.
This has resulted in two critical, systemic issues:
- Data Lockout: Basic information about the distribution of open-source software is almost fully hidden from the people who build that software. The registries collect a wealth of data about how end-users are downloading packages, but keep it to themselves. Maintainers don’t get access to the data and insights, which limits their ability to support end-users or grow any business associated with their project. This lack of data sharing is holding open-source back and must change.
- Platform Lock-in: Maintainers of widely used open-source projects experience tremendous difficulty moving their packages to another registry if/when better options emerge, since that update is itself a breaking change for current users.
Open source maintainers are locked into the very platforms that lock them out of their own data.
We need to recognize and advocate for the value of sharing data about who is using open-source code, and how it is being used. Sharing data between users and maintainers strengthens the open source community, and by enabling feedback and communication creates better software for all.
Benefits of using the Gateway
The Scarf Gateway addresses the two issues we identified above: Open-source maintainers currently don't have access to already existing data about how their software is used, and they are tightly coupled to their current registry provider.
Unlock your data
Putting the entry point to your software distribution channel under your control ensures that you’ll never be cut out of your distribution data again. The Scarf Gateway surfaces your usage data without requiring unnecessary analytics hooks, building bespoke telemetry solutions, or hosting your own registry. You'll be able to answer questions like these that you previously could only have guessed at:
- Of your total downloads last month, how many were unique?
- How many of your total downloads last month came from businesses vs individuals? Which companies, specifically?
- How many of your downloads came from CI vs from a person?
- Which container runtimes are being used to run your image?
- Where in the world are your users located?
These questions are all trivial to answer with existing metadata about your download traffic. Scarf can help you unlock that data; it should have been yours all along.
Unlock your software distribution
By taking back ownership of your software’s distribution with Scarf, you need never be constrained by the challenge of switching registries. Push your containers to a new registry and tell Scarf where to get the new images. Your download traffic will update instantly with no breaking changes, no user impact, and no painful or complex migrations!
Try the Scarf Gateway beta now
The Scarf Gateway is in an open beta period that's available to everyone. The current feature set is free, and will remain free. We have a ton of incredible free and paid features on the roadmap, and you can get started right now. Follow us for updates, and let us know what you think!
Latest blog posts
Tools and strategies modern teams need to help their companies grow.