Our mission here at Scarf centers around increasing and enhancing the connections between open source software maintainers and end users. Our products so far have been primarily focused on maintainers, giving them control over and visibility into how their software is distributed. But we also believe that the status quo does not serve end users very well. Scarf is working to reduce the complexity and increase the efficienty of the end-user open source integration experience, leveraging a new project called Nomia.

Any given project can be built, installed, and combined with other projects in a huge variety of ways. Unfortunately, in many cases users are given a single binary to download and instructions for the most common use case. If a user is running on a different platform, or needs a patched version of some core library, or needs to use two different projects that have different distribution mechanisms, they're out of luck without significant expertise and manual work. This gets especially painful when users want to combine open source projects and their own proprietary projects in a way appropriate for their end goals.

The difficulties don't stop once the user gets the projects they need up and running once. Dependencies need to work correctly on developer machines, for business QA testers, in CI, and of course in production. These different contexts may bring different requirements, whether in terms of the technical functioning of the system, or the access rights and technical capabilities of the user. And while all of these different use cases need to be respected, the project depended upon must otherwise operate in the same way, at least as far as visible functionality is concerned.

And even once the user figures out how to get their dependencies working everywhere they're needed, in the form they're needed, they're still not done! Depending on the complexity of the project, building everything needed in each context can be very expensive in time or computational resources, even though much of the work is identical for each case. And if dependencies are changing frequently, especially when they include internal projects under active development, those costs can really pile up!

The Scarf Environment Manager and Nomia

As I mentioned in my last post, there are a number of systems that achieve significant benefits in reproducibility, efficiency, and composability through the use of semantically meaningful names. A file hash in git or a code reference in Unison are not just arbitrary labels; they precisely capture the important aspects of the identity of the resource being named, and fit into a general system for combining those names. One of the systems we discussed, Nix, even addresses the package management use case directly. Can we use these principles to address all of our end users' needs in acquiring, configuring, and using open source software in all the relevant contexts without undue cost?

Today we're announcing the first tool in the newly revamped Scarf CLI: The Scarf Environment Manager. The Environment Manager is a tool for, you guessed it, managing the environments that user sessions, development shells, and automated systems run within. This includes managing the packages that are part of the environments and treating the environments themselves as first-class resources which can be merged, shared, etc. The goal is for the Environment Manager to be the way for open source users, especially users who want to take advantage of Scarf's ecosystem and benefits, to acquire and use open source packages.

This first-class focus on environments and the integration with every feature and service Scarf offers are valuable in and of themselves. That said, the real power of the Environment Manager (and the CLI as a whole) comes from how its resources are referenced. Packages and environments are each identified by names, names which combine with each other systematically and convey precisely what the identified resource is.

Suppose you want to work on a Go project. You want an environment where your normal tools are available, like your editor, plus your Go-specific tools like the Go test coverage tool, plus the dependencies of the project itself. The project you're working on is in a repository that happens to uncover a corner case bug in git, so you need to use a version of git with a fix you just developed locally. In the current naming syntax, which is subject to change, this combined environment might be named {% c-line %}merge-user-and-project-envs(my-env.git: build-c-package(src: ../git-patched).output){% c-line-end %}. Once the relevant features are implemented, the Environment Manager would automatically detect that you're working on a Go project and combine your global user environment, your Go environment, and the environment for the project, with the global user environment overridden with a patched git version. This automatic detection would be a property of the names used, with each step being manually overridable. This high level name would be successively reduced to more concrete names, such as first {% c-line %}merge-envs([ my-env (git: build-c-package(src: ../git-patched).output), my-env?lang=go, go-project-env(./go.mod)){% c-line-end %} and later {% c-line %}env-from-packages([ vim, build-c-package(src: ../git-patched).output, go-pkgs:cover, go-pkgs:argv, go-pkgs:cockroach ]){% c-line-end %} .

As the names get more concrete, they get more reproducible and shared components of work can be identified: You may never have built the top-level environment, but if you have already built go-pkgs:cockroach and your coworker has built your patched git version, those can be reused. In this way, users can specify exactly what packages they want, include them in whatever environments they want, and automatically share the common work between contexts.

This system for naming resources is not limited to environments and packages. The Environment Manager is built on top of a new general purpose system called Nomia. Nomia is designed to give semantically rich names to any kind of resource whatsoever and enable composition of those names, across domains. In principle, it can subsume all of the examples from the last post, and their combinations: Imagine using Nelson to deploy a Unison service which has a native depencency on a system library in nixpkgs, except each system is operating on the same principles and underlying infrastructure. In addition, your specification of the deployment, the program, and the package dependency are all in the same language. As more Nomia namespaces like those underlying the Environment Manager are implemented, these capabilities will become real.

Nomia is currently being built at Scarf, but it is intended to be completely Scarf-agnostic. We anticipate that Nomia will have its own open source community and development cycle, and its core functionality will be usable without relying on Scarf tools or infrastructure. In fact, Nomia is heavily inspired by Nix and we believe it would be valuable to power the user-level Nix tools and use nixpkgs built on top of Nomia. If accepted by the community, this would give Nix users access to the whole system (including non-package resources!) via tools they already know.

Diving in

Up at the Scarf CLI repo we have pushed an initial prototype of the Environment Manager, which is usable today for basic environment management. Currently, it is built ultimately on top of Nix and nixpkgs, but leverages Nomia concepts for the UX, and is internally implemented on a mock-up of the programmer API to Nomia. The README explains more about how to install and use it.

More details on the design of Nomia are available in the repo, and community conversation is taking place on our Discord. In addition to the README and some initial documentation, the repo includes a paper that gives a deep detailed theoretical presentation of Nomia, including a potential roadmap. On top of the core functionality required by the Environment Manager, we expect to build the fully general system. We are tentatively planning specific functionality to support local developer service management and incremental builds incorporated into package management.

Go check it out! If you just want to learn more, give us feedback, or join in on implementation, it will give you a head start on taking advantage of what we're building. While Scarf has a specific vision with respect to environment management, our roadmap and resources are quite flexible, and we're eager to work with the broader community to identify  the highest value path to making this a reality!

‍