Dependency Management in Go
We love Go. We could spend a whole series on why we love Go (and we might in the future) but for now I want to talk about something that we don’t like: dependency management.
The current state of dependency management in Go is problematic at best. You have a number of competing dependency managers — glide and dep are the top two — all fighting for the same space in the community, but none of them are really a complete solution. Some support proxies, some don’t; some support multi-depth repository paths, others don’t; just to name a few issues. Its safe to say that Robert Griesemer, Rob Pike, and Ken Thompson made a huge mistake in not releasing an official and proper Go package manager with Go’s initial release. In that void was left the chaos we have now.
When I first started developing in Go, like most people I think, I simply used go get to install dependencies. This worked for a short while until I ran into issues of conflicting package requirements and everything else that goes with having a single global package store, and in fairness go get was never really designed to take on that role in the first place. Its strength is in installing tools and apps that have been developed in Go, not pulling in libraries to develop Go apps. I asked a tool to do something it wasn’t good at and got frustrated when it didn’t work.
So then I moved on to Glide, but Glide came with its own issues. When I started working on projects hosted in Gitlab those issues came to the surface in a very fast “everything is broken” sort of way. Glide only supports “top-level” repositories like you see in Github, where repo URL’s are github.com/user/repo. Gitlab, on the other hand, allows you to organize code into “projects” which adds in extra hops in that repo path. Trying to import gitlab.com/org/project/repo causes Glide to ignore that last /repo location, throwing an error because it can’t find any valid Go code. Because of this, I migrated all of my projects over to Dep.
And Dep worked well — very well — up until yesterday where its true weakness was revealed: dependency chaining. I had on my hands a small-scale left-pad event, where a single dependency of the package I was trying to import was broken, leaving me with un-compileable code. Not only did I have to wait for the package maintainers to cut a new release tag, but I also had to wait for the package dependency to update as well. Because of this we’ve had to resort to the least-efficient method of dependency management: pull once and push your dependencies to git with your codebase. In Go’s case, we’re now committing our vendor directory.
So how would we fix Go’s dependency management issues? I’m glad you asked.
I would love to implement a package repository similar to pypi or npm. A central, single, authoritative repository for Go packages that has automated tools for package creators to publish and test their code. Not only would this allow us to create a true package manager that does dependency resolution on its own but it would centralize packages and their documentation. Godoc is great, but there’s room for improvement. This comes with the added benefit of being able to set up private code repositories in places other than a git-based service like Github or Gitlab. By basing the repository on standard open-source software — Apache’s httpd for example — it would allow companies to create internal repos without having to license Github Enterprise or stand up a Gitlab server, integrating with their existing Artifactory or other repositories.
One of these days we’ll get the kinks ironed out. Whether its with what exists already or with a new solution, we’ll get there. We’re not there yet, but we’re working on it.
