Golang, Containers and private repos

May 13, 2021

A smörgåsbord of guidance involving Golang modules, private repos and containers. Everything herein is documented elsewhere (I’ll provide links) but I wanted to consolidate the information primarily for my own benefit.

GOPRIVATE

Using private modules adds complexity because builders need to be able to access private modules. Customarily, as you’re hacking away, you’ll likely not encounter issues but, when you write a Dockerfile or develop some CI, you’ll encounter something of the form:

go mod tidy
go: github.com/[email protected]: reading github.com/.../go.mod at revision v0.0.1: unknown revision v0.0.1

And perhaps be puzzled why. The reason is that go mod tidy tries to go get dependencies defines in your go.mod using a public module mirror (or proxy) and, if any of these is a private module, it will be unable to find them.

You can prove this to yourself by attempting to lookup the module defined in the private repo in the proxy:

curl https://sum.golang.org/lookup/github.com/[email protected]
not found: github.com/[email protected]: invalid version: unknown revision v0.0.1

Whereas well known artifacts may be found:

curl https://sum.golang.org/lookup/github.com/kubernetes/[email protected]
4265325
github.com/kubernetes/kubernetes v1.21.1 h1:bYy11b1u4wITl8ybHnIUKI1EBnVLacIoMyxieYjc92M=
github.com/kubernetes/kubernetes v1.21.1/go.mod h1:ef++isEL1PW0taH6z7DXrSztPglrZ7jQhyvcMEtm0gQ=

go.sum database tree
4302955
FVjVn56yZtRVoLVLy4ITpc6qq4VgUmDVpv4q33Dst+M=

— sum.golang.org Az3grq3tBuxwQ3U0j1ji4FRq/JG6NP+8t5stHF0iCajt+RGhvDzMEl4QY1FlQ6neNhbaxMlv0bdJHVrE79Uu/GVsWww=

The solution is simple, tell the go tools to not use the proxy for your private repos, e.g.:

GOPRIVATE=github.com/my-private-repo go mod tidy

NOTE it may be better to set-it-and-forget-it once in your shell but, the problem then is that you may forget it and why it’s needed.

Containers

That solves the problem if the repos are accessible but, often when you’re building containers, you won’t (shouldn’t) be using your regular user credentials:

Assuming we’re taking a copy of the current working directory and trying to go mod tidy it:

docker run \
--interactive --tty --rm \
--volume=${PWD}:/app \
golang:1.16.4

Then:

GOPRIVATE=github.com/... go mod tidy
go: github.com/[email protected]: reading github.com/.../go.mod at revision v0.0.1: unknown revision v0.0.1

The same error!

This time it’s because the container is unable to access the repo. The solution appears in several forms but I’m using:

git config \
--global url."https://${TOKEN}@github.com".insteadOf "https://github.com"

What’s the value of ${TOKEN}? It’s a GitHub personal access token (PAT). Because I’m using GitHub Container Registry which also needs a PAT. I’m sharing the token between GitHub Actions that need to get repos, build and push to GHCR and Dockerfiles that need to get repos.

Many GitHub Actions workflows, include the git config command as a precursor step to enable e.g. subsequent git clone ....

I prefer to create platform-independent builds using containers and so I include the git config in Dockerfiles and then pass the token to the docker build --build-arg=TOKEN=${TOKEN}

Something of the form:

ARG GOLANG_VERSION="1.16.4"
ARG GITHUB_TOKEN

FROM golang:${GOLANG_VERSION} as build

ARG GITHUB_TOKEN

WORKDIR /${PROJECT}

COPY go.mod go.mod
COPY go.sum go.sum

RUN git config \
    --global url."https://${GITHUB_TOKEN}@github.com".insteadOf "https://github.com"

ENV GOPRIVATE=github.com/...
RUN go mod download

...
```

### `go list`

A handy command that I've likely written about elsewhere is `go list`. Specifically:

```bash
go list -f '{{.Dir}}' github.com/${REPO}
/home/username/go/pkg/mod/github.com/brabantcourt/${REPO}@v0.0.X
/home/username/go/pkg/mod/github.com/brabantcourt/${REPO}@v0.0.Y
/home/username/go/pkg/mod/github.com/brabantcourt/${REPO}@v0.0.Z
```

> **NOTE** This only lists modules included in the current project's `go.mod`.