I found myself last week looking at a bit of code in K8s which I thought I could make better, so I set about trying to understand how to clone, change and test it.
Luckily K8s has some good docs, trust these over me as they’re a great guide. This blog is more of a brain dump of how I got on trying with Devcontainers and VSCode. This is my first try at this so I’ve likely got lots of things wrong.
Roughly I knew what I needed as I’d heard about:
As the K8s build and testing cycle can use up quite a bit of machine power I didn’t want be doing all this on my laptop and ideally I wanted to capture all the setup in a nice repeatable way.
Enter DevContainers for VSCode, I’ve got them setup on my laptop to actually run on a meaty server for me and I can use them to capture all the setup requirements for building K8s.
After a lot of wrangling I got a Dockerfile
and devcontainer.json
file together which mount and install the required bits (warning: There may be lots wrong with this, I’m pretty new to the world of building K8s from source).
Now all I need to do is clone K8s, drop these two files into a .devcontainer
folder and open the folder in VSCode. Once open you get prompted to open the folder in the devcontainer and away you go.
What do you do now?
- Make the change you want to make
- Build an image for kind with your changes:
kind build node-image --type bazel \
-v 5 --image kindest/node:lawrence
- Start a cluster based off that image:
kind create cluster --name lg \
--image kindest/node:lawrence
- You can run the e2e tests too (still getting to grips with this)
kubetest --up --test --down \
--deployment=kind \
--kind-node-image=kindest/node:lawrence \
--provider=skeleton
The best part? All of this runs nicely in VSCode like its running locally but behind the scenes it’s on a remote server. The left window is my laptop and the right is the server doing a build.

Let me try it
// For format details, see https://aka.ms/vscode-remote/devcontainer.json or the definition README at | |
// https://github.com/microsoft/vscode-dev-containers/tree/master/containers/go | |
{ | |
"name": "Go", | |
"dockerFile": "Dockerfile", | |
"runArgs": [ | |
// Uncomment the next line to use a non-root user. On Linux, this will prevent | |
// new files getting created as root, but you may need to update the USER_UID | |
// and USER_GID in .devcontainer/Dockerfile to match your user if not 1000. | |
// "-u", "vscode", | |
"–cap-add=SYS_PTRACE", | |
"–security-opt", | |
"seccomp=unconfined", | |
"–name", "kubernetesdevcontainer", | |
// Mount go mod cache | |
"-v", "k8s-gomodcache:/go/pkg", | |
"-v", "k8s-bazelcache:/root/.cache/bazel/", | |
// Cache vscode exentsions installs and homedir | |
"-v", "k8s-vscodecache:/root/.vscode-server", | |
// Keep command history | |
"-v", "k8s-bashhistory:/root/commandhistory", | |
// Mount docker socket for docker builds | |
"-v", "/var/run/docker.sock:/var/run/docker.sock", | |
// Use host network | |
"–network=host", | |
// Mount ssh | |
"-v", "${env:HOME}${env:USERPROFILE}/.ssh:/root/.ssh", | |
"-v", "/tmp:/tmp", | |
// Mount azure, git and docker config | |
"-v", "${env:HOME}${env:USERPROFILE}/.azure:/root/.azure", | |
"-v", "${env:HOME}${env:USERPROFILE}/.gitconfig:/root/.gitconfig", // Required due to volume mount used for .vscode-server | |
], | |
"workspaceFolder": "/go/src/k8s.io/kubernetes", | |
"workspaceMount": "src=${localWorkspaceFolder},dst=/go/src/k8s.io/kubernetes,type=bind", | |
// Use 'settings' to set *default* container specific settings.json values on container create. | |
// You can edit these settings after create using File > Preferences > Settings > Remote. | |
"settings": { | |
"terminal.integrated.shell.linux": "/bin/bash", | |
"go.gopath": "/go", | |
"go.useLanguageServer": true, | |
"[go]": { | |
"editor.snippetSuggestions": "none", | |
"editor.formatOnSave": true, | |
"editor.codeActionsOnSave": { | |
"source.organizeImports": true, | |
} | |
}, | |
"gopls": { | |
"usePlaceholders": true, // add parameter placeholders when completing a function | |
// Experimental settings | |
"completeUnimported": true, // autocomplete unimported packages | |
"watchFileChanges": true, // watch file changes outside of the editor | |
"deepCompletion": true, // enable deep completion | |
}, | |
"files.eol": "\n", // formatting only supports LF line endings | |
}, | |
// Uncomment the next line if you want to publish any ports. | |
// "appPort": [], | |
// Uncomment the next line to run commands after the container is created. | |
// "postCreateCommand": "go version", | |
// Add the IDs of extensions you want installed when the container is created in the array below. | |
"extensions": [ | |
"ms-vscode.go" | |
] | |
} |
FROM golang:1.13-stretch | |
# Avoid warnings by switching to noninteractive | |
ENV DEBIAN_FRONTEND=noninteractive | |
# This Dockerfile adds a non-root 'vscode' user with sudo access. However, for Linux, | |
# this user's GID/UID must match your local user UID/GID to avoid permission issues | |
# with bind mounts. Update USER_UID / USER_GID if yours is not 1000. See | |
# https://aka.ms/vscode-remote/containers/non-root-user for details. | |
ARG USERNAME=vscode | |
ARG USER_UID=1000 | |
ARG USER_GID=$USER_UID | |
ENV GO111MODULE=on | |
# Configure apt, install packages and tools | |
RUN apt-get update \ | |
&& apt-get -y install –no-install-recommends apt-utils dialog rsync \ | |
# bazel stuff | |
&& apt-get -y install openjdk-8-jdk pkg-config zip g++ zlib1g-dev unzip patch \ | |
# | |
# Verify git, process tools, lsb-release (common in install instructions for CLIs) installed | |
&& apt-get -y install git iproute2 procps lsb-release \ | |
# | |
# Install Azure CLI | |
&& curl -sL https://aka.ms/InstallAzureCLIDeb | bash \ | |
# | |
# Clean up | |
&& apt-get autoremove -y \ | |
&& apt-get clean -y \ | |
&& rm -rf /var/lib/apt/lists/* | |
ENV GIT_PROMPT_START='\033[1;36mkube-devcon>\033[0m\033[0;33m\w\a\033[0m' | |
# Save command line history | |
RUN echo "export HISTFILE=/root/commandhistory/.bash_history" >> "/root/.bashrc" \ | |
&& echo "export PROMPT_COMMAND='history -a'" >> "/root/.bashrc" \ | |
&& mkdir -p /root/commandhistory \ | |
&& touch /root/commandhistory/.bash_history | |
# Git command prompt | |
RUN git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt –depth=1 \ | |
&& echo "if [ -f \"$HOME/.bash-git-prompt/gitprompt.sh\" ]; then GIT_PROMPT_ONLY_IN_REPO=1 && source $HOME/.bash-git-prompt/gitprompt.sh; fi" >> "/root/.bashrc" | |
ENV DEVCONTAINER="TRUE" | |
# Install docker used by go releaser | |
RUN bash -c "cd /tmp && curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-19.03.5.tgz && tar –strip-components=1 -xvzf docker-19.03.5.tgz -C /usr/local/bin" | |
# Install Go tools | |
RUN \ | |
# –> Delve for debugging | |
go get github.com/go-delve/delve/cmd/dlv@v1.3.2 \ | |
# –> Go language server | |
&& go get golang.org/x/tools/gopls@v0.3.2 \ | |
# –> Go symbols and outline for go to symbol support and test support | |
&& go get github.com/acroca/go-symbols@v0.1.1 && go get github.com/ramya-rao-a/go-outline@7182a932836a71948db4a81991a494751eccfe77 \ | |
&& rm -rf /go/src/ && rm -rf /go/pkg | |
# Install K8s stuff we need | |
RUN \ | |
# Build and install kubetest from test infra | |
git clone https://github.com/kubernetes/test-infra.git && cd test-infra && GO111MODULE=on go install ./kubetest \ | |
# Install kind | |
&& curl -Lo ./kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64" \ | |
&& chmod +x ./kind \ | |
&& mv ./kind /usr/local/bin/kind \ | |
# Install bazel | |
&& wget -O bazelinstaller.sh https://github.com/bazelbuild/bazel/releases/download/2.1.0/bazel-2.1.0-installer-linux-x86_64.sh \ | |
&& chmod +x bazelinstaller.sh \ | |
&& ./bazelinstaller.sh \ | |
# Install required bazel version (may change) | |
&& curl -LO https://releases.bazel.build/0.23.2/release/bazel-0.23.2-linux-x86_64 && chmod +x bazel-0.23.2-linux-x86_64 && cp bazel-0.23.2-linux-x86_64 /usr/local/lib/bazel/bin \ | |
# Install kubectl | |
&& curl -sSL -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl \ | |
&& chmod +x /usr/local/bin/kubectl | |
# Configure bash | |
RUN echo "alias kindbuild='kind build node-image –type bazel -v 5 –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD) && kind create cluster –name lg –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD)'" >> ~/.bashrc | |
RUN echo "source /etc/bash_completion" >> "/root/.bashrc" | |
RUN echo "alias k=kubectl" >> "/root/.bashrc" | |
RUN echo "source <(kubectl completion bash)" >> "/root/.bashrc" | |
RUN echo "source <(kubectl completion bash | sed 's/kubectl/k/g')" >> "/root/.bashrc" |