Coding, How to, kubernetes

How to build Kubernetes from source and test in Kind with VSCode & devcontainers

I found myself last week looking at a bit of code in K8s which I thought I could make better, so I set about trying to understand how to clone, change and test it.

Luckily K8s has some good docs, trust these over me as they’re a great guide. This blog is more of a brain dump of how I got on trying with Devcontainers and VSCode. This is my first try at this so I’ve likely got lots of things wrong.

Roughly I knew what I needed as I’d heard about:

  1. Bazel for the Kubernetes build
  2. Kind to run a cluster locally
  3. Kubetest for running end2end tests

As the K8s build and testing cycle can use up quite a bit of machine power I didn’t want be doing all this on my laptop and ideally I wanted to capture all the setup in a nice repeatable way.

Enter DevContainers for VSCode, I’ve got them setup on my laptop to actually run on a meaty server for me and I can use them to capture all the setup requirements for building K8s.

After a lot of wrangling I got a Dockerfile and devcontainer.json file together which mount and install the required bits (warning: There may be lots wrong with this, I’m pretty new to the world of building K8s from source).

Now all I need to do is clone K8s, drop these two files into a .devcontainer folder and open the folder in VSCode. Once open you get prompted to open the folder in the devcontainer and away you go.

What do you do now?

  • Make the change you want to make
  • Build an image for kind with your changes:
kind build node-image --type bazel \
 -v 5 --image kindest/node:lawrence
  • Start a cluster based off that image:
kind create cluster --name lg \
--image kindest/node:lawrence
  • You can run the e2e tests too (still getting to grips with this)
kubetest --up --test --down \
--deployment=kind \
--kind-node-image=kindest/node:lawrence \
--provider=skeleton

The best part? All of this runs nicely in VSCode like its running locally but behind the scenes it’s on a remote server. The left window is my laptop and the right is the server doing a build.

Let me try it

Here is a Gist of my devcontainer files for you to play with. If you spot something silly I’m doing please let me know!

// For format details, see https://aka.ms/vscode-remote/devcontainer.json or the definition README at
// https://github.com/microsoft/vscode-dev-containers/tree/master/containers/go
{
"name": "Go",
"dockerFile": "Dockerfile",
"runArgs": [
// Uncomment the next line to use a non-root user. On Linux, this will prevent
// new files getting created as root, but you may need to update the USER_UID
// and USER_GID in .devcontainer/Dockerfile to match your user if not 1000.
// "-u", "vscode",
"–cap-add=SYS_PTRACE",
"–security-opt",
"seccomp=unconfined",
"–name", "kubernetesdevcontainer",
// Mount go mod cache
"-v", "k8s-gomodcache:/go/pkg",
"-v", "k8s-bazelcache:/root/.cache/bazel/",
// Cache vscode exentsions installs and homedir
"-v", "k8s-vscodecache:/root/.vscode-server",
// Keep command history
"-v", "k8s-bashhistory:/root/commandhistory",
// Mount docker socket for docker builds
"-v", "/var/run/docker.sock:/var/run/docker.sock",
// Use host network
"–network=host",
// Mount ssh
"-v", "${env:HOME}${env:USERPROFILE}/.ssh:/root/.ssh",
"-v", "/tmp:/tmp",
// Mount azure, git and docker config
"-v", "${env:HOME}${env:USERPROFILE}/.azure:/root/.azure",
"-v", "${env:HOME}${env:USERPROFILE}/.gitconfig:/root/.gitconfig", // Required due to volume mount used for .vscode-server
],
"workspaceFolder": "/go/src/k8s.io/kubernetes",
"workspaceMount": "src=${localWorkspaceFolder},dst=/go/src/k8s.io/kubernetes,type=bind",
// Use 'settings' to set *default* container specific settings.json values on container create.
// You can edit these settings after create using File > Preferences > Settings > Remote.
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"go.gopath": "/go",
"go.useLanguageServer": true,
"[go]": {
"editor.snippetSuggestions": "none",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": true,
}
},
"gopls": {
"usePlaceholders": true, // add parameter placeholders when completing a function
// Experimental settings
"completeUnimported": true, // autocomplete unimported packages
"watchFileChanges": true, // watch file changes outside of the editor
"deepCompletion": true, // enable deep completion
},
"files.eol": "\n", // formatting only supports LF line endings
},
// Uncomment the next line if you want to publish any ports.
// "appPort": [],
// Uncomment the next line to run commands after the container is created.
// "postCreateCommand": "go version",
// Add the IDs of extensions you want installed when the container is created in the array below.
"extensions": [
"ms-vscode.go"
]
}
view raw 1_devcontainer.json hosted with ❤ by GitHub
FROM golang:1.13-stretch
# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
# This Dockerfile adds a non-root 'vscode' user with sudo access. However, for Linux,
# this user's GID/UID must match your local user UID/GID to avoid permission issues
# with bind mounts. Update USER_UID / USER_GID if yours is not 1000. See
# https://aka.ms/vscode-remote/containers/non-root-user for details.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID
ENV GO111MODULE=on
# Configure apt, install packages and tools
RUN apt-get update \
&& apt-get -y install –no-install-recommends apt-utils dialog rsync \
# bazel stuff
&& apt-get -y install openjdk-8-jdk pkg-config zip g++ zlib1g-dev unzip patch \
#
# Verify git, process tools, lsb-release (common in install instructions for CLIs) installed
&& apt-get -y install git iproute2 procps lsb-release \
#
# Install Azure CLI
&& curl -sL https://aka.ms/InstallAzureCLIDeb | bash \
#
# Clean up
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*
ENV GIT_PROMPT_START='\033[1;36mkube-devcon>\033[0m\033[0;33m\w\a\033[0m'
# Save command line history
RUN echo "export HISTFILE=/root/commandhistory/.bash_history" >> "/root/.bashrc" \
&& echo "export PROMPT_COMMAND='history -a'" >> "/root/.bashrc" \
&& mkdir -p /root/commandhistory \
&& touch /root/commandhistory/.bash_history
# Git command prompt
RUN git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt –depth=1 \
&& echo "if [ -f \"$HOME/.bash-git-prompt/gitprompt.sh\" ]; then GIT_PROMPT_ONLY_IN_REPO=1 && source $HOME/.bash-git-prompt/gitprompt.sh; fi" >> "/root/.bashrc"
ENV DEVCONTAINER="TRUE"
# Install docker used by go releaser
RUN bash -c "cd /tmp && curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-19.03.5.tgz && tar –strip-components=1 -xvzf docker-19.03.5.tgz -C /usr/local/bin"
# Install Go tools
RUN \
# –> Delve for debugging
go get github.com/go-delve/delve/cmd/dlv@v1.3.2 \
# –> Go language server
&& go get golang.org/x/tools/gopls@v0.3.2 \
# –> Go symbols and outline for go to symbol support and test support
&& go get github.com/acroca/go-symbols@v0.1.1 && go get github.com/ramya-rao-a/go-outline@7182a932836a71948db4a81991a494751eccfe77 \
&& rm -rf /go/src/ && rm -rf /go/pkg
# Install K8s stuff we need
RUN \
# Build and install kubetest from test infra
git clone https://github.com/kubernetes/test-infra.git && cd test-infra && GO111MODULE=on go install ./kubetest \
# Install kind
&& curl -Lo ./kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64" \
&& chmod +x ./kind \
&& mv ./kind /usr/local/bin/kind \
# Install bazel
&& wget -O bazelinstaller.sh https://github.com/bazelbuild/bazel/releases/download/2.1.0/bazel-2.1.0-installer-linux-x86_64.sh \
&& chmod +x bazelinstaller.sh \
&& ./bazelinstaller.sh \
# Install required bazel version (may change)
&& curl -LO https://releases.bazel.build/0.23.2/release/bazel-0.23.2-linux-x86_64 && chmod +x bazel-0.23.2-linux-x86_64 && cp bazel-0.23.2-linux-x86_64 /usr/local/lib/bazel/bin \
# Install kubectl
&& curl -sSL -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl \
&& chmod +x /usr/local/bin/kubectl
# Configure bash
RUN echo "alias kindbuild='kind build node-image –type bazel -v 5 –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD) && kind create cluster –name lg –image lawrencegripper/node:\$(git rev-list –max-count=1 HEAD)'" >> ~/.bashrc
RUN echo "source /etc/bash_completion" >> "/root/.bashrc"
RUN echo "alias k=kubectl" >> "/root/.bashrc"
RUN echo "source <(kubectl completion bash)" >> "/root/.bashrc"
RUN echo "source <(kubectl completion bash | sed 's/kubectl/k/g')" >> "/root/.bashrc"
view raw 2_Dockerfile hosted with ❤ by GitHub
Current devcontainer setup (WIP)
Standard

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s