A few months ago, I wrote and delivered a Kubernetes orchestration workshop, based on my Swarm orchestration workshop. While doing so, I hit a few snags; and since I’m attending KubeCon this week, I thought this would be the perfect occasion to track down Santa Kube and give them my wishlist for Christmas! 🎄🎅🏿❤️
As a foreword: I don’t consider myself a Kube expert, and while I did a bit of research, I might have missed some obvious workarounds. If that is the case, I trust my readers to let me know, and I will be happy to update this post and give them all the credit (as well as a chunk of my eternal gratitude!)
kubectl logs
The kubectl logs command lets you stream logs from a single container,
and it lets you retrieve logs from multiple pods using a selector. However,
it doesn’t let you combine both:
# This is okay
kubectl logs --follow my-little-pod
# This is okay too
kubectl logs --selector run=deploy-oh-my-deploy
# But not this
kubectl logs --selector run=dpeloy-oh-my-deploy --follow
I’ve been spoiled by the Docker CLI, which lets me stream the logs of multiple replicas at the same time, and will prefix each line with the source of the log line:
I would be out-of-this-world-ly delighted if I could get the convenience of multi-container log streaming, together with the power of Kubernetes selectors.
Knowing which log line comes from which container in which pod would be fantastic, too. 🌈
I wrote my own bashery that basically repeatedly fetches logs
using --since and then does something ugly to remove duplicate lines,
but then I realized that this was so wrong on so many levels and stopped
before that code would spring to life and try to devour my face.
I have tried kail, but the installation was
far from painless (why can’t I just go get a Go program? 😭) and it
doesn’t play nice with white terminals (at first, I thought it was buggy
because a lot of text was in white on a white background).
And then there is stern which looks pretty dope too. Update: I tried it! Thanks @lestrrat for encouraging me to try it by the way.
Using stern to view logs is painless.
- Download binary from GitHub.
chmod +xthe freshly downloaded binary../stern_linux_amd64 pod_or_deployment_or_whatever_name --tail 1 -t.- Profit!
I just wish this were embedded in the kubectl CLI. I would also get
much joy and happiness from a thing that could be installed with just go get
(without messing with additional dependency management tools)_ since I could
then install it without installing Go!
kubeadm token
When writing the provisioning scripts for my Kubernetes training clusters,
I used kubeadm. It works great, except for one little detail: extracting
the token generated by kubeadm init is way too complicated.
I’m pretty sure that somebody will point out that I missed an obvious option (because for Christmas’ sake, it HAS TO be there!), but I looked high and low and couldn’t find it. Show me how! And I will then be able to deprecate the pretty horrible things that I’m currently doing to retrieve that token in a stable-ish way. (I didn’t feel dirty enough to parse the porcelain-ish output of the command!)
For reference, the equivalent command on Docker is docker swarm join-token -q worker
(it outputs the join token for the cluster, without further decoration).
Stray pods
Selectors and labels are an amazing combo (like peanut butter and jelly). They allow you, for instance, to remove a faulty pod from a load balancer, just by changing its labels, in such a way that the pod is no longer “selected” by its load balancer; but the pod still exists and you can examine it at leisure.
Even better: if that pod is managed by e.g. a replica set, you can change its labels so that the pod is set aside, and the replica set creates a replacement pod immediately.
However! If you do that a lot, you end up with a bunch of stray pods. (Or orphan pods? I don’t know which word would be more appropriate there.)
For years and years, people have complained that Docker would eat up all
their disk space. For years and years, Docker’s answer was, “just craft up
some shell script to remove old containers and images, then drop that in a
crontab”; but we had to wait
for Docker 1.13 to introduce docker system prune to provide a better answer.
I’m still getting familiar with the Kubernetes data model. I haven’t figured
out if each resource created by another resource has some kind of “parent”
link, that would help to track who-created-who and whether the parent resource
is still watching after their child or if the latter is now outside of the
scope of the former’s selector. So I don’t know if this is as simple as
cobbling together 100 lines of script; but I think we shouldn’t wait 3
years for some kubectl prune pods command to show up!
(Or perhaps I’m completely missing the point, and there is a better way to handle these situations? I’m always happy to be enlightened. ⚡🙇🏻♂️ )
Figure out port numbers when they are in image metadata
Docker images have some meta data indicating which ports are exposed, but for some reason, Kubernetes can’t or won’t use that.
For instance, if I do:
kubectl run nginx-is-love --image=nginx
kubectl expose nginx
… kubectl will tell me that I need to specify the port number
for my NGINX deployment. Which is weird, because that port number
definitely is in the image! Proof is, if I do docker run -dP nginx
it gets exposed automatically.
I suppose that this piece of metadata is lost somewhere in translation, but I don’t know where.
Support recent versions of Docker
Kubernetes officially supports Docker 1.12, 1.13, and 17.03. Anything after 17.03 is not supported. Which is kind of sad, because Docker 17.03-ce has been EOL for almost 6 months now. 🤷
Why does Kubernetes use such an old version of Docker? Because in the early days of Docker, the API had to evolve quickly, and breaking changes happened regularly. Building on top of a fast-moving API is hard, and this prompted two things: freezing the version of Docker used in most Kubernetes deployments, and the development of the CRI interface, to support other container engines like rkt, CRI-O, and containerd.
Interestingly, the very same thing happened for Docker itself. It initially relied on LXC for container execution; and when LXC started to evolve too quickly (and when it became impossible to support all the different versions of LXC in existence out there), Docker introduced libcontainer and promoted it as the primary execution engine.
There is a big difference in the two situations, though: LXC
doesn’t have an API. LXC is leveraged by invoking lxc-start,
so the API contract is … lxc-start’s manpage. On the other hand,
Docker exposes a versioned API, and has been offering backward
compatibility on that API for a while. Breakages are considered
release critical bugs. In other words: if you use the versioned API
(i.e. /v1.31/containers instead of /containers) and stick to a
given API number, but get a different behavior on a newer version
of the Docker Engine, you can report that bug and it will be treated
as a release critical bug. That means that if you’re using Docker CE,
you can track the latest version (and get bug fixes and new features)
with the guarantee that if something breaks, people will look at it.
I’m glad that the CRI exists, and allows to have multiple options for the container engine today. And it’s great that in the context of Kubernetes, picking one over the other won’t (or at least, shouldn’t) have any impact on the user. But I’m also glad that there are initiatives to support the Docker API in a more stable way.
By the way, I’m fully aware that this one will probably require collaboration from my coworkers as well. If you’re working on this but need some contacts at Docker to move forward, I’ll be more than happy to make intros and beg, bribe, seduce, or threaten my loved coworkers so that everybody wins at the end!
And while we’re here, I’m also going to acknowledge the pretty cool stuff that Virtual Kubelet is doing, and that moves the API border at a different level.
Final words
The real conclusion of this post is that quite frankly, nothing significant got in my way when I learned how to do with Kubernetes the stuff I had been doing with Docker since June 2015. I’m sure that you’ll agree that none of the points mentioned above is a showstopper to anyone wondering if they can apply their Docker knowledge (and Swarm in particular) to Kubernetes.
And of course, I’m well aware that Swarm has its shortcomings as well. This post is, by no mean, advocating for or against either orchestrator.
Thanks; and if you want to chat more about all this, please reach out, I’ll be thankful to learn from you!