Uncategorized

OSS and Stress

I wanted to take a moment to lay out a journey I didn’t really realize I was on until this morning. Like many big problems it all came out when dealing with a simple little GitHub issue that was raised on a repo I published about 3 years ago.

It turns out that some of the stuff in this repo was out of date, and the repo wasn’t useful anymore. Someone nicely pointed this out to be in an issue, and suggested updating the README to warn others. I said “Sure, happy to accept a PR to do that” to which they closed the issue and didn’t raise a PR. I have to admit that this (not contributing a simple PR to update the Readme) upset me a lot, and I over-reacted in my response.

I then realised 2 things…

1.       My over-reaction wasn’t really about this issue

2.       Am I really expected to maintain YEARS and YEARS of contributions and repositories while doing my day job?

How did I end up here?

Well, it starts slow. You publish a few things while you’re working on a pet project, or helping someone out.  You keep doing that for years and years and years, and then sit back and watch the stress poor in.

In the early days I loved getting issues raised, reviewing PRs and fixing stuff up. I’d get 1 or 2 a month, I didn’t have a family, I was a junior dev, and I loved programming.

Fast forward 8 years, I’m a dad, a senior dev and I still love programming BUT now I get 5 or 6 issues every week. Each takes 30 mins as it requires me to context switch. The PR’s (if they’re big) take hours to review and test. I kidded myself that if I added more automation, CI, integration testing that would help… well it did, but it doesn’t stop the problem . What it’s allowed me to do is stack up more projects and more stuff to look after. More things for people to contribute to and raise issues on.

So today I blew up after waking up to failed integration tests from my day job, 2 other PRs and a backlog of 3 issues that need at least 3 days of my time, and then this little issue that “broke” me. My current day job is already eating into my time with my daughter (2 years old and awesome … don’t want to miss anything), and now the things that I’ve published because I thought they were useful to others are causing me stress. Where do I put this stuff? How do I not let people down?

What am I going to do?

Well this one is harder but I can’t see any other way forward. I need to take time out, go through my projects, and mark stuff as “Dead” or “Finished”. I need to step back from looking after things I no longer use myself, and make it clear to others that PRs and issues are unlikely to get my attention. Where possible, I need to hand over the code to others, or make it clear that the code I’ve contributed needs to find maintenance from elsewhere.

It’ll let people down, it’ll mean stuff people might rely on doesn’t get updated, but at the moment I’m putting their needs ahead of my own and I can’t square the circle. It’ll make me sad to let some people down. I think marking some of my repos as “unmaintained” at least makes it clear to others what level of support they’ll get, but it still makes me sad that there will be people out there that may have taken a dependency on something I’ve no longer got the time (and, in some cases, interest) in maintaining.

In future, I also need to make sure that I’m considering the implicit cost of putting things out into the open. Some contributions are easy, but some have an ongoing cost which racks up and compounds over time with other contributions. Spotting these, and being cautious about them, feels like the way to go.

Lastly I have a new appreciation for those who have contributed WAY more than I have and no idea how they keep up with it all!

Standard
Quick-post

Debugging Cloud-Init on Ubuntu (in Azure or anywhere)

I’ve recently been working with cloud-init in Azure to setup Ubuntu machines and for the most part I’ve really like it as it solves lots of problems and fits my use case BUT debugging it has been a pain so I thought I’d write up some notes here for others.

Did it work?

After deployment SSH onto the node and run this:

cloud-init status --long

Did the stuff I expect get onto the node?

So things didn’t go your way, it looks like it hasn’t behaved or your just curious. Well this lets you see the contents of the ​cloud-initas it landed on your box.

sudo cat /var/lib/cloud/instance/user-data.txt.i

What exactly failed? I want the verbose logs

So the script didn’t run or something else failed and you’ve got some logging that writes to console in there, never fear this will show you what happened.

sudo cat /var/log/cloud-init-output.log

I’d like to know early if things are broken, can I validate these things on my dev machine?

Yup, this will pick up some errors but be warned the validation is limited.

  1. Get the tooling (use docker run -it  ubuntu if not on ubuntu): `sudo apt install cloud-init`
  2. Run the validation: cloud-init devel schema --config-file your-cloud-init.txt
  3. Profit

 

Note

I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard
Coding, Quick-post

Docker and Healthchecks outside of Kubernetes

So I’ve been working with a containerized solution recently which runs outside of Kuberenetes using an Azure VMSS to scale out. I won’t dive into the reasons why we went down this route but one really interesting thing came of out of it.

How do you automatically healthcheck a container outside of Kubernetes?

Well it turns out docker has this covered in newer versions. You can specify a HEALTHCHECK inside the docker file to monitor the containers state

How do you ensure it restarts when unhealthy?

Well here you have a couple of options but both rely on using --restart=always when starting the container:

  1. You `healthcheck` command runs inside the container so you can have it kill the root process of the container causing the container to restart – Example: https://github.com/opencb/opencga/pull/1121/files
  2. You can use `AutoHeal` container which monitors the docker deamon via it’s socket and handles and containers which report unhealthy https://hub.docker.com/r/willfarrell/autoheal/

Note: I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard