On complexity



We’re doing routine package upgrades on our environment. I’ve realised something might be helpful: Environment-specific hooks! It would be like pre/post install scripts of the packages, but for more general purpose. Are you upgrading postgresql package? It will restart the database service, so let’s just pause pgbouncer before the operation and resume it afterwards. Are you doing php-fpm upgrade? Make sure that you wipe some caches, so random issues don’t surprise you, etc.
In theory: You define your rules somewhere, something like “if package X is upgrading on server with role Y, run this command(s) on server(s) with role Z”.
Then feature requests and exceptions started to come: What if I want to skip upgrading a package on specific conditions? What happens if a command fails on pre-hook? What if the package upgrade itself fails? It’s slow, can we do some actions in parallel?

Since I am not a full-time developer, the quality of my code is questionable at best. So, as you can imagine, it turned out a bit messy with all the additions. I needed to do some multiprocessing, caching, rate limiting, shit like “loop in a loop” etc. But it’s working for a long time without much interception (or complaints) so I count this one successful.

Recently touched to it to add few small things, triggered the pile of thoughts I had about complexity:

  • Documentation is a pipe dream. I’m experienced enough to understand at least that. It’s nice to have, amazing to be still valid, but impossible to maintain globally. With enough moving parts, a project starts to live in one (or a few) individual’s brain only (a.k.a bus factor).
  • Shit code is definitely unavoidable, as long as “shit” is subjective. But what makes me happy is the code with small comments, which answers “but why the hell we’re doing this thing now?” question on my mind.
  • Recently had an incident, which can be explained by Law of Hyrum: One system having an upgrade, “fixing” the issue of invalid paths (like the ones which include double slashes) to be accepted and starts returning 404. You can guess what happens next.
  • The second law of thermodynamics is kinda valid for everything, including projects. Things tend to get more complex and disorganized by time. Edge cases pile up. Check out # TODO: Temporary until .. comments in random projects, which was added by a developer who already left the company few years ago. Generally those are the Schrödinger’s lines; you can’t understand the effects fully, until you touch them.
  • If complexity is unavoidable, it should be smashed into pure functions as much as possible. This way we get easily explainable and readable parts which interact with each other with defined input/outputs. (Did I accomplish this? Not yet 🙄. But that is the dream.)
  • Exceptions/errors happen. But if someone needs to dive into your code to understand why something is failing, that shit is broken. Most of the time I am doing semi-unnecessary debug outputs like “now fetching the data from ${remote_system}”, but I believe this would help the person who got an error while running my code: “Ah, let me check if ${remote system} is up!”, if I fail to catch it.
  • Concepts of issues are same everywhere. But they have levels which shows how primitive an organization is. You are having capacity issues? Let it be “not enough cpu power” or disk alerts, you’re just starting your amazing journey. Please reach me when you have race conditions or at least unexplainable exceptions.

All of this causes me to accept the real world, and more specifically reject most job interviews after reading/talking some details about them. Some people strive for simplicity, some thinks adding one more element will make the system perfect.
No thanks, I am becoming a zealot of former.



633 Words

2022-01-30 17:24 +0100