Skip to content

Entry #2 - August 28th, 2021

Hey, Journal:

So, second entry, huh? Who would have thought? This is as far as I've ever gotten in a journal...

The last three days I've mostly worked on pydock, which is my response to that concern I raised in the previous entry about the myriad of suboptimal solution for managing Python environments. My take is the N+1 suboptimal solution, but hey, at least it's mine, right?

The idea is dead simple, so much so that I'm intrigued no one has tried it before (or maybe they did try but didn't survive to tell the story). pydock is a wrapper around Docker, that tries to give the impression of an environment manager, but under the hood is actually just creating Docker images and containers. The nicest part is that every "environment" is just a couple of files (a dockerfile and a requirements.txt), out of which the whole environment can be deterministically rebuilt (because packages in requirements.txt are always pinned to exact versions). Anyway, I've been coding this thing for a couple days, on and off while doing other stuff, and it's reaching a point where I think I'll stick to it.

This whole endeavour reminded me of the discussion about reinventing the wheel. We all know that's bad. Everytime you have to solve a problem that someone else already solved, you're putting yourself in the position of making the same mistakes they did and thus wasting a lot of precious time. But there are obvious advantages to reinventing wheels once in a while.

For starters, it's very pedagogical. By redoing something that's well known, we give ourselves a chance to hone our skills in a safe environment, knowing that if we get stuck, there's plenty of guidance lying around. It's no wonder most education is about reinventing a bunch of wheels just to learn how they work. But that's not all. Retrying something that's apparently well solved might show us that, in reality, it wasn't that well solved after all.

The thing is, times change, technology improves, problems mutate, and we are often stuck with solutions that were optimal for a previous version of reality, which is close, but not exactly like the one we're living. Since reinventing the wheel is such a tabu in most of engineering and science, we resort to incremental improvement. That is, we try to adjust the suboptimal solution with the minimal effort that makes it optimal again, now that the landscape has changed. It's like optimizing a slowly but ever-changing function by gradient descent with near infinitesimal steps: you're always converging to a local optimum that can be arbitrarily worse than the global optimum.

So, from time to time, it's good to restart the search from scratch. Just take a look around and ask: "What is a problem in my field that hasn't been solved from scratch in a while?" And then try to solve that problem without thinking in terms of the previous solution. Just think again from first principles, and apply everything that the previous guy who solved it didn't have at hand. You'll probably end in a completely different region of the solution space.

I'm not saying this is how all, or even most of innovation and science should be done. On the contrary, standing on the should of giants, doing incremental improvement, is a pretty solid way to move forward. But maybe 5% of the times looking at an old problem with a blank slate can be just what you need to do to come up with a beautifully innovative solution.

Anyway, I started at Python environment managers and went down the rabbit hole of innovation and reinventing the wheel. I guess that's my way of saying: "I think I'll reinvent this wheel and see how it goes." For the record, I'm not attempting to solve environment management in a better way to existing alternatives. For now, it's just a different way, one that fits my very specific needs; and if it helps others, so be it. I'll let you know how it goes.