Resilient CICD

This might not come in as something new to many. But this has been a constant and compelling discussion I’ve seen teams have on and on.

CICD is evolving for good. We’re moving away from the era of self hosted CIs . Managed is the new thing. And even if it’s not completely managed, modern CI providers give you the ability to choose your own agents. The agent can be as simple as your laptop (or even phone in some cases).

With this comes the evolution of CICD architecture. CI services are heavily investing in containerised agents. You don’t need the hassle of managing packages, no sudden surprises when a new service comes up. Moreover, your build stage could be running in java container while your test stage could be running in a nodejs container. The possibilities are endless.

But there’s one catch. These containerised agents are isolated. One agent doesn’t know what the other does. That means your agent running the tests might not know what the build agent did.

That all works well as long you do not have inter stage dependencies. Which brings me to my main point. Should we at all have inter stage dependencies?

I’m in favour of keeping it simple. Let every stage run on a dataset that doesn’t rely on the previous stage’s output. My reasons behind this being:

1. It doesn’t break the principle of containerisation. Every stage starts as if it’s running from scratch.
2. You know what you don’t know. No hidden surprises. You know each of your stage starts from scratch, provides predictability.
3. Portability — I can easily port my CI to any other tool without having to make mega changes. Of course, the syntax will change based on the provider. But as long as we’ve kept the implementation agnostic of the provider, it shouldn’t be a massive effort.
4. Easy handling of artifacts: Simply store your artifacts in a centralised hub. Reduced hassle of transferring the artifacts from one stage to the other.
5. The S in SOLID — Heavy reliance on the output from the previous stage seems to me like a smell. I might be wrong. But shouldn’t a stage be self sufficient of what it needs to do, without having a bloated responsibility of fetching the output from a previous stage. Again, if this stage is downloading artifacts from an artifactory, that doesn’t necessarily look like a smell.
6. If two stages are tied to each other, we might as well club them. That’s what I generally do. It may sometimes lead to bloated responsibility in a single stage. Maintain caution.
7. Ease of understanding — Frankly speaking, I find it real hard to visualise volume mounts being transferred from container to container, just to share the artifacts. It starts simple, but escalates quickly sometimes.

This isn’t necessarily applicable in all scenarios. In one of the projects, we were able to leverage the dependency to reduce our build times by half. I think it’s mostly dependent on the CI provider you’re using. If it’s a managed one, you don’t have much choice. But if it’s self hosted one, your imagination might sometimes drive you crazy.

Cheers!