Integration Testing with Kubernetes

Using terraform and Rust

Introduction

I’m building a distributed ML system with custom Kubernetes controllers. Figuring out how to test it turned out to be an incredibly entertaining rabbit hole: provisioning local clusters, understanding how Kubernetes controllers work, dealing with flaky tests, and making it all work seamlessly with Rust.

This series documents the journey, and my goal by sharing this adventure is to save time for those who might decide to chase the rabbits themselves. The code is available on GitHub.

• Part 1: Local k8s testing infrastructure (this post)

• Part 2: Custom CRD and controller implementation

• Part 3: Simulating multiple regions

Before we begin, let me start with the test in Rust that we’re going to be optimizing the infrastructure for:

Looks simple enough! And it already teases some of the choices and patterns that we’re going to use.

---

Requirements

On a high level, I’d like the following to be true about the test setup:

• CI Native: The tests can be run on Github Actions’ runners

• IAC: All infrastructure dependencies are managed via terraform

• Rust Integration: Tests can be executed concurrently via cargo test

• Debuggable: I can run asserts directly against Kafka and Postgres

There are a number of questions that immediately pop up after thinking about the problem:

• Should we have one or more clusters?

• How to achieve test isolation?

• How to connect to Postgres or Kafka?

• How to interact with Kubernetes from Rust?

• How to minimize the time it takes to run the tests?

We’ll answer these questions throughout this series.

---

The Stack: Making Choices

The pattern of doing integration testing with a local k8s cluster is actually well-covered. ArgoCD¹ demonstrates how to leverage namespace-level isolation. I find the namespace-level isolation perfect for my current needs but I want to stop for a second and discuss two other options: vCluster and multiple kind clusters.

Namespace-level isolation is the cheapest option and should work for most of the tests. An example of where you might want to consider an alternative is a CRD upgrade test via a webhook, which produces a cluster-level side-effect. The most obvious case of vCluster not being enough is testing a new k8s version. We’re going to discuss this in greater detail in future posts.

For my use-case I’ve settled on the following choices:

• kind is the official tool dedicated to running local clusters on top of docker

• kube-rs is an excellent crate that provides abstractions to interact with k8s control plane¹ and also simplifies the job of writing the controllers²

• strimzi provides a kafka helm chart and a CRD for provisioning topics³

• kyverno provides a CRD for the TTL-based cleanup policy that makes the test cleanup easy

This is heavyweight infrastructure. The payoff comes when you need to test multiple interdependent k8s resources or custom controllers. I wanted the test setup to be as simple as specifying what I need using a Builder pattern. That meant investing in proper infrastructure that will shine later.

---

Directory Structure

We’re going to be using the following directory structure and expand on top of it with more functionality in the future:

.
├── .github/workflows
│   └── e2e-kubernetes.yaml     # GitHub Actions CI/CD workflow
│
└── e2e-kubernetes
    ├── Cargo.toml              # Rust dependencies
    ├── README.md               # Project documentation
    ├── src
    │   └── main.rs             # Rust application
    ├── terraform
    │   ├── kind.tf             # kind cluster with inotify fixes
    │   ├── namespace.tf        # Kafka namespace
    │   ├── strimzi.tf          # Strimzi operator deployment
    │   ├── kafka.yaml          # Kafka CR + KafkaNodePool
    │   ├── kafka.tf            # Kafka deployment logic
    │   ├── providers.tf        # Terraform providers
    │   ├── variables.tf        # Configuration variables
    │   └── outputs.tf          # Bootstrap servers, endpoints
    └── tests
        └── kafka_e2e.rs        # Kafka E2E tests

---

Provisioning: Terraform and Ready Conditions

Let’s unpack what goes on when we run terraform apply. We provision a cluster using a kind provider and install strimzi on top of it using helm. So far so good.

The tricky part is how we install kafka:

Here we use a terraform_data resource which is a native replacement for the familiar null_resource. We are safe to use kubectl without specifying the context because it’s running within CI and there’s only one cluster there⁴.

The key insight: Custom resources expose status conditions. We wait for condition=Ready to ensure that the Kafka topic is fully provisioned before the tests run:

Strimzi’s helm chart installs a CRD, which we use to define our kafka configuration. On success we can see the following:

This pattern works for any CRD - not just Kafka. If you’re working with custom resources, kubectl wait --for=condition=Ready is what you should wait for (also possible to do via an API).

---

Test Isolation: TestSetupBuilder Pattern

The TestSetupBuilder encapsulates the create namespace → apply CRDs → wait for Ready → return clients flow. Here’s what happens:

Setup Phase:

• TestSetupBuilder::new(”my-test”) generates a unique namespace with UUID suffix

• .with_ttl(60) stores TTL config (Kyverno will delete namespace after 60s)

• .build() applies namespace with label: janitor/ttl=60s

Resource Provisioning:

• setup.create_kafka_topic(”messages”) constructs KafkaTopic yaml

• Applies the CRD and waits for Ready condition

Test Execution:

• setup.kafka_producer() returns FutureProducer

• setup.kafka_consumer() returns StreamConsumer

• Both connect to localhost:30092 (NodePort from kind)

Cleanup (automatic via TTL):

• Test completes or panics

• Kyverno watches TTL label

• Deletes namespace after 60s

• Finalizers processed by Strimzi

The beauty of this pattern: tests clean up even when they panic. No manual teardown needed⁵. An example of a successful run can be found here.

---

Next Up: Custom CRDs and Controllers

We’ve built the foundation: a local Kubernetes cluster with declarative infrastructure, automatic cleanup, and reliable test isolation.

In Part 2, we’ll implement our own CRD and write a controller to manage its lifecycle. We’ll explore how to use kube-rs to watch resources, reconcile state, and handle edge cases.

---

Acknowledgements

Big thanks to my friend Kamaleshwar for reviewing & giving valuable feedback to this post.

1. It also has an excellent page dedicated to the hierarchy of tests

2. https://kube.rs/controllers/intro/

3. We use it primarily to cleanup the topics by deleting namespace and triggering the cascade deletion of all the resources within it

4. If you need a multi-cluster setup, consider explicitly providing the path to the cluster config.

5. One gotcha here is the cleanup order: topic before the cluster. I’m using a bit hacky solution for now that I plan to improve upon later