Design Of Thee Container Orchestration Benchmark
This document discusses the CNBM-CO benchmark design and how we got there as well as related efforts.
The benchmark is executed as follows (cf also above architecture):
- User provisions the cluster and provides a running cluster to the benchmark.
- Benchmark itself runs in the the cluster, triggered by the local
- Results are dumped to stdout as CSV/JSON, locally.
Supported targets at the moment are:
Benchmark run types
The CNBM-CO benchmark is composed of a number of micro-benchmarks, called benchmark run types (or simply run types) which are covering different areas.
The following sequence:
secondspotentially with different runtimes (Docker, UCR, CRI-O).
N containers and measures the distribution over nodes in
map: nodeid -> set of containers.
Measures API calls from within cluster in
- list containers
- list pods
- list services/endpoints
Measure service discovery in
- Start a service and measure how long it takes until it can be discovered from different nodes.
- How long does a query/look-up take (while scaling services)?
Recovery performance in case of re-scheduling a pod/ (container) in
For individual run types we consider one or more of the following dimensions:
- Number nodes, that is, worker nodes that are hosting containers.
- Number of containers.
- Container runtime type (Docker, UCR, CRI-O).
- Failure rate (per container, nodes, network).
While no prior art exists that has the same scope as the CNBM-CO benchmark there are a number of (related) efforts we reviewed and learned from:
- Scheduler Performance Test, by Hongchao Deng and Xiang Li (Kubernetes SIG Scale)
- The Cloud Container Cluster Common Benchmark allingeek/c4-bench
- A Go-based framework for running benchmarks against Docker, containerd, and runc engine layers
- From the Kubernetes blog: 1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2
- OpenShift v3 Scaling, Performance and Capacity Planning and the
- Deploying 1000 nodes of OpenShift on the CNCF Cluster (Part 1)
- From the CoreOS blog: Exploring Performance of etcd, Zookeeper and Consul Consistent Key-value Datastores