I recently built a dedicated three-node Ceph cluster with Rook on bare metal to serve block, file and object storage to a small fleet of Kubernetes clusters. The Rook YAML turned out to be the easy part. The part that decides whether the cluster performs is the network underneath it, so this build went benchmark-first: prove the 10G network is actually delivering 10G before a single OSD exists, then benchmark Ceph against that baseline.

Topology

Three identical servers, each with:

  • 4x large (~3.7 TiB) SSDs for capacity OSDs
  • 1x small SSD for metadata pools
  • 1x NVMe as the BlueStore DB/WAL device
  • 2x 10G NICs, bonded

They run a compact three-node OKD cluster (SCOS, all nodes are control plane, no workers) provisioned with the agent-based installer - a long way from the cloud install I described in OKD 4.4 on AWS. Ceph gets the whole cluster to itself: storage consumers live on other clusters and come in over the network, which is exactly why the network gets benchmarked before anything else.

Network design: bond first, jumbo frames everywhere

Each node bonds both 10G interfaces with LACP and enables jumbo frames. This is declared at install time with an NMStateConfig per node, picked up by the agent-based installer:

apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: stor01
spec:
  config:
    interfaces:
      - name: bond0
        type: bond
        state: up
        mtu: 9000
        ipv4:
          enabled: true
          dhcp: true
        link-aggregation:
          mode: 802.3ad
          options:
            miimon: "100"
          port:
            - eno1
            - eno2
      - name: eno1
        type: ethernet
        state: up
        mtu: 9000
      - name: eno2
        type: ethernet
        state: up
        mtu: 9000

The member NICs need mtu: 9000 as well, not just the bond. Addressing is DHCP with static leases, so stor01 through stor03 always come up as 10.0.0.11 through 10.0.0.13.

The MTU subtlety: host 9000, pods 8900

Here is the detail that quickstarts skip: your pods do not get the host MTU. OVN-Kubernetes encapsulates pod traffic in Geneve, and the overhead is 100 bytes, so the cluster network MTU ends up 100 lower than the host:

$ oc get network.config cluster -o yaml | egrep -i 'mtu|defaultNetwork'
  clusterNetworkMTU: 8900

Both layers need verifying, because a mismatch anywhere in the path (host, switch, overlay) degrades Ceph silently: OSD heartbeats and small ops work fine while large writes fragment or stall. Verify the host layer with do-not-fragment pings; 8972 bytes of payload plus 28 bytes of ICMP/IP headers is exactly a 9000-byte frame:

[core@stor02 ~]$ ping -M do -s 8972 10.0.0.11
8980 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=3.39 ms

[core@stor02 ~]$ ping -M do -s 9000 10.0.0.11
ping: sendmsg: Message too long

Then the same from inside a pod, where the ceiling is 8900:

$ ip a s eth0
2: eth0@if104: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8900 qdisc noqueue state UP

$ ping -M do -s 9000 10.0.0.11
ping: local error: message too long, mtu=8900

$ ping -M do -s 8872 -c 1 10.0.0.11
8880 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=4.56 ms

Jumbo frames confirmed on both layers. One more check that later proved its value: walk the switch ports and confirm every port in the data path really runs at 10G with a jumbo MTU. On our leaf switch one port turned out to be 1G at MTU 1500 - it was only an uplink, not in the Ceph path, but if a storage node had landed on it, no amount of Ceph tuning would have explained the numbers.

Baseline: iperf3 before Ceph exists

Before deploying Rook I wanted three numbers: node-to-node throughput on the host network, the same through the OVN overlay, and same-node as a sanity check. All with iperf3 -P 4 -t 30.

Server pinned to stor01 on the host network:

$ oc run iperf3-server-node -n netperf --image=quay.io/networkstatic/iperf3:latest \
    --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"stor01"},"hostNetwork":true,"dnsPolicy":"ClusterFirstWithHostNet"}}' \
    -- /bin/sh -c "iperf3 -s"

Client pinned to stor02, also on the host network:

$ oc run iperf3-client -n netperf --rm -i --restart=Never \
    --image=quay.io/networkstatic/iperf3:latest \
    --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"stor02"},"hostNetwork":true,"dnsPolicy":"ClusterFirstWithHostNet"}}' \
    -- iperf3 -c 10.0.0.11 -P 4 -t 30
...snip...
[SUM]   0.00-30.00  sec  34.5 GBytes  9.88 Gbits/sec   37             sender
[SUM]   0.00-30.00  sec  34.5 GBytes  9.88 Gbits/sec                  receiver

9.88 Gbit/s aggregate with 37 retransmits over 30 seconds: line rate. Then the same pair as regular pods, client talking to a ClusterIP service, so the traffic crosses the Geneve overlay:

[SUM]   0.00-30.00  sec  33.8 GBytes  9.67 Gbits/sec  261             sender
[SUM]   0.00-30.00  sec  33.8 GBytes  9.67 Gbits/sec                  receiver

9.67 Gbit/s. More retransmits (261), but the overlay costs about two percent of throughput. That killed a design debate before it started: no need to put Ceph on the host network for this workload, the default SDN path is fine.

Same-node through the SDN, client and server both on stor01:

[SUM]   0.00-30.00  sec  49.9 GBytes  14.3 Gbits/sec    4             sender

Faster than line rate, as expected - the traffic never touches the physical NIC. Useful as a marker: if a “node-to-node” test ever reports 14 Gbit/s, the scheduler put both pods on the same node and the test measured nothing.

Rook: explicit devices, explicit device classes

With the network proven, the Rook install itself holds few surprises: the operator chart plus the cluster chart, both rendered by Kustomize with the Helm generator. Rook v1.18 deploying Ceph v19.2 (Squid). The one place I refuse to use the quickstart defaults is device selection - no auto-discovery, every OSD device listed per node:

cephClusterSpec:
  storage:
    useAllNodes: false
    useAllDevices: false
    config:
      metadataDevice: "/dev/disk/by-path/pci-0000:81:00.0-nvme-1"
      osdsPerDevice: "1"
    nodes:
      - name: "stor01"
        devices:
          - name: "/dev/disk/by-path/pci-0000:02:00.0-sas-phy1-lun-0"
            config:
              deviceClass: "hdd"
          - name: "/dev/disk/by-path/pci-0000:02:00.0-sas-phy2-lun-0"
            config:
              deviceClass: "hdd"
          # ...snip: two more hdd devices...
          - name: "/dev/disk/by-path/pci-0000:02:00.0-sas-phy4-lun-0"
            config:
              deviceClass: "ssd"
      # ...snip: stor02 and stor03, same layout...

By-path device names survive reboots and disk replacements; /dev/sdX names do not. One thing that surprises people reading this config: the capacity devices are SSDs, yet they are tagged deviceClass: "hdd". Ceph device classes are just labels for CRUSH rules to bind to, so I use hdd to mean “bulk capacity” and ssd to mean “fast metadata” regardless of what the hardware actually is. The tags matter because the pools bind to them: the RBD block pool replicates 3x across hosts on the hdd class, CephFS and the object store keep their metadata pools on ssd:

cephBlockPools:
  - name: ceph-blockpool
    spec:
      failureDomain: host
      replicated:
        size: 3
      deviceClass: hdd
    storageClass:
      enabled: true
      name: ceph-block
      isDefault: true

Three nodes, replication size 3, failure domain host: every write lands on all three nodes, so the network carries two replica copies for each client write. That is why the iperf3 numbers above are not optional homework.

Storage benchmarks: RADOS and RBD

All raw-Ceph tests ran from the rook-ceph-tools pod against a throwaway pool:

$ ceph osd pool create bench 128 128
$ ceph osd pool application enable bench rados

$ rados bench -p bench 60 write --no-cleanup --object-size 4M
...snip...
Bandwidth (MB/sec):     810.887
Average IOPS:           202
Average Latency(s):     0.0788847

$ rados bench -p bench 60 seq
Bandwidth (MB/sec):   1596.33
Average IOPS:         399
Average Latency(s):   0.0386791

811 MB/s sustained writes on a 3x replicated capacity pool is healthy: with replication the cluster is moving roughly 2.4 GB/s of real data, which is where the 10G links start earning their keep. The 1.6 GB/s reads are partly cache-assisted; treat sequential read numbers from a fresh pool with suspicion.

RBD-level, 4M IO with 32 threads:

$ rbd create bench/rbd-bench --size 10240

$ rbd bench bench/rbd-bench --io-type write --io-size 4M --io-threads 32 --io-total 20G
elapsed: 39   ops: 5120   ops/sec: 128.1   bytes/sec: 512 MiB/s

$ rbd bench bench/rbd-bench --io-type read --io-size 4M --io-threads 32 --io-total 20G
elapsed: 11   ops: 5120   ops/sec: 463.811   bytes/sec: 1.8 GiB/s

Finally, the numbers a consumer actually sees: fio from a Fedora pod against a 100 GiB PVC on the ceph-block StorageClass, direct=1, iodepth=32, numjobs=4:

PatternRead BWWrite BWRead IOPSWrite IOPS
Seq write 128 KiB-445 MiB/s-~3.6k
Seq read 128 KiB2610 MiB/s-~20.9k-
Rand 4 KiB, 70% read129 MiB/s55 MiB/s~32.9k~14.1k
Rand 4 KiB, 70% write38 MiB/s89 MiB/s~9.8k~22.8k

The random 4K numbers are the honest ones for database-style workloads, and they are solid for replicated OSDs with the DB/WAL offloaded to NVMe. The sequential read row is warm-cache performance, not disk performance.

Clean up after yourself; benchmark pools full of 4M objects are not free:

$ rbd rm bench/rbd-bench
$ ceph osd pool delete bench bench --yes-i-really-really-mean-it

The PG autoscaler that did nothing

A few weeks after the cluster went into service, ceph osd df stopped looking right: three of the twelve capacity OSDs held 767 GiB of the block pool while the other nine sat nearly empty. The pool had every property I configured - replication 3, correct device class, correct CRUSH rule - and still funneled all data to the same three disks.

The cause was one level down. Every pool the charts had created was still at pg_num 1:

$ ceph osd pool ls detail | grep blockpool
pool 2 'ceph-blockpool' replicated size 3 min_size 2 crush_rule 1 ...snip... pg_num 1 pgp_num 1 autoscale_mode on

A placement group is the unit Ceph distributes across OSDs. One PG on a size-3 pool maps to exactly one triplet of OSDs, so the entire pool lands on three disks no matter how many you have. Normally the pg_autoscaler raises pg_num as a pool grows, and autoscale_mode on says it is responsible here. But in Ceph Squid 19.2.3 the module is broken and does nothing, which you can see by asking it for its own status:

$ ceph osd pool autoscale-status
$

Empty output, no error. And the reason my benchmarks never caught it is right there in the setup commands above: ceph osd pool create bench 128 128 creates the benchmark pool with an explicit 128 PGs. The pools that matter were created by the Helm chart and relied on the autoscaler.

The fix is explicit pg_num values on every pool definition, sized as PGs x replicas / OSDs:

 cephBlockPools:
   - name: ceph-blockpool
     spec:
       failureDomain: host
       replicated:
         size: 3
       deviceClass: hdd
+      parameters:
+        pg_num: "32"

32 PGs x 3 replicas / 12 capacity OSDs = 8 PGs per OSD for the block pool. The CephFS metadata pool got 16 (three ssd-class OSDs), the CephFS data pool 8. Once the new values reconciled, the data spread across all twelve capacity OSDs and stayed there.

Takeaways

  • Benchmark the network before Ceph exists. Once OSDs are in the path, you cannot tell a slow disk from a slow link from a wrong MTU.
  • Host MTU 9000 means pod MTU 8900 on OVN-Kubernetes. Verify both layers with do-not-fragment pings; a mismatch fails silently and only under load.
  • Check every switch port in the data path for speed and MTU. One forgotten 1G port can hide behind healthy-looking Ceph status for a long time.
  • The OVN overlay cost about two percent of throughput here. Measure it before you complicate your deployment with host networking.
  • List OSD devices explicitly, by-path, with explicit device classes. Ceph auto-discovering a disk you meant for something else is a bad day.
  • Do not assume the PG autoscaler works. Confirm ceph osd pool autoscale-status returns output and that no production pool sits at pg_num 1, or one OSD triplet ends up holding an entire pool.
  • Record the benchmark results next to the manifests in git. When someone reports “storage is slow” a year later, the baseline is the first thing you will want.

Resources