Needs Arising

In a certain telecom operator’s data center, a few servers were rented to save costs; they are all low-config and relatively old servers. Three of them were chosen to form a Proxmox VE hyper-converged cluster, with virtual machines on the platform configured for high availability, and storage being distributed decentralized Ceph. These three nodes have 64G memory (16 single 4G sticks) and 6 low-speed mechanical disks of 2TB capacity each, rotating at 7200 RPM, which offer rather poor performance. Some edge businesses and less critical applications were placed on this low-performance platform. Despite resource constraints (90% total memory usage), it has been running smoothly for several years. Last week, the data center switched power, causing the switch used by the Proxmox VE cluster to lose power, resulting in all three nodes rebooting. One node failed to start after multiple attempts.

The data center provided a server, Proxmox VE was deployed, and it ran normally. Considering the resource constraints, it was decided to abandon Ceph as storage and use single-node local ZFS instead, forming a cluster without high availability. The benefits of this compromise are saving resources (one OSD uses 5G memory) and minimal direct mutual influence between clusters. The cluster is used merely to manage all Proxmox VE nodes through a unified interface.

Implementation Steps

Step 1: Create a Proxmox VE three-node cluster.

Step 2: On the first node, create ZFS storage, selecting all unused free disks. If the disk has partitions or data, it will not appear in the ZFS candidate disk list. You need to clear the data using the command line on the Proxmox VE host system Debian with “wipefs –a /dev/sdd” to make it available as a candidate disk. In my environment, there are a total of 5 disks, all integrated into ZFS.

In the Proxmox VE cluster web management interface, select the target node, then select the “Disks” submenu under the target node’s right-side menu, and continue by clicking the “Create: ZFS” button at the top.

Give the ZFS Pool a unique name within the Proxmox VE cluster, select all available disks, choose the “RAID level” from the dropdown list, and after confirming, click the “Create” button at the bottom right.

Note: When there are few disks, choose a RAID level that can be supported; otherwise, creation will fail.

Step 3: Verify that ZFS is only effective for the local node. Log into the Proxmox VE cluster web management interface and check the storage distribution of each node to see if it matches expectations.

A more direct way is to log into the Debian host system of any node in the Proxmox VE cluster and check the file “/etc/pve/storage.cfg”. The relevant text blocks for ZFS are as follows:

Note the last line of each text block; its purpose is to limit which nodes can use the ZFS Pool. If there is no such limitation, other nodes will also “see” this ZFS Pool, which is not what we want.

Troubleshooting and Handling

Sometimes, after creating the second local node ZFS Pool, it cannot be found in the Proxmox VE web management interface. The way to handle this is to log into the Proxmox VE cluster host system Debian, edit the file “/etc/pve/storage.cfg”, and append the text block to the end of the file according to the name set when creating the ZFS of this node. If edited correctly, it will immediately display correctly in the web management interface.