This article introduces the cluster management tool pvecm of Proxmox VE, including how to create, add, and remove nodes, as well as the deployment requirements for the cluster. It emphasizes that all nodes must be on the same subnet, require time synchronization, and the necessity of SSH communication. The steps for isolating a node are also detailed, involving operations such as disabling services, modifying configurations, and restarting.

1. Introduction to pvecm

Proxmox VE’s cluster management tool pvecm is used to create a “group” composed of multiple physical server nodes. This group of servers is called a “cluster.” Proxmox VE uses the Corosync Cluster Engine to ensure stable and reliable communication within the cluster, which can have up to 32 physical nodes (more is possible, depending on network latency, which should be less than 2ms). Using pvecm, you can create a new cluster, add nodes to the cluster, remove nodes from the cluster, view cluster status information, and perform various other cluster management operations. The Proxmox VE cluster file system (pmxcfs) ensures that configuration information is transparently sent to all nodes in the cluster and remains consistent.

2. Cluster Deployment Requirements

All nodes must be on the same subnet so that each node can communicate using Corosync multicast (details can be found in the Corosync Cluster Engine documentation). Corosync uses UDP ports 5404 and 5405 for cluster communication.  

The date and time on all nodes need to be synchronized.  

Nodes must be able to establish SSH communication over TCP port 22.  

If you need to configure HA, at least 3 physical server nodes are required to ensure the effectiveness of the cluster quorum mechanism. Additionally, all nodes must use the same version of Proxmox VE.  

It is recommended to allocate dedicated network interfaces for cluster communication, especially when configuring shared storage, to ensure stable and reliable cluster communication.

3. Steps to Create a Cluster

1. Create Cluster

   root@s130:/etc/ssh# pvecm create mycluster

   # The cluster name cannot be modified once created, so please plan the name before creation.

2. View Cluster Status

   root@s130:/etc/ssh# pvecm status

3. Add Cluster Node

   # To avoid virtual machine ID conflicts, Proxmox VE stipulates that no virtual machines can be configured on a new node before joining the cluster. Additionally, the existing configuration information in the new node’s /etc/pve directory will be completely overwritten by the cluster configuration.

   root@s129:~# pvecm add 192.168.0.130

4. View Cluster Status Again

   root@s129:~# pvecm status

5. Check Cluster Nodes

   root@s129:~# pvecm nodes

6. Remove Cluster Node

   # Removal can only be performed from other nodes in the cluster. First, migrate all virtual machines on the node to be removed to other nodes. Ensure that there is no data or backup on the node to be removed that you need to keep, or that related data has been properly backed up. You must shut down the node to be removed before executing the deletion command and ensure that the deleted node does not restart. If there are virtual machines on the node, you can first back them up using vzdump, then delete the virtual machines on the node, and restore the original virtual machines with new IDs after rejoining the cluster.

   root@s130:/etc/ssh# pvecm delnode s129

   # Rejoining after deletion will result in an error (aside from reinstalling, no method for rejoining has been found; if you know how to rejoin, please let me know, thank you!). Please operate with caution.

7. Get Command Help

   root@s129:~# pvecm help

4. Steps to Isolate a Node

1. First, stop the pve-cluster service on the node to be isolated.

   # systemctl stop pve-cluster.service

   # systemctl stop corosync.service

2. Then set the cluster file system of the node to be isolated to local mode.

   # pmxcfs -l

3. Next, delete the corosync configuration file.

   # rm /etc/pve/corosync.conf

   # rm -rf /etc/corosync/*

4. Finally, restart the cluster file system service.

   # killall pmxcfs

   # systemctl start pve-cluster.service