Node becomes unresponsive after splitting from the cluster and merging back

[ikysil]

OrientDB Version: 3.2.38

Java Version: Java 21 (Amazon Corretto)

OS: Linux 5.15.167.4-microsoft-standard-WSL2, amd64

Expected behavior

• Node should re-open known databases
• Node should process client requests
• Node should process quorum requests

Actual behavior

• Node is ONLINE but has no databases listed in the cluster view
  • other nodes report empty database list for affected node as well
• Node does not respond to the client requests
  • com.orientechnologies.orient.core.exception.OStorageException: Cannot connect to the remote server/database
• Node counts towards quorum from POV of other nodes but does not respond to quorum requests

Steps to reproduce

1. Start cluster of three nodes - odb1, odb2, odb3
   • Roles of nodes are not relevant, but at least one MASTER is required
2. Create a new database using OrientDB Studio connected to the MASTER node
3. Wait until database becomes ONLINE on each node
4. Disrupt node odb2 heartbeats communications
   • e.g. using docker pause or docker compose pause
   • e.g. block traffic using network firewall
   • any node fits the scenario
5. Wait until odb1 and odb3 detect missing odb2 and kick it from the cluster view
6. Restore node odb2 heartbeats communications - reverse action of step 4 above
   • odb2 notices the pause with

   System clock apparently jumped from 2025-03-12 08:36:20.974 to 2025-03-12 08:40:11.880 since last heartbeat (+225906 ms) [ClusterHeartbeatManager]
   Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 225906 ms, Heartbeat-Timeout: 60000 ms [ClusterHeartbeatManager] 
   

7. Wait until odb2 starts MERGING to the cluster
8. Wait until odb2 becomes ONLINE

Workaround

Restarting any of the remaining nodes (odb1 or odb3) triggers the synchronization of the database on odb2, the node is participating in the cluster as expected after that.