Ceph RBD Mirroring for Disaster Recovery

Ceph RBD mirroring ensures your private cloud data stays safe by replicating block device images between clusters. It’s a disaster recovery tool that minimizes downtime and data loss during failures. Here are the basics:

• Two Modes: One-way (active-passive) and two-way (active-active) replication.
• Methods: Journal-based for real-time replication or snapshot-based for periodic syncing.
• Key Benefits: Reduces downtime, improves Recovery Time Objective (RTO) and Recovery Point Objective (RPO).
• Setup Essentials: Two healthy Ceph clusters, reliable high-bandwidth network, and proper pool configurations.
• Best Practices: Monitor replication lag, optimize performance, and regularly test failovers.

With Ceph RBD mirroring, you can build a reliable disaster recovery plan for private OpenStack clouds and protect critical workloads from unexpected outages.

Requirements for Ceph RBD Mirroring Setup

Cluster Design and Network Setup

To set up Ceph RBD mirroring effectively, you’ll need at least two healthy Ceph clusters located in separate data centers. These data centers should connected by a reliable, high-bandwidth network for the best backup and disaster recovery protections.

The network setup plays a big role in the success of mirroring operations. According to Ceph documentation:

“Each instance of the rbd-mirror daemon must be able to connect to both the local and remote Ceph clusters simultaneously (i.e. all monitor and OSD hosts). Additionally, the network must have sufficient bandwidth between the two data centers to handle mirroring workload.”

Bandwidth is a major factor. For instance, replicating 1 TB of data over a 1 Gbps link takes approximately three hours, while a 10 Gbps link reduces this to just twenty minutes. To handle the demands of inter-cluster and client communications, provisioning at least a 10 Gbps connection is recommended.

Both clusters should also share similar CRUSH hierarchies in terms of capacity and performance. To keep operations smooth, consider using enterprise-class SSDs for Ceph Monitor and Manager hosts, especially for workloads that are metadata-intensive.

Latency requirements vary depending on the replication strategy. Asynchronous mirroring is more forgiving and can handle higher latency, making it a good choice for geographically distributed deployments. However, if you’re using synchronous replication in a stretch cluster setup, the round-trip time (RTT) between sites must not exceed 10 ms.

For those seeking a pre-built solution, OpenMetal provides hosted private clouds powered by OpenStack and Ceph. These are well-suited for implementing dual-site architectures designed for enterprise disaster recovery. It’s incredibly fast and easy to spin up a cloud, even in a new location, as needed with us.

Once your clusters and network are ready, the next step is configuring the specific features and pools required for mirroring.

Feature and Pool Configuration Requirements

After optimizing your clusters and network, the focus shifts to aligning feature and pool configurations for successful mirroring.

First, ensure that pools with identical names exist on both clusters. If you’re using journal-based mirroring, proper journaling must be enabled. Keep in mind that enabling RBD journaling can nearly double write latencies.

Certain image features are critical for mirroring. These include exclusive-lock and journaling capabilities. The exclusive-lock feature ensures that only one client can modify an image at any given time, while journaling supports consistent replication. In this setup, primary images remain writable, while non-primary images are read-only.

Mirroring can be configured in two modes:

• Pool mode: Enables mirroring for all images in a pool.
• Image mode: Allows selective mirroring for specific images.

If you’re integrating with OpenStack Cinder, you’ll need to configure mirroring in image mode.

Version compatibility is another consideration. Mixing different versions of Ceph server and client software is unsupported and could lead to unexpected issues. Additionally, for users of Red Hat Ceph Storage 5, only containerized daemons are supported.

To optimize performance, monitor cluster activity closely during the initial setup and testing phase. Running multiple migrations in parallel without testing can reveal bottlenecks that might otherwise go unnoticed.

Lastly, ensure that your network bandwidth allocation accounts for not just replication traffic, but also client requests and recovery operations. Proper bandwidth management helps prevent excessive latency caused by overlapping demands.

Making RBD Snapshot-based Mirroring Robust for Disaster Recovery | Ceph Days NYC 2024

This video is a great resource to learn more about Ceph RBD mirroring as you follow along with this article:

How to Set Up Ceph RBD Mirroring

After configuring your clusters and network, you can set up Ceph RBD mirroring in three main steps to create a dependable disaster recovery solution.

Enable Pool-Level Mirroring

Start by enabling mirroring on the pools that will be part of your disaster recovery setup. This step needs to be performed on both clusters. Use the following command on each cluster:

rbd mirror pool enable <pool-name> <mode>

Replace <pool-name> with the name of your pool and <mode> with either pool or image. If you choose pool mode, all images with journaling enabled are mirrored automatically, offering broad coverage. On the other hand, image mode requires you to enable mirroring for each image individually, giving you more control.

For instance, in a two-site setup with a pool named “data”, you would run:

• rbd mirror pool enable data pool (on site-a)
• rbd mirror pool enable data pool (on site-b)

If you’re working with OpenStack Cinder or need selective mirroring (e.g., for snapshot-based setups), opt for image mode by replacing pool with image in the command.

Deploy and Configure rbd-mirror Daemons

Once pool mirroring is enabled, you’ll need to set up the rbd-mirror daemon to handle replication. This daemon ensures that image updates are synchronized between clusters. For one-way mirroring, deploy the daemon on the secondary cluster. For two-way mirroring, deploy it on both clusters.

To deploy the daemon, use Ceph’s orchestrator and run:

ceph orch apply rbd-mirror --placement=NODENAME

Replace NODENAME with the name of your target host. Assign each daemon a unique Ceph user ID to prevent conflicts and ensure proper authentication. You can manage the daemon with systemd using:

systemctl enable ceph-rbd-mirror@rbd-mirror.<unique_id>

For testing purposes, you can run the daemon in the foreground with logging enabled:

rbd-mirror -f --log-file={log_path}

Connect Peer Clusters

The final step is to establish secure communication between the clusters. Ceph provides both automated and manual methods for connecting peer clusters.

For automated peer discovery, generate a bootstrap token on your primary site:

rbd --cluster site-a mirror pool peer bootstrap create --site-name site-a image-pool

Then, import the token on your secondary site:

rbd --cluster site-b mirror pool peer bootstrap import --site-name site-b image-pool <token>

Replace <token> with the token you generated. If you prefer manual configuration, you can add a peer with this command:

rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b

Ensure that the peer cluster’s monitor and client key are securely stored in your local Ceph monitor configuration.

Once the clusters are connected, the rbd-mirror daemons will start replicating data based on your schedules and policies, creating a reliable disaster recovery system.

For those using OpenMetal’s hosted private cloud with OpenStack and Ceph, these steps integrate smoothly with your environment, to create a strong and reliable disaster recovery framework.

Ceph RBD Mirroring Best Practices

Incorporate these key practices to strengthen your disaster recovery setup. Fine-tuning your Ceph RBD mirroring configuration involves balancing performance adjustments with thorough monitoring for dependable disaster recovery while keeping overhead low and efficiency high.

Performance Tuning and Optimization

Your network bandwidth must handle at least (N × X) throughput, plus an extra 20–30% buffer. Here, X represents your average write throughput (in Mbps) for images in the primary cluster. For example, if your average write throughput is 100 Mbps across 5 images, ensure the network connection to the secondary site supports at least 500 Mbps, with an additional safety margin.

Both clusters should have matching capacity and performance capabilities to avoid bottlenecks during replication.

Adjust journaling settings to balance consistency and performance. Larger journals reduce flush frequency but use more memory, while smaller journals may increase replication lag during heavy write periods. Tailor these settings to your workload needs.

Allocate sufficient resources for rbd-mirror daemons based on your replication load. Monitor CPU and memory usage during peak write times to ensure they’re adequate. For environments with numerous images or high write throughput, distribute the load by deploying multiple rbd-mirror daemons across different nodes.

Review and update CRUSH rules as your system scales to prevent bottlenecks. Proper placement group distribution helps maintain an even load across your storage infrastructure, avoiding hotspots that could hinder replication.

Once you’ve optimized performance, maintaining a strong monitoring framework is critical.

Monitoring and Alerting Setup

Performance optimization alone isn’t enough – effective monitoring is important. The Ceph Dashboard integrates well with tools like Prometheus, Grafana, and Alertmanager to deliver monitoring capabilities. These tools can be deployed using cephadm for simplified configuration and management.

Start by enabling the Prometheus module in the ceph-mgr daemon to expose internal Ceph metrics. Install node-exporter on every cluster node to collect host-level metrics, such as CPU, memory, and network usage. This combination provides insights into both Ceph-specific and system-level performance.

Create Grafana dashboards focused on RBD mirroring metrics. Key areas to monitor include replication lag, journal utilization, and network throughput between sites. Set alerts for replication delays that exceed your recovery point objective (RPO). Other critical metrics include mirror image health, daemon connectivity, and peer cluster availability.

Configure Alertmanager to prioritize actionable notifications. For example, mirror failures should trigger immediate alerts to on-call engineers, while performance degradation alerts might allow for longer evaluation periods.

Centralized logging with Loki and Promtail simplifies troubleshooting by consolidating logs across your disaster recovery environment. This is especially useful for diagnosing intermittent network issues or daemon failures.

Keep in mind that RBD image monitoring is turned off by default to reduce performance overhead. Enable it selectively for critical images instead of applying it across all pools.

If your environment uses self-signed certificates, disable certificate verification in the dashboard configuration for Prometheus and Alertmanager to avoid connectivity problems. However, in production environments, proper certificate management is essential to ensure security.

Regularly test your monitoring setup to confirm alerts work as expected during real failures. Schedule drills that simulate various failure scenarios to validate both the technical monitoring and the operational response procedures.

For OpenMetal users, these monitoring practices integrate with OpenStack and Ceph environments, offering enterprise-grade observability for disaster recovery setups. OpenMetal also includes Datadog monitoring at no extra charge to our customers, giving you another capable and powerful monitoring option.

Testing Your Disaster Recovery Setup

Regular testing is the key to ensuring your Ceph RBD mirroring strategy aligns with disaster recovery goals and keeps your data safe. Without thorough testing, you risk uncovering issues during an actual emergency – when fixing them becomes much harder.

How to Test Failover Procedures

To start, set up your test environment to mimic real-world conditions. Before running any failover tests, sync your VM and container configurations to the secondary site using tools like rsync. This ensures that when images on the backup cluster are promoted, you can actually boot your applications and confirm they work as intended.

For planned failover tests, where both clusters are operational, follow a specific sequence to avoid split-brain scenarios. Begin by demoting the image on site A with the rbd mirror image demote command. This step ensures the primary image stops accepting writes and finishes any pending journal entries. Once the demotion is complete, promote the corresponding image on site B using rbd mirror image promote.

“RBD only provides the necessary tools to facilitate an orderly failover of an image. An external mechanism is required to coordinate the full failover process (e.g. closing the image before demotion).” –Ceph docs

For emergency failover tests, simulate a primary site outage. In these cases, you’ll need to force the promotion on site B using the --force flag with the promote command. Be aware that this approach might result in some data loss, depending on the replication lag at the time of the failure.

After promoting images on the secondary cluster, boot your VMs and containers. But don’t stop there – make sure the applications function correctly under regular operational loads.

Track recovery metrics like Recovery Time Objective (RTO) and Recovery Point Objective (RPO), and document the actual recovery times. Use Ceph’s monitoring tools during testing:

• Run ceph status for an overview of cluster health.
• Use ceph -w to monitor the cluster log in real-time.
• Check replication progress with rbd mirror image status.

Pay close attention to failover time (how quickly operations shift to the backup site) and failback time (how long it takes to return to the primary site once it’s restored). After confirming the failover process works, evaluate how to handle partial or intermittent failures.

Managing Partial Failures

Partial failures, such as network partitions or split-brain scenarios, can complicate disaster recovery. These occur when communication between sites is disrupted, leaving both clusters thinking they should be the primary.

Ceph RBD mirroring uses exclusive locks and journaling to maintain crash-consistent replicas, but network issues can still lead to tricky recovery situations. When connectivity problems arise, use the rbd mirror image status command to check the state of each image across both clusters.

If a split-brain event happens, resolving it requires manual intervention. Determine which cluster holds the most recent and reliable data by examining monitoring logs and application timestamps. Demote the outdated image with rbd mirror image demote, then trigger a complete resync using rbd mirror image resync. Keep in mind that resyncing can take significant time, depending on the size of the images and the network bandwidth available.

Monitor heartbeat messages between Ceph OSDs during partial failures. By default, health checks are triggered if heartbeat times exceed 1,000 milliseconds (1 second). Use ceph health detail to pinpoint which OSDs are experiencing delays. For a deeper dive into network performance, run the dump_osd_network command to gather detailed networking data.

When recovering from network partitions, establish clear rules for determining which data takes priority. Document which applications write to which images, and create decision trees to resolve conflicts based on your business needs. Some organizations prioritize the most recent transactional data, while others focus on data integrity.

During disaster recovery drills, test a variety of partial failure scenarios. Simulate conditions like network latency, packet loss, and complete communication breakdowns between sites. These tests will give you a better understanding of how your workloads respond under pressure and highlight areas for improvement.

These procedures integrate well with OpenMetal’s hosted OpenStack and Ceph deployments, strengthening your disaster recovery framework.

Wrapping Up: Ceph RBD Mirroring

Ceph RBD mirroring offers a streamlined way to handle disaster recovery for private OpenStack clouds. By asynchronously replicating block device images across clusters, it ensures point-in-time consistent replicas of your data changes.

But it’s not just about protecting data. RBD mirroring also reduces downtime and supports business continuity by enabling a smooth failover to secondary clusters during primary site outages. This functionality lays the groundwork for a resilient disaster recovery strategy and ongoing improvements.

“RBD Mirroring in Ceph comes into play a powerful feature designed to keep your block storage resilient, even in the face of disaster… I’ve found RBD Mirroring to be a game-changer for disaster recovery (DR) and business continuity.” – Mohamad Reza Moghadasi, Software Engineer

As highlighted earlier, successfully implementing RBD mirroring involves a comprehensive approach. This includes everything from designing clusters and configuring networks to setting up pools, deploying daemons, and connecting peer clusters. However, the work doesn’t end with deployment. Regular tasks like monitoring replication lag, ensuring sufficient network bandwidth between clusters, encrypting data transfers, and tracking Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) are needed to maintain efficiency and security.

For workloads where data is critical, consider adding off-site backups to further minimize the risk of data loss. This layered approach strengthens your overall disaster recovery plan, offering protection that goes beyond mirroring.

OpenMetal’s hosted private clouds, built on OpenStack and Ceph, provide an ideal platform for deploying these strategies. With enterprise-grade infrastructure and the adaptability of open source solutions, your RBD mirroring setup can scale alongside your business while providing the reliability your operations demand.

Disaster recovery isn’t a one-time task – it requires ongoing testing and monitoring to ensure your systems remain dependable. When implemented and managed correctly, Ceph RBD mirroring provides the resilience needed to support reliable disaster recovery for private OpenStack environments.

FAQs

What is the difference between one-way and two-way replication in Ceph RBD mirroring, and when should you use each?

In Ceph RBD mirroring, one-way replication is a method where data flows from a primary cluster to a secondary cluster. This setup works well for disaster recovery, as the secondary cluster serves as a backup without the need to sync data back to the primary cluster. It’s straightforward to set up and uses fewer resources, making it a practical choice for environments prioritizing basic data protection.

On the other hand, two-way replication allows data to sync between two clusters in both directions. This setup is ideal for active-active configurations, where both clusters handle read and write operations. It ensures continuous synchronization, providing higher availability and redundancy – key for disaster recovery scenarios that require real-time data consistency across multiple locations.

How does Ceph RBD mirroring maintain data consistency and reliability during network issues or outages?

Ceph RBD mirroring boosts data consistency and reliability through journal-based replication. This method logs every change to block device images in a sequential manner, enabling replicas to remain consistent at specific points in time across different clusters. The rbd-mirror daemon plays a key role by synchronizing updates between the primary and secondary images, ensuring changes are applied in the correct sequence.

If a network failure or partial outage occurs, the journaling system steps in to capture all modifications accurately. These changes can then be replayed, keeping the secondary image in a crash-consistent state. This approach reduces the chances of data loss and allows recovery to the most recent stable state without hassle.

How can I monitor and optimize Ceph RBD mirroring to ensure data safety and fast recovery?

To maintain an efficient and reliable Ceph RBD mirroring setup, consider these important practices:

• Keep an eye on performance: Leverage Ceph’s built-in monitoring tools to track mirroring lag and set up alerts for potential failures. This proactive approach helps you address problems early, ensuring data remains consistent and accessible.
• Improve your network setup: A low-latency, high-bandwidth connection between clusters is crucial. Poor network performance can slow down replication and compromise reliability.
• Adjust configurations as needed: Regularly analyze performance metrics and tweak settings to eliminate bottlenecks and make the best use of available resources.

By following these steps, you’ll strengthen the reliability of your Ceph RBD mirroring, safeguarding your data and reducing recovery times when issues arise.