Cross-Region Replication

Search…
TigerGraph's Cross-Region Replication (CRR) feature allows users to keep two or more TigerGraph clusters in different data centers or regions in sync. One cluster is the primary cluster, where users would perform all normal database operations, while the other is a read-only Disaster Recovery(CR) cluster that syncs with the primary cluster. CRR includes complete native support for syncing all data and metadata including automated schema, user, and query changes. 
For users of TigerGraph, cross-region replication will help deliver on the following business goals:
Disaster Recovery: Support Disaster Recovery functionality with the use of a dedicated remote cluster 
Enhanced Availability: Enhance Inter-cluster data availability by synchronizing data using Read Replicas across two clusters
Enhanced Performance: If the customer application is spread over different regions, CRR can take advantage of data locality to avoid network latency.
Improved System Load-balancing: CRR allows you to distribute computation load evenly across two clusters if the same data sets are accessed in both clusters.
Data Residency Compliance: Cross-Region replication allows you to replicate data between different data centers or Regions to satisfy compliance requirements. Additionally, this feature can be used to set up clusters in the same region to satisfy more stringent Data sovereignty or localization business requirements.
Besides providing disaster recovery and enhanced business continuity, CRR also allows you to set up the clusters as part of Blue/Green deployment purposes for agile upgrades.
This page describes the procedure to set up a DR cluster, and the steps to perform a failover in the event of a disaster. 
What is included
The following information is automatically synced from the primary cluster to the DR cluster:
All data in every graph
All graph schemas, including tag-based graphs
All schema change jobs
All users and roles 
All queries in every graph. Queries that are installed in the primary cluster will be automatically installed in the DR cluster.  
Exclusions
The following information and commands are not synced to the DR cluster
GraphStudio metadata
This includes graph layout data and user icons for GraphStudio. 
Loading jobs
gadmin configurations
gsql --reset command
The following GSQL commands:
EXPORT and IMPORT commands
DROP ALL and CLEAR GRAPH STORE
When the primary cluster executes an IMPORT , DROP ALL, orCLEAR GRAPH STORE GSQL command, or the gsql --reset bash command, the services on the DR cluster will stop syncing with the primary and become outdated. 
See Sync an outdated DR cluster on how to bring an outdated DR cluster back in sync. 
Before you begin
Install TigerGraph 3.2 or higher on both the primary cluster and the DR cluster in the same version.
Make sure that your DR cluster has the same number of partitions as the primary cluster. 
Make sure that your DR cluster is in the same VPC as your primary cluster.
Make sure TigerGraph is not installed with a local loopback IP such as 127.0.0.1. 
Setup
The following setup is needed in order to perform a failover in the event of a disaster.
This setup assumes that you are setting up a DR cluster for an existing primary cluster. If you are setting up both the primary cluster and DR cluster from scratch, you only need to perform Step 3 after TigerGraph is installed on both clusters. 
Step 1: Backup primary data
Use GBAR to create a backup of the primary cluster. See Backup and Restore on how to create a backup. 
If you are setting up both the primary cluster and the DR cluster from scratch, you can skip Steps 1, 2, and 4 and only perform Step 3. 
Step 2: Restore on the DR cluster
Copy the backup files from every node to every node on the new cluster.  Restore the backup of the primary cluster on the DR cluster. See Backup and Restore on how to restore a backup. 
Step 3: Enable CRR on the DR cluster
Run the following commands on the DR cluster to enable CRR on the DR cluster. 
1
# Enable Kafka Mirrormaker
2
gadmin config set System.CrossRegionReplication.Enabled true
3
​
4
# Kafka mirrormaker primary cluster's IPs, separator by ','
5
gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs PRIMARY_IP1,PRIMARY_IP2,PRIMARY_IP3
6
​
7
# Kafka mirrormaker primary cluster's KafkaPort
8
gadmin config set System.CrossRegionReplication.PrimaryKafkaPort 30002
9
​
10
# The prefix of GPE/GUI/GSQL Kafka Topic, by default is empty.
11
gadmin config set System.CrossRegionReplication.TopicPrefix Primary
12
​
13
# Apply the config changes, init Kafka, and restart
14
gadmin config apply -y
15
gadmin init kafka -y
16
gadmin restart all -y
Copied!
Step 4: Force install a query on primary
Run the INSTALL QUERY -force <query_name> command on the primary cluster to force install any query (Replace <query_name> with the name of any query). After the command is finished, all other metadata operations on the primary cluster will start syncing to the DR cluster. 
Restrictions on the DR cluster
After being set up, the DR cluster will be read-only and all data update operations will be blocked. This includes the following operations:
All metadata operations
Schema changes
User access management operations
Query creation, installation, and dropping
User-defined function operations
Data-loading operations
Loading jobs operations
RESTPP calls that modify graph data
Queries that modify the graph
Fail over to the DR cluster
In the event of catastrophic failure that has impacted the full cluster due to Data Center or Region failure, the user can initiate the failover to the DR cluster. This is a manual process. Users will have to make the following configuration changes on the DR cluster to upgrade it to the primary cluster.
1
gadmin config set System.CrossRegionReplication.Enabled false
2
gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs 
3
gadmin config set System.CrossRegionReplication.PrimaryKafkaPort
4
gadmin config set System.CrossRegionReplication.TopicPrefix Primary 
5
gadmin config apply -y
6
gadmin restart -y
Copied!
Set up a new DR cluster after failover
After you fail over to your DR cluster, your DR cluster is now the primary cluster. You may want to set up a new DR cluster to still be able to recover your services in the event of another disaster. 
To set up a new DR cluster over the upgraded primary cluster:
1.Make a backup of the upgraded primary cluster
2.Run the following command on the new cluster. The commands are the mostly same as setting up the first DR cluster, except that in the fourth command, the value for System.CrossRegionReplication.TopicPrefix becomes Primary.Primary instead of Primary
3.On the new DR cluster, restore from the backup of the upgraded primary cluster
1
# Enable Kafka Mirrormaker
2
gadmin config set System.CrossRegionReplication.Enabled true
3
​
4
# Kafka mirrormaker primary cluster's IPs, separator by ','
5
gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs PRIMARY_IP1,PRIMARY_IP2,PRIMARY_IP3
6
​
7
# Kafka mirrormaker primary cluster's KafkaPort
8
gadmin config set System.CrossRegionReplication.PrimaryKafkaPort 30002
9
​
10
# The prefix of GPE/GUI/GSQL Kafka Topic, by default is empty.
11
gadmin config set System.CrossRegionReplication.TopicPrefix Primary.Primary
12
​
13
# Apply the config changes, init Kafka, and restart
14
gadmin config apply -y
15
gadmin init kafka -y
16
gadmin restart all -y
Copied!
There is no limit on the number of times a cluster can fail over to another cluster. When designating a new DR cluster, make sure that you set the System.CrossRegionReplication.TopicPrefix parameter correctly by adding an additional .Primary . 
For example, if your original cluster fails over once, and the current cluster's TopicPrefix is Primary, then the new DR cluster needs to have its TopicPrefix be Primary.Primary. If it needs to fail over again, the new DR cluster needs to have its TopicPrefix be set to Primary.Primary.Primary. 
Sync an outdated DR cluster 
When the primary cluster executes an IMPORT, DROP ALL, or CLEAR GRAPH STORE GSQL command, or the gsql --reset bash command, the services on the DR cluster will stop syncing with the primary and become outdated.
To bring an outdated cluster back in sync, you need to generate a fresh backup of the primary cluster, and perform the setup steps again. However, you can skip Step 3: Enable CRR on the DR cluster, because CRR will have already been enabled.   
​
Advanced Platform-layer Commands
System Administration FAQs
Last modified 6mo ago
Export as PDF
Copy link
Contents
Step 1: Backup primary data
Step 2: Restore on the DR cluster
Step 3: Enable CRR on the DR cluster
Step 4: Force install a query on primary
Restrictions on the DR cluster
Fail over to the DR cluster
Set up a new DR cluster after failover
Sync an outdated DR cluster