Load Testing of Data Replication

Load testing of database replication finds the upper limit of how well the system can perform, and it provides assurance that replication will make it through times of peak usage. Let’s look at how to simulate production activity for SymmetricDS data replication in a lower environment so you can deploy with confidence.

About the Load Test

Our load test will orchestrate many simulated SymmetricDS clients syncing data to an actual SymmetricDS server. We’ll start with a simple batch of data on the “heartbeat” channel that updates the sym_node_host table. The test framework allows additional batches of data on other channels to be added, so it can be customized to use actual data from an environment. The clients are simulated using The Grinder load testing framework.

Build the Project

Clone or download the SymmetricDS Load Test project from GitHub. It requires the Gradle build tool is installed on your system.

First, build the agent and install it to a directory. Each agent will run our tests in the background, using multiple threads and processes. Multiple agents can also be installed across different machines to scale out resources.

$ gradle assemble
$ unzip build/distributions/symmetric-loadtest.zip -d ~/
$ mv ~/symmetric-loadtest ~/loadtest-agent

Second, build the console and install it to the directory. You only need one console to control multiple agents.

$ gradle assemble -Pconsole=true
$ unzip build/distributions/symmetric-loadtest.zip -d ~/
$ mv ~/symmetric-loadtest ~/loadtest-console

Running

Run the console, which opens a graphical application to control the agents and the load test.

$ cd ~/loadtest-console/bin
$ ./symmetric-loadtest

Run each agent, which can be run as a background process. Repeat this step for each agent you installed.

$ cd ~/loadtest-agent/bin
$ ./symmetric-loadtest

Each agent will connect to the console. Under the “Processes” tab of the console, you should see each agent listed there.

Setup the Tests

In the console, go to the “Script” tab. Here are the files included:

• loadtest.py – A Python script that contains a “pull” and “push” test
• loadtest.properties – Configuration file that controls how the tests will behave
• heartbeat.csv – Example batch of data to replicate for the “heartbeat” channel

Edit the “loadtest.properties” file. Either specify the number of runs (0 means run forever) or how long the test should run. This example runs each test 50 times or for 2 minutes, which ever comes first:

grinder.runs=50
grinder.duration=120000

Tell each agent how many threads to use and how often to run each test. Tests run simultaneously across all the threads. This example has each agent using 3 threads, and each thread is running tests every 10 seconds.

grinder.threads=3
time.between.sync.ms=1000

Configure unique node IDs for each agent to use. This example gives three node IDs to each of the three agents.

locations.agent.id.0.process.id.0=5,6,7
locations.agent.id.1.process.id.0=8,9,10
locations.agent.id.2.process.id.0=11,12,13

The SymmetricDS server needs these nodes to be configured in its database. Run SQL on its database to create each node and set its password. The “server.auth.token” property should match the node_password field of the sym_node_security table.

insert into sym_node (node_id, node_group_id, external_id, sync_enabled, created_at_node_id)
values ('5', 'client', '5', 1, 'server');
insert into sym_node_security (node_id, node_password, registration_time, created_at_node_id)
values ('5', 'test', current_timestamp, 'server');

Configure the test to target a SymmetricDS server. It needs to know the node ID and sync URL of the target server that the clients will connect to.

target.node.id=server
server.url=http://localhost:31415
server.path=/sync/server-central

Perform the Load Test

After configuring the test and saving the file, select Distribute -> Distribute Files from the menu to send the configuration out to all the agents. To perform the load testing, select Action -> Start Processes. Now switch to the “Results” tab to watch statistics.

As the database replication runs, the tests per second (TPS) and batches will increase. When a test runs, it may not successful sync a batch of data. The SymmetricDS server can reject data synchronization requests with a “too busy” response, when the number of requests exceeds its http.concurrent.workers.max parameter. The parameter can be adjusted to control an acceptable amount of load on the database.

Going Further

We’ve simulated a load test for SymmetricDS database replication that included “heartbeat” batches of data. Now you can add more channels and batches to the test by configuring them in the “loadtest.properties” file. To add batch files from an actual environment, copy them from the SymmetricDS staging directory (under the “tmp” sub-directory) or save them from the Manage -> Outgoing Batches screen. You can also run your batch processes on the server’s database to simulate changes that the clients will pull and acknowledge. Now you have a working load test harness to build a simulated production load.