Running a Data Migration in SymmetricDS

With all of the changes happening in the modern day workplace, the need to migrate data has continued to become more and more prevalent. Between the wide variety of new non-relational database types and the ever expanding cloud options, there are plenty of reasons to want to move your data from one system to another. The hardest part about data migrations, however, is finding the right tool for the job. In this article, we will be discussing:

• What SymmetricDS does
• Why it is the best tool for the job
• How it protects and guides your data to a quick and successful migration

SymmetricDS is software that allows users to migrate and synchronize data between multiple databases of differing or similar dialects. Users could be moving all of their data off of a Postgres server and onto a Kafka server, or synchronizing changes from a SQL Server database to multiple different Oracle servers. Regardless of the exact details, SymmetricDS allows the user to intuitively and quickly move their data from one place to another. Users simply select a database to start with, select the tables that they would like to have replicated, and then select the database node that they would like those tables to go to.

SymmetricDS has wide support for most database platforms. This gives it a lot of flexibility and utility when it comes to moving data around. SymmetricDS also allows for data migration while the source is live and receiving changes. Any changes that are picked up by SymmetricDS are stored in its data capture table. This table will hold onto changes until the data migration is complete. Once the initial loading of the data is complete, it will then start to do whatever changes it had stored. This ensures that while you are completing your initial load, none of the changes on the database are lost. In the case of an outage, the same thing occurs, and the changes are stored until the system has recovered.

When it comes to large databases, they can sometimes store a lot of historical data that isn’t always needed immediately by users or applications connected to that database. SymmetricDS allows users to selectively load data into the application, giving them full control of what is migrated first, and what can be migrated later. This allows customers to load active data on larger databases, so that it is readily available, while loading in the older historical data at a later point. This method allows users to experience smaller downtimes, and have more control over exactly what gets migrated and when.

In the case of a failure during the data migration, SymmetricDS has a built-in automatic recovery operation that will take place. SymmetricDS tracks batches in transactions and retries in the case of a fault. This not only allows for automatic recovery, but also results in a low-maintenance system. This allows the users to have a more stress-free data migration, knowing that if something does go wrong, SymmetricDS is capable of handling it.

Another area SymmetricDS excels in is the speed in which it can move data. Between multi-threading and the use of bulk loaders, SymmetricDS can move hundreds of thousands of rows in the blink of an eye. When migrating data, it is assumed that users will be doing large volumes of inserts into their new platform of choice, and that is where bulk loaders shine. Simply put, a bulk loader is a tool that allows the user to insert large amounts of data quickly. Most databases have these, and SymmetricDS uses them to complete data migrations in the shortest amount of time possible. In addition to the use of bulk loaders, SymmetricDS has the ability to execute many of its actions in parallel, cutting down on the time to replicate even more.

Finally, when it comes to dealing with dependencies in the form of foreign keys, SymmetricDS takes these into account when migrating and syncing your data. SymmetricDS scans the tables that it has been given, and orders them to be loaded in foreign key order. This will prevent any errors due to tables relying on a key that is not present yet. It will also extract parent rows first if the foreign key is self-referencing. In addition, to give the users the fastest migration possible, when creating tables on the target database, SymmetricDS will create them without any foreign keys or indexes, to load the data as fast as possible. Once all of the data has been loaded, it will then add any foreign keys and indexes at the end.

Overall, when doing a data migration, there are a number of things to consider. Although some tools may be the fastest, they might not recover if there is a failure, and some may have the best recovery option out there, but they take way too long to actually get the data into the target. SymmetricDS does an outstanding job of mixing all these factors together to provide users with a well-rounded and pleasant experience, and peace of mind knowing their data is in good hands.