Overview

SymmetricDS Pro 3.13 release includes 15 features, 46 improvements, and 74 bug fixes.

Security Fixes

Issue Summary Severity

5191

Upgrade log4j2 again again

Medium

5263

CVE-2022-22965: Spring Version needs to use 5.2.20.RELEASE or 5.3.18 in order to not be vulnerable to this security issue

Medium

Performance Fixes

Issue Summary Severity

3563

Purging large number of extract requests, query time out

Medium

What’s New

User Interface Refresh

The web console was given a new look and feel, with the option to switch between light and dark modes. The underlying UI architecture was upgraded to a progressive web application library based on standard Web Components.

Log Mining for PostgreSQL

Log mining is available for PostgreSQL database in additional to trigger-based change capture. It is based on logical replication slots provided by PostgreSQL 9.4 and newer.

Cosmos

Azure Cosmos non-relational database is now supported for change capture, as well as load-only and extract-only replication. Tables and rows from relational databases are mapped to containers and items on Cosmos.

Elasticseach

Elasticsearch and Amazon OpensSearch document databases are now supported for load-only replication. Tables and rows from relational databases are mapped to indexes and JSON documents.

MongoDB

MongoDB support is enhanced to include change capture, as well as load-only and extract-only replication. Tables and rows from relational databases are mapped to collections and JSON documents.

Faster Startup

The startup time is improved by skipping the steps for checking runtime tables and synchronizing triggers. The step for altering runtime tables is still performed when the software version has changed from the last startup. The step for synchronizing triggers is still performed when a configuration change requires it. The old behavior is still available by setting parameter auto.config.database.fast to false and auto.sync.triggers.at.startup to true, respectively.

Cleaner Shutdown

A shutdown hook was added to properly clean up resources by stopping jobs and closing connections. A cleaner shutdown should help reclaim resources and avoid corruption when using embedded databases.

Purging

Purge was added for trigger history, with a default retention of 30 days. The parameter purge.trigger.hist.retention.minutes was added to control retention. Purge for extract requests was improved to be more efficient for large systems.

Issues

New Features

3.13.0 (Pro)
4943 - Upgrade to Vaadin 14 user interface components
5075 - Elasticsearch Dialect Implementation
5081 - MongoDB change data capture or load-only node and upgraded driver to 4.3.1
5094 - Log miner for Postgres database
5096 - Console event listener extension
5100 - OpenSearch Dialect Implementation
5109 - Microsoft Azure CosmosDB support
5110 - Add Engine Startup Failure Dialog

3.13.0
5015 - Speed up start up by only checking tables when version changes
5099 - Purge trigger history

3.13.1 (Pro)
5129 - Import configuration in the background instead of hanging the UI

3.13.4 (Pro)
5233 - SingleStore dialect
5234 - SingleStore dialect

3.13.4
5213 - Allow to save startup parameters in local keystore or in Azure Key Vault

3.13.5 (Pro)
5267 - SingleStore log based support for non ANSI SQL statements

Improvements

3.13.0 (Pro)
4954 - Installer log which files modified
4963 - Console login is slow
5044 - Enhanced status in the node list
5104 - Uninstaller leaves folder and files behind
5111 - Default parameter console.auth.config.client to true

3.13.0
3563 - Purging large number of extract requests, query time out
3897 - Cleanly exit all job threads when engine is stopped
4800 - Subselect router expression improvement
4955 - Ignore batch status is sometimes overwritten
4958 - Initial load select boolean directly without template
4993 - Remove "databases" sub-directory
4995 - Upgrade Gradle to latest possible version
5098 - Auto sync triggers at startup parameter default off and option to force rebuild
5101 - Changing parameters should not restart all jobs
5107 - Parameter staging.dir should be appended with the engine name
5117 - Increase size of file_name column in sym_file_snapshot

3.13.1 (Pro)
5124 - Add "enabled for initial load" checkbox to table routing form
5125 - Improve appearance of "Starting SymmetricDS" screen
5132 - List registration nodes first in the login dialog
5135 - Improve appearance of sample subselect SQL on subset screen of quick config wizard
5136 - Improve node setup wizard when creating an additional master node
5147 - Improve appearance of group link screen in quick config wizard

3.13.2 (Pro)
5084 - Test Connection button on Manage Nodes screen should be disabled in some situations
5153 - Improve appearance of reminder dialogs

3.13.3 (Pro)
3789 - Sort SQL configuration export by primary key to make it easier to diff
5185 - Upgrade Vaadin to 14.8.1

3.13.3
5179 - Provide wrapper.run.as.user property in sym_service.conf for open source users

3.13.4 (Pro)
5205 - Azure managed instance support for bulk loading
5225 - Alphabetize items in catalog, schema, and table select fields in transform form
5231 - Support snapshot file improvements
5232 - Oracle enhanced permission checks for log based replication
5246 - MSSQL auto create trigger options to exclude rowversion
5249 - Make it clear when links on Dashboard Shortcuts panel aren’t working due to license violations
5251 - Suppress Vaadin’s "Got an RPC for non-existent node" messages
5257 - Rows in the SQL Explorer TreeGrid should only expand or collapse when the arrow is clicked
5262 - Make default node selection in Data Load Wizard less confusing

3.13.4
5200 - Allow specification of local user/password for running Windows service
5204 - MSSQL support for engine edition

3.13.5 (Pro)
5275 - Make "The background refresher service is shutting down" messages debug level
5286 - Rearrange lookup table layout on subset screen of Quick Config Wizard
5287 - Add system memory and jvm bitness to snapshot
5294 - Move images to src\main\resources\META-INF\resources
5297 - Automatically set column width in SQL Explorer results grid
5298 - Open Node Setup Wizard before Quick Config Wizard after initial setup
5299 - Add current node to grid on Nodes dashboard panel

3.13.5
5320 - Use unique index for PK column names in trigger hist when table is missing primary key

Bug Fixes

3.13.0 (Pro)
4953 - Switching nodes changes the column widths of the grids on the Dashboard screen

3.13.0
2970 - db connection leaks after engine stop
3119 - Stopping service breaks connections with error in Interbase logs
3382 - MV-Store DB corrupt after stopping SymmetricDS replication software
4682 - Oracle putting geometry types in where clause for conflict detection
5013 - Registration server config_version not updating
5059 - var_row_data is defined as mediumtext
5102 - Interbase Table SYM_TEMP_CONTEXT
5113 - Data mapping is incorrect on an insert fallback to update potentially when column count on target is not the same as row data

3.13.1 (Pro)
5126 - UI can get stuck on the "Starting SymmetricDS" screen when the server can’t connect to a database but the client(s) can
5127 - Editing staging files via the Manage Staging Area screen doesn’t work if compression or encryption is enabled
5130 - MongoDB quick config support for changing databases to view other collections
5131 - OutgoingLoadPanel is calling a query for each table reload request

3.13.1
5118 - During a full load do not ok existing batches for the same load id.
5145 - Multiple active trigger histories result when more than one trigger assigned to a table
5150 - Upgrade log4j2

3.13.2 (Pro)
5152 - Multiple license expiration reminder dialogs can open at the same time
5154 - Detecting AS/400 as generic dialect instead of AS/400 dialect
5155 - Registration URL field in Node Setup Wizard does not keep its custom value when clicking Next
5156 - Grid on Configure Table Routing screen sometimes doesn’t update after deleting all rows
5165 - Selecting an engine that has failed to start doesn’t open the Engine Startup Failure dialog

3.13.2
5158 - Upgrade log4j2 again
5166 - Initial load stuck in loop with "there is no content to read" error

3.13.3 (Pro)
5170 - Typing a class name in the Advanced Set Logging Levels dialog doesn’t work as expected
5173 - DB2 zOS support for clobs
5181 - Misspelling in message about license maintenance date exceeded
5193 - Can’t reject a registration request in error

3.13.3
5171 - DB2 zOS support for clobs
5191 - Upgrade log4j2 again again

3.13.4 (Pro)
5201 - HSQLDB validation query results in "unexpected end of statement" error
5208 - Design panel errors when transforms are not properly configured
5209 - Auto.config.registration.svr.sql.script parameter doesn’t get removed when setting up SymmetricDS with custom configuration
5215 - Test Connection button fails on Mail Server setup screen
5217 - Buttons in health panel on dashboard sometimes do nothing
5219 - On Login dialog, sort the nodes that are registration servers to the top safely
5220 - Selecting multiple rows on a Configure screen and clicking the New button opens the wrong editor
5221 - Grid selection should not clear when filters change on Configure screens
5223 - Downloading snapshots doesn’t always work in Firefox
5224 - Prevent triggers from being synced via the UI if they’re already in the process of being synced
5230 - Transform form fields do not populate correctly
5235 - Restarting engine from failed engine dialog not working
5236 - Conflict winner can cause routing warning about mismatch table name
5245 - Prevent duplicate Export button from appearing in DB Export dialog
5250 - Fix database icons not always showing up in Design tab
5254 - Assigning a run period via the monitor form’s "Customize" button doesn’t work
5256 - Oracle log miner unsupported dmlType LOB_TRIM
5258 - Catch exceptions when parsing an outgoing batch to view its sent data
5259 - Log based configurations do not sort initial load properly by foreign keys
5261 - Conflict form doesn’t honor console.web.hide.system.info

3.13.4
5194 - DBCompare support for load only target nodes and log based source nodes.
5214 - Column count error when load transform falls back to an insert with a column that is included on insert only
5216 - Metadata error can skip data when using dataloader.ignore.missing.tables parameter
5255 - Monitor doesn’t run when its type requires a cluster lock and it’s supposed to run less often than the monitor job
5260 - Sybase ASE with 2k page file fails to create SYM tables (600 is the maximum allowable size of an index.)
5263 - CVE-2022-22965: Spring Version needs to use 5.2.20.RELEASE or 5.3.18 in order to not be vulnerable to this security issue
5264 - Update any driver, module, or library that has a security vulnerability or serious defect

3.13.5 (Pro)
5266 - OnPrem to Cloud profile configuration not working
5300 - Oracle Log Mining issue with deletes when no primary keys defined
5301 - Manage Logging panel opens log file and never closes when layout in log4j2.xml is different from original installed layout
5303 - izpack failing to find release-notes.html file in stream (too far back)
5310 - MSSQL log based replication support for database names that include a "-"
5316 - Snapshot error on sym_outgoing_batch_summary for load-only node

3.13.5
2423 - Create table with max varchar supported by target
5162 - Keystore becomes invalid after Generating and adding a new Keypair
5272 - Conflict Resolution of deletes of parent records fail to delete child records when child tables have no primary keys
5273 - Rebuild trigger does not take into account the target platform
5274 - Prevent "Failed to get hash code for field delimiter" warning message for SQL Server
5277 - Fix phantom table issue in SQL Explorer database tree
5279 - FK correction for self referencing table.
5285 - Create tables xml needs to escape double quotes on the database type
5289 - Method in AbstractSqlTemplate Causes Recursive Looping with Android
5313 - Null default values become string values when replicating tables from Oracle to MariaDB
5318 - Conflict resolution batch with token parsing error when newlines in data
5327 - Upgrade spring and okhttp

Tables

The following changes were made to the definition of configuration and runtime tables. Table changes are applied to the database automatically using data definition language (DDL) during startup.

New Tables

Table Name Description

sym_console_table_stats (Pro)

Statistics for all tables replicating

New Columns

SYM_INCOMING_BATCH
Column Name Description

conflict_win_count

Not implemented. The number of times a conflict was detected for a row and it was resolved as the winning row.

conflict_lose_count

Not implemented. The number of times a conflict was detected for a row and it was resolved as the losing row.

SYM_OUTGOING_BATCH
Column Name Description

conflict_win_count

Not implemented. The number of times a conflict was detected for a row and it was resolved as the winning row.

conflict_lose_count

Not implemented. The number of times a conflict was detected for a row and it was resolved as the losing row.

Modified Tables

SYM_FILE_INCOMING
  • file_name size changed from 128 to 260

SYM_FILE_SNAPSHOT
  • file_name size changed from 128 to 260

Parameters

The following changes were made to add new parameters, modify their default value, modify their description, or remove them from use.

New Parameters

auto.config.database.fast

If both auto.config.database and this parameter are true when symmetric starts up, it will only try to create the necessary tables if the current software version is different from the version used during the last start up (stored on sym_node.symmetric_version). (Default: true)

auto.sync.triggers.at.startup.force

If this is true, then force rebuild of all triggers at startup (Default: false)

batch.screen.use.batchps (Pro)

Enable use of batchps command over remote status for outgoing batches screen, which uses current list of batches being processed. When disabled, it will use batchstatus command with a list of batches to track over remote status. (Default: true)

console.remote.meta.data.thread.count (Pro)

Number of threads to be used for fetching meta data from remote nodes (Default: 5)

dataloader.ignore.sql.event.errors

Indicate that the data loader should ignore errors while loading a SQL event and the execution of the statement fails. (Default: false)

http.connect.timeout.ms

Sets the connection timeout on the internal HttpUrlConnection (Default: 90000)

mongodb.use.mongo.ids (Pro)

If set assumes all tables replicating with MongoDB will have an _id column which is what MongoDB utilizes. (Default: false)

mssql.auto.create.trigger.options (Pro)

When auto-creating table triggers, some options will be detected and applied for the user, such as excluding ROWVERSION columns. (Default: true)

opensearch.load.aws.access.key

The AWS secret access key (aws_secret_access_key) to use as credentials for uploading to S3 (Default: )

opensearch.load.aws.secret.key

The AWS secret access key (aws_secret_access_key) to use as credentials for uploading to S3 (Default: )

oracle.jdbc.lob.handling

This indicates how to process LOB values. "plain" is the default, which uses the JDBC interface directly. "createtemporarylob" uses the creation of CLOB/BLOB temporary objects to process the data. "streamlob" uses the stream interface of the JDBC interface (part of JDBC 4.0). (Default: plain)

purge.snapshot.file.retention.minutes (Pro)

This is the retention time for how long support snapshot files will be retained (Default: 14400)

purge.trigger.hist.retention.minutes

This is the retention time for how long an inactive trigger history will be retained (Default: 43200)

single.store.audit.log.dir (Pro)

The path to the auditlogs dir for SingleStore (Default: /var/lib/memsql/{cluster-id}/auditlogs)

snapshot.max.batches

Max number of batches to write to statistics listing for support snapshot. (Default: 10000)

snapshot.max.files

Max number of files to write in directory listing for support snapshot. (Default: 50000)

snapshot.max.node.channels

Max number of nodes and channels for batch statistics, after which it will group by node only. (Default: 5000)

sync.triggers.expand.table.clone (Pro)

When sync triggers expands a table name to include node external ID, use the first table found and clone it for all other nodes to improve performance. (Default: true)

trigger.capture.ddl.delimiter

The delimiter to use when capturing changes from a DDL trigger. MS SQL-Server only. See: trigger.capture.ddl.changes (Default: $)

Modified Parameters

auto.sync.triggers.at.startup

If this is true, then run the sync triggers process at startup (Old Default: true) (New Default: false)

console.auth.config.client (Pro)

Allow configuration changes to occur on remote non-registration nodes. (Old Default: false) (New Default: true)

registration.reinitialize.enable

{REMOVED}

instance.id.location

{REMOVED}