Overview

SymmetricDS Pro 3.12 release includes 13 features, 30 improvements, and 35 bug fixes.

Security Fixes

Issue Summary Severity

2349

Provide more granular level of permissions (Pro)

Medium

2861

Client certificate authentication

Medium

4325

Two-Factor Authentication (Pro)

Medium

4338

Use HTTP/2 for encrypted HTTPS synchronization

Medium

4360

Record node login failures and prevent too many logins

Medium

4361

Encrypt node passwords (Pro)

Medium

4384

Password Complexity Meter (Pro)

Medium

4449

Authentication for JMX and default to localhost only

Medium

4474

Web server require or accept client ssl certificate authentication

Medium

4475

Manage → Security screen import/export certificate authority (Pro)

Medium

Performance Fixes

Issue Summary Severity

4391

Rate staging performance (Pro)

Low

4453

Purge service high CPU for query data range

Medium

What’s New

Design Screen

The web console has a new "Design" tab to visually build configuration from a higher level. The "Configure" tab is still available for editing configuration options.

Log Mining for SQL-Server

Log mining is not available for SQL-Server in additional to trigger-based change capture. It is based on change tracking provided by SQL Server 2008 and newer.

Conflicts by Time of Capture

The new default conflict detection is USE_CHANGED_DATA with a resolver of NEWER_WINS. The NEWER_WINS resolution was enhanced to automatically use the source capture time of the change. (Previously, the user had to setup a detection with USE_TIMESTAMP and specify a timestamp column from the table.) When the target node is version 3.12 or newer, the source node will send the capture time in the CSV protocol, which is used when resolving conflicts. The default for dataloader.apply.changes.only parameter was changed to true, which will help avoid conflicts of a row when different columns are updated.

Modules

A modules system was added to reduce the size of the setup program. There are modules for JDBC drivers, streaming platforms, and the Swagger UI that will download automatically as needed. The most popular drivers (Oracle, SQL-Server, PostgreSQL, MySQL, and H2) are still included in the setup program, but they may become modules in the future. For command line support, see the "symadmin help module" command.

HTTP/2

Jetty was upgraded to support HTTP/2 with ALPN for TLS 1.2 encryption. When configuring server options during installation, HTTP/2 is an enabled option by default. When upgrading, edit the symmetric-server.properties file to enable the https2.enable property. The new protocol is faster, more efficient, and more secure. To continue supporting Java 8 and Android platforms, the okhttp3 library is used for client connections that support HTTP/2.

Node Password Lockout

When a node fails to authenticate too many times in a row, it will be locked out as a security feature. A new field named failed_logins was added to the sym_node_security table, which is incremented for each failed login. A successful login resets the counter back to 0. When failed_login reaches the limit for the node.password.failed.attempts parameter, which defaults to 5, then the log changes from "it had the wrong password" to "it had too many login attempts". Set the failed_logins back to 0 to unlock the node.

Index on SYM_DATA

When upgrading, you’ll see an index on sym_data is dropped and recreated for the columns data_id and channel_id. The index was changed from non-unique to unique, which will help query execution plans associated with routing and extraction of batches to run more efficiently.

Ingres Database

The Ingres database is now supported for change capture, as well as load only and extract only.

Oracle Character Sets

The Oracle triggers were enhanced to handle multiple character sets in the same database. To enable the feature, set parameter oracle.use.ntypes.for.sync=true in the engine file. The triggers will use nchar and nclob types, and the sym_data table is altered to use nclob. As a nice side effect for all Oracle users, the triggers should be slightly more efficient because they now use nvl2() instead of decode() function.

Logging System

The logging was upgraded to use log4j2 logging system. There is now a log4j2.xml file to configure instead of log4j.xml file. The logging format and rotation of log files should continue to work the same way.

Issues

New Features

3.12.0 (Pro)
4291 - SQL-Server log miner
4325 - Two-Factor Authentication
4357 - Modify node registration panel to allow user to authorize opening registration
4384 - Password Complexity Meter
4391 - Rate staging performance

3.12.0
4334 - Provide tool to backup and restore configuration that does not reside in the database
4338 - Use HTTP/2 for encrypted HTTPS synchronization
4356 - Extension points to provide credentials and authorize remote node registration

3.12.1 (Pro)
4416 - Ingres Dialect Implementation

3.12.1
4415 - Ingres Dialect Implementation

3.12.2 (Pro)
4468 - Active Session Screen/Idle Timeout
4477 - New version notification GUI element

3.12.2
2861 - Client certificate authentication

Improvements

3.12.0 (Pro)
1432 - When a user cannot edit a panel, instead of hiding edit buttons, disable the buttons.
1663 - Windows authentication with SQL-Server
1692 - If you can’t edit configuration, at least let you view it read-only
2349 - Provide more granular level of permissions
4301 - Simplify conflicts screen to encourage valid settings
4361 - Encrypt node passwords
4388 - User Email Capture

3.12.0
4167 - sym_router table should require router_type value, providing a default value of 'default'
4192 - Non-unique index on sym_data should be unique
4279 - Remove JAR files that are not used as often to download separately
4302 - Conflict resolution NEWER_TIME based on capture time of row
4303 - Change default of dataloader.apply.changes.only to true to support better conflict resolution
4322 - Upgrade to swagger 2, optional swagger-ui as module
4324 - Upgrade to log4j2
4351 - Change registration to send request parameters as POST instead of GET
4352 - Sybase dialect should enable row locking and set identity gap
4360 - Record node login failures and prevent too many logins

3.12.1 (Pro)
4443 - Uninstalling node causes query errors
4451 - Installer should convert JDBC drivers to modules

3.12.1
4444 - Push job gets error writing request body
4445 - Log4j2 log entries missing engine name, jar in stack trace
4448 - Missing JDBC drivers - add "symadmin module convert" to download drivers as modules
4453 - Purge service high CPU for query data range

3.12.2 (Pro)
4434 - Add GUI support for before trigger scripts
4455 - Setup installer better validation, panel usability, NPE fix, error removing files on upgrade
4475 - Manage → Security screen import/export certificate authority

3.12.2
4457 - Batch conflict possible when blocking row has self referencing foreign key
4460 - Ingres Dialect: Documentation
4465 - Missing JDBC driver for 'org.firebirdsql.jdbc.FBDriver'.
4474 - Web server require or accept client ssl certificate authentication

Bug Fixes

3.12.0
3972 - Tables become out of sync due incorrect conflict detection (and/or invalid conflict resolution)
4226 - Table trigger configuration - Sync on Insert/Delete/Update Condition, wrong Sync Condition example
4229 - [Docs] Invalid column name in "Example 2. Sample Group Links"
4263 - mx4j without auth
4401 - Oracle ntype characters lost from conversion in capture
4402 - Default the file.sync.fast.scan back to false since it only works reliably on Unix

3.12.1 (Pro)
4411 - NoClassDefFoundError during node setup for Kafka, Azure, Snowflake
4422 - Node registration gets javax.crypto.BadPaddingException
4425 - Add load only node does not continue when node is not the master node
4432 - Diagram table foreign key name is too long for Oracle
4438 - Table count should be case insensitive for license check
4447 - MSSQL Log based replication fails on sync triggers

3.12.1
4407 - Monitor Email Notifications Stopped Working
4413 - MSSQL support for binary types in the primary key
4419 - HTTP2 Connection response body not closed
4420 - Error "hostnameVerifier parameter specified as non-null is null"
4421 - Primary key ordering based on the sequence of the PK
4427 - Ingres Dialect: Foreign Key DDL Reader issue and System Generated Indexes
4435 - Kafka key.serializer issue
4436 - Unable to handle unknown csv values: ts
4441 - Kafka module missing during setup of Kafka node.
4442 - Invalid lines in batch with HTTP2
4449 - Authentication for JMX and default to localhost only
4452 - Trigger Rebuild: custom_before_insert_text is ignored when rebuilding triggers

3.12.2 (Pro)
4478 - Manage → Logging panel is slow

3.12.2
4458 - Null pointer exception while extracting batch
4459 - Too many rows sent in reload for table when self-referencing foreign key
4462 - Oracle failed to create trigger, identifier is too long
4463 - Invalid node in sym_table_reload_request causes NPE
4464 - Failed to load batch ORA-00932: inconsistent datatypes: expected - got CLOB
4467 - MSSQL text fields are not permitted in where clause
4473 - Convert to modules, module is already installed
4476 - Kafka avro module missing, no errors from install
4479 - Confluent jackson modules missing from install

3.12.3 (Pro)
4482 - Control Center shows JMX is disabled after enabling from installer

Tables

The following changes were made to the definition of configuration and runtime tables. Table changes are applied to the database automatically using data definition language (DDL) during startup.

New Tables

Table Name Description

sym_console_role (Pro)

Role for users to assign privileges to use screens

sym_console_role_privilege (Pro)

List of privileges for a role

sym_design_diagram (Pro)

sym_diagram_group (Pro)

New Columns

SYM_CONSOLE_USER (Pro)
Column Name Description

otp_key

Two Factor Authentication Key.

is_email_verified

Indicates if an email has been verified.

confirm_code

Confirmation code for email verification and forgot password

confirm_create_time

Create time for a confirmation code

SYM_NODE
Column Name Description

database_name

The database product name identified by SymmetricDS.

SYM_NODE_SECURITY
Column Name Description

failed_logins

Number of failed login attempts

Modified Tables

SYM_DATA
  • Index idx_d_channel_id isUnique changed from false to true

SYM_ROUTER
  • router_type default value changed from null to "default"

  • router_type isRequired changed from false to true

Parameters

The following changes were made to add new parameters, modify their default value, modify their description, or remove them from use.

New Parameters

console.password.otp (Pro)

Required two-factor authentication for login (Default: false)

console.password.verificationcode.timeout (Pro)

Timeout for Verification Codes in Minutes (Default: 15)

console.user.session.timeout (Pro)

Timeout User Sessions in Minutes (Default: 30)

extract.row.capture.time

Extract the capture time of each row and put it in the batch to be used by the conflict manager for picking the winning row during a conflict. Enable for the best precision in resolution, which includes a unix timestamp for each occurrence of insert, update, and delete. Disable to include a single unix timestamp to represent the entire batch, when accuracy is less important or conflict management isn’t needed. (Default: true)

log.miner.mssql.sort.in.memory (Pro)

For SQL-Server log miner, collect changes from all tables into memory, then sort them by change number and foreign keys. This attempts to preserve the order of changes across the database. When the same row is updated multiple times, SQL-Server will merge and return the last change number for it, which prevents accurate ordering. When disabled, the log miner collects and syncs changes from each table ordered by change number, processing each table in order by foreign keys. (Default: true)

log.miner.mssql.sort.in.memory.max.rows (Pro)

The maximum number of rows to sort in memory. When the number of changes exceeds the maximum, it will process like the sort in memory parameter is disabled. (Default: 1000000)

log.miner.mssql.use.tsql.detect.changes (Pro)

Use a single T-SQL statement that returns which tables have changes waiting. The single round trip with SQL-Server can save time. When disabled, a statement is run for each table to check for changes. (Default: true)

node.password.failed.attempts

Number of failed login attempts by a node before lockout (0 = never lockout, -1 = never lockout or record) (Default: 5)

sybase.allow.only.row.level.locks.on.runtime.tables

Automatically alter data, data_event and outgoing_batch tables to allow only row level locking. (Default: true)

sybase.change.identity.gap.on.runtime.tables

Automatically change attribute on data, data_event and outgoing_batch tables to set the identity gap to the number provided. This prevents skipping a large number of identities that can cause routing to stop. Use 0 to disable. (Default: 1000)

Modified Parameters

dataloader.apply.changes.only

Indicates that old data should be used to create the update statement. If old data is equal to the new data and this property is set to true, then no update statement will be run. (Old Default: false) (New Default: true)

file.sync.fast.scan

For Unix, fast scan will look for files that were modified since the last run of file sync tracker and commit their changes using the data loader max commit row setting. When it finds modified directories, it compares to the file snapshot to find changes. For a large file system, this is faster and more efficient than the normal tracker. This setting works on most Unix systems, but reportedly not working on some Windows systems. (Old Default: true) (New Default: false)