Copyright © 2017 JumpMind, Inc

Version 3.2.2

Permission to use, copy, modify, and distribute this Metl User Guide for any purpose and without fee is hereby granted in perpetuity, provided that the above copyright notice and this paragraph appear in all copies.

Preface

This user guide introduces Metl, a simple, web-based integration platform that allows for several different styles of data integration including file based Extract/Transform/Load (ETL), messaging, and remote procedure invocation via Web Services. This guide is intended for users, developers, and administrators who want to install the software, configure integrations, and manage its operation. Thank you to all the members of the open source community whose feedback and contributions helped us build better software and documentation. This version of the guide was generated on 2018-07-20.

1. Introduction

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Metl was built to solve fairly simple day to day integration tasks without the need for custom coding, heavy infrastructure, or high costs. It can be deployed in the cloud or in an internal data center, and was built to allow developers to extend it to fit their needs by writing their own components that can be included and leveraged by the existing Metl infrastructure.

1.1. Overview

Metl is a web application. It can be deployed as a web application archive (war) file to an application server (i.e. Tomcat, JBoss, Websphere) or can be deployed stand-alone as a war file deployed under a bundled Jetty Servlet container. See installation section for details.

Once installed, the web application is accessed via a web browser and is used to design, deploy and manage integrations.

Integrations can be grouped or packaged into one or more projects. Projects are a means to logically group integrations together in some logical way. As an example, a project might contain all of the integrations in or out of a specific system. See projects for additional details.

Projects contain Flows, Models and Resources.

Flows are the integrations themselves. That is, flows allow the definition of data be retrieved from source systems, transformed, and written to target systems. Flows are graphical in nature and allow the developer to configure data movement.

Flows are constructed of components connected by links that describe the path of data through the flow. There are various components available, and custom components can be built and integrated in to the existing Metl infrastructure.

See flows for additional details.

Models provide structure to your data. When dealing with structured data, the data can be defined by modeling. Metl Models allow you to describe your data. Models can either be relational or hierarchical in nature. See models for additional details.

Resources represent connections to physical endpoints where data is read from or written to. The following resource types are available in Metl:

  • Database - A JDBC connection to a JDBC compliant relational database

  • Directories - Connections to file systems. Local, FTP, SFTP and SMB are all supported

  • HTTP Resource - An HTTP connection to REST or SOAP based services

  • Mail Session - An SMTP connection that can be used to send and receive email

See resources for additional details.

1.2. Use Cases

  • Flat File Data Integrations

  • RDBMS Data Integrations

  • Pub / Sub (Queue based) Data Integrations

  • API / Service Based Data Integrations

1.3. Why Metl?

  • Integrations can be configured in hours vs coding them in days or weeks

  • Metl is easy to install. Simply download the .war file and run

  • Metl is web based. No tools to install on each desktop. Deploy locally, in your data center or in the cloud

  • Metl doesn’t require heavy infrastructure. Any windows or linux box will do

  • Metl is open. No black box. Download the source. Write your own components if desired

  • Metl is proven in large integration scenarios

  • Metl is free. The entire toolset is open source under the GPL license

  • Metl is supported. JumpMind, Inc. provides professional support and services for any need

1.4. System Requirements

Metl is written in Java and requires a Java Runtime Environment (JRE) Standard Edition (SE) or Java Development Kit (JDK) Standard Edition (SE). Most major operating systems and databases are supported. The minimum operating system requirements are:

  • Java SE - Runtime Environment or Development Kit version 8 or above

  • Memory - 1 (GB) available

  • Disk - 256 (MB) available

Metl is accessed from a web console, which requires one of the following supported web browsers:

  • Google Chrome 23 or newer

  • Internet Explorer 8 or newer

  • Mozilla Firefox 17 or newer

  • Safari 6 or newer

1.5. License

METL is free software licensed under the GNU General Public License (GPL) version 3.0. See http://www.gnu.org/licenses/gpl.html for the full text of the license. This project includes software developed by JumpMind (http://www.jumpmind.com/) and a community of multiple contributors. METL is licensed to JumpMind as the copyright holder under one or more Contributor License Agreements. METL and the METL logos are trademarks of JumpMind.

2. Installation

Metl is provided as a web application archive (war) file. The .war file can be deployed in two different ways.

  • Deployed stand-alone using Metl’s embedded Jetty server

  • Deployed to an existing servlet container (i.e. Tomcat, JBoss, Websphere, Oracle Application Server, etc.)

Deployed stand-alone using Metl’s embedded Jetty server

The simplest way to run Metl is to run it stand-alone using the internal Jetty container. In order to run Metl stand-alone, simply run the following from a command line.

A Java 8 run-time environment must be installed and in the path to run java.
java -jar metl.war

In the simplest form shown above, Metl will choose a configuration direction based on the logged in user’s home directory (i.e. /users/myusername/.metl) for linux based systems and (c:/users/myusername/.metl) on windows based systems. In this form metl will use a file-based H2 database for for the Metl data store.

Deployed to an existing servlet container

In order to deploy Metl to an existing servlet container, simply copy the metl.war file to the deploy directory of the application server.

Specifying the Metl configuration directory

The default configuration directory can be set by specifying the system property org.jumpmind.metl.ui.init.config.dir.

java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war

Application properties file (metl.properties)

Metl will create a default properties file in its configuration directory if one does not exist. The property file is named metl.properties and has the following content.

db.driver=org.h2.Driver
db.url=jdbc:h2:./metl-dev
execution.retention.time.ms=3600000
execution.retention.time.ms.cancelled=60000
log.to.console.enabled=true
log.to.file.enabled=true
table.prefix=METL
#log.file=

Specifying an alternate database resource for metl configuration and run-time data is as simple as specifying a jdbc driver and url in the metl.properties file.

db.driver=net.sourceforge.jtds.jdbc.Driver
db.url=jdbc:jtds:sqlserver://myserver:1433/mydatabase
db.user=myuserid
db.password=mypassword

By default, Metl uses a DBCP connection pool. In addition to those above, all DBCP configuration settings can be set in the metl.properties file including:

db.validation.query=
db.pool.initial.size=
db.pool.max.active=
db.pool.max.idle=
db.pool.min.idle=
db.pool.max.wait.millis=
db.pool.min.evictable.idle.millis=
db.test.on.borrow=
db.test.on.return=
db.test.while.idle=

Installing Metl as a service

Metl can be installed as a linux/unix or windows service. To install Metl as a service, run the following from a command line.

java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war install

By default, Metl will be installed as a service with service name metl. A service configuration file named metl_service.conf will be written to the configuration directory and can be used to further configure the Metl service.

Starting the Metl service

To start the Metl service, run the following from a command line

java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war start

or, use the operating system tools to start and stop the service. On linux:

service metl start

For windows, use the local services control panel to start the Metl service.

Stopping the Metl service

To stop the Metl service, run the following from a command line

java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war stop

or, use the operating system tools to start and stop the service. On linux:

service metl stop

For windows, use the local services control panel to stop the Metl service.

Uninstalling Metl as a service

To uninstall Metl as a service, run the following from a command line.

java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war uninstall

Once Metl has been started, it can be accessed via a web browser as follows:

http://myservername:myport/metl

If Metl is running on the default port (42000) on the local machine, the url would be as follows:

http://localhost:42000/metl

3. The Metl User Interface

Accessing the Metl User Interface

Once Metl has been started, it can be accessed via a web browser as follows:

http://myservername:myport/metl

If Metl is running on the default port (42000) on the local machine, the url would be as follows:

http://localhost:42000/metl

The following home screen will be displayed:

home

Navigation Menu

The navigation menu shown below is the mechanism for traversing the system.

navigation menu

The Metl UI is split into five major areas:

  • Design - Design and Develop Integrations

  • Deploy - Deploy integrations to a Metl run time agent

  • Manage - Manage integrations that have been run on a given agent including looking at past executions, etc.

  • Explore - Explore resources that are configured for this Metl instance

  • Admin - Administrative management of Metl including defining system users, managing logging levels, etc.

In addition to these five major navigation points, the navigation menu also allows for:

  • Help - Access this help documentation

  • Logout - Log the current user out of the Metl UI

4. Design

The Design screens are used to design and develop integrations. The main design screen is shown below.

design

The design screen is split into two main panels, the navigation pane and the content/editor pane. The navigation pane shown below allows for traversing projects, flows, models and resources.

navigation panel

When Metl is first installed, it comes installed with a sample project that includes sample flows, models and resources. These samples allow the user to immediately see Metl in action. In addition, these sample flows are referenced in the component documentation in order to better understand how individual components work.

The content/editor pane is where flows, resources and models are edited. The screenshot below shows a database resource being edited.

content editor panel

4.1. Projects

Projects are a means to group integrations together in some logical way. Typically, the boundaries of a project should equate to a deployable unit. That means the scope or context of the project should be those flows that must be bundled together when deployed. I.E. if two flows depend on each other and are so tightly coupled that changes to one dictate changes in the other, then those flows should be bundled together in a single project. The project is the level at which flows and their associated resources and models are versioned (see Versions) and imported/exported (see Import/Export).

A new project can be added via the menu.

projects

4.1.1. Versions

Projects can have multiple versions. When a project is first created an initial version is also created with name master. After the master project is released (deployed to production), future changes to the project can be completed in a new version such that each production migration of the project can be tracked. By making each production release of a project under a new project version visibility to differences between versions is also available. New versions of a project can be created from any subsequent version. I.E. if you have versions 1.0.0, 1.1.0, and 1.2.0, a new version can be created from ANY of the previous 3 versions. A patch release (1.1.1) version can be created from the 1.1.0 version while simultaneously making changes to the 1.2.0 version for some future release date. This is very similar to branching and merging concepts that are available in most source control systems.

To create a new version of a project from an existing version, first select the source version from which the new version should be created and then click File New Project Version as shown below.