Metl User Guide

*.csv

*.csv,*.txt

select
   field1,
   field2
from
   mytable
where
   field3=:INPUT_MODEL_ENTITY_NAME.INPUT_MODEL_ATTRIBUTE_NAME

select
   field1,
   field2,
   count(*) /* OUTPUTMODEL_ENTITY_NAME.OUTPUT_MODEL_ATTRIBUTE_NAME */
from
   mytable

4,Diane,Prince,F
3,Clark,Kent,M
1,Peter,Parker,M
2,Bruce,Wayne,M

if (ENTITY.ATTRIBUTE=='Value 1') return true;
if (ENTITY.ATTRIBUTE=='Value 2') return true;

if (text.indexOf("Some text string") > 0) return true;
if (text.indexOf("Some other text string") > 0) return true;

binding.variables.containsKey('chainId') && chainId.equals('001')

CHANGE_TYPE == 'DEL'

ENTITY_NAMES.contains("my_entity_name")

<Persons>
    <Person>
        <Id></Id>
        <FirstName></FirstName>
        <LastName></LastName>
        <Gender></Gender>
    </Person>
</Persons>

* ADDRESS1
* ADDRESS2
* CELL_PHONE
* CITY
* FIRST_NAME
* GENDER
* HOME_PHONE
* LAST_NAME
* POSTAL_CODE
* STATE
* WORK_PHONE

* ADDRESS1
* ADDRESS2
* CITY
* FIRST_NAME
* GENDER
* LAST_NAME
* POSTAL_CODE
* STATE

* FIRST_NAME
* LAST_NAME
* HOME_PHONE

* FIRST_NAME
* LAST_NAME

<Persons>
    <Person>
        <Id></Id>
        <FirstName></FirstName>
        <LastName></LastName>
        <Gender></Gender>
    </Person>
</Persons>

String body = "My Text Message Body";
callback.sendTextMessage(null, body);
[id="sequence",reftext="Sequence"]
===== Sequence
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-sequencing-48x48-color.png[]
|Use When|A sequence number needs to be created for a given attribute in a model based record, and the starting value of the sequence can
be derived by a sql statement.
|Samples|
|Description|The sequence component allows an attribute in a model based message to be filled with a generated sequence number.  The initial
value for the sequence is derived based on a sql statement configured in the component.  From that point forward, every new message/record
that comes into the component gets an incremented sequence number placed in the designated sequence number attribute field.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The Database Resource that will be used to execute the Starting Sequence Sql
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Sequence Attribute Name|The name of the attribute in form [entity].[attribute] that should be filled in with the sequence number generated
|Select Starting Sequence Sql|The Sql statement that will be executed in order to retrieve the starting sequence value.  This sql will be run
once on component start up to retrieve the sequence seed value
|Is Sequence Shared|Whether the sequence is shared amongst multiple sequence generators
|Shared Name|If the sequence is shared, this is the name that each of the sequence generators will use to reference the shared sequence
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
[id="serializer",reftext="Serializer"]
===== Serializer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/testtube-orange-48.png[]
|Use When|An entity message needs to be automatically formatted to JSON or XML
|Samples|Format a model object to JSON prior to sending to a link:http-response(Http Response) for a web service
|Description|Formats entity messages based on the input model

|Inbound Message Type|Model Based Message

|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Format|Can have values of AUTOMATIC, JSON or XML.  Default is AUTOMATIC.  If AUTOMATIC is chosen, then the serializer will attempt to look at the flow parameters to determine what format should be used.  If an Http Request started the flow, then the Accept header, the Content-Type header and the format parameter will all be inspected in order to determine the type.
|Structure|Options for how to format the document.  Options are: BY_TABLE and BY_INBOUND_ROW
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|
|===========================================

[id="sorter",reftext="Sorter"]
===== Sorter
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-sorter-48x48-color.png[]
|Use When|Data within a model based message needs to be sorted by one of the attributes within the model or data needs to be sorted by multiple attributes within an entity in the model
|Samples|
|Description|The Sorter is used to sort data elements within a message that is being processed through a flow.

The sort can be defined at either the message level or by entity/attribute using the component editor screen.  When sorting by entity/attribute the data will be sorted per entity, where each entity will be treated as though it is a separate database table and the ordering occurs across the selected attributes (columns).

NOTE: Using the entity/attribute sort option will use an H2 database to load the records into using the model entity as a table and the attributes as columns.  Then it will perform an 'order by' in a sql statement to re-order the records.  The tables will determine the key structure based on the way the entity is modeled in Metl and will use the key to load the H2 table(s).  Word of caution, the key needs to be unique otherwise if your flow allows duplicate records this sort will remove these duplicates.  Also, you may not have null values in the attributes you select as the key to the table.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Sort Entity.Attribute (optional)|The single model entity and attribute to sort the data by.  The entity and attribute must be concatenated with a period (entity.attribute).

NOTE: Either an entry must be defined here or attributes selected in the Component Editor screen for the component to function.  A value in this field overrides any attribute(s) selected on the Component Editor screen.
|<<rows-per-message,Rows Per Message>>|
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

====== Component Editor
Double clicking on the Sorter component in the flow or clicking the Component Editor button will result in the Sorter editor being displayed as shown below.

image:images/editors/sorter-editor.png[]

The Sorter editor displays a row for every entity/attribute contained within the input model and allows the selection of the attributes on which to use as the sort along with the ability to order them.

When a sort checkbox for an attribute is selected, the screen will re-sort all selected attributes to the top by entity.  To change the sort order, select one or more attributes within an entity and use the buttons at the top to Move the selected item(s) up, down, to the top or to the bottom.

NOTE: You can only re-order the attributes within a single entity at any one time.
[id="format-fixed",reftext="Format Fixed"]
===== Format Fixed
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-temp-rdbms-48x48-color.png[]
|Use When|Model based data needs to be staged and queried.
|Samples|
|Description|The Temp RDBMS component stages model based message data in a temporary database to be queried. This component is the same as the <<rdbms-reader,RDBMS Reader>> but instead of providing a data source, the data source is automatically created using the user provided <<input-model,Input Model>>. As model messages are received, they are loaded into the temporary database.

If the <<run-when,Run When>> parameter is configured to 'PER UNIT OF WORK', only one database is created and the SQL expression is executed after all model based messages have been received. If the <<run-when,Run When>> parameter is configured to 'PER MESSAGE', each new model based message will initiate the creation of a new temporary database. The provided SQL will be executed after each message has been loaded into the database. Once the SQL has been executed against the temporary database, the database is cleaned up and removed from memory or the file system.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|<<run-when,Run When>>|
|Sql|The sql query or script that will be executed for this reader. SQL hints are supported.
|<<rows-per-message,Rows Per Message>>|
|In Memory Database|Whether the temporary database should be created in memory or on the file system.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
TIP: If you would like to see the contents of the temporary h2 database, replace the Temp RDBMS SQL with the text "script to '/home/user/temp_rdbms_db.sql'". When this runs, the database contents will be exported to the local file system. After the export completes, Metl will correctly throw an error since the query did not return any results. This should be done for debugging only.

[id="text-replace",reftext="Text Replace"]
===== Text Replace
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-regex-replace-48x48-color.png[]
|Use When|A piece of text in a text based message needs replaced with alternate text
|Samples|
|Description|The text replace component uses a regular expression to replace a piece of text with another piece of text within a text based message.

|Inbound Message Type|Text Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<enabled,Enabled>>|
|Search For (regex)|A regular expression that describes the text for which to search
|Replace With|A text string that should be used to replace the text found by the Search For (regex) parameter
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
[id="transformer",reftext="Transformer"]
===== Transformer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-transform-48x48-color.png[]
|Use When|An attribute in a model based message needs transformed in one way or another (trim, substring, constants, etc.)
|Samples|
|Description|The transformer component allows transformation functions to be applied to record elements from inbound messages.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

====== Component Editor
Double clicking on the Transformer in the flow will result in the Transformer editor being displayed as shown below.

image:images/editors/transformer-editor.png[]

The editor displays a row for every entity and attribute in the input model, and allows and allows transform functions to be applied
to each.  The following transform functions are available:

[options="header" cols="<25%,<75%"]
|===========================================
|Function|Description
|abbreviate(maxwidth)|Abbreviates the attribute using ellipses. This will turn "Now is the time for all good men" into "Now is the time for..."
|currentdate()|Replaces the attribute with the current date and time
|currentdate(format)|Replaces the attribute with the current system date and time formatted based on the format expression.  See <<date-format-expression>> for details on the date format specification.
|daysFromNow(numberDays)|Replaces the attribute with the date of current date + numberDays
|flowParameter(parameterName)|Replaces the attribute wih the flow parameter specified by name parameterName
|format(spec)|Formats the attribute into a string according to the provided spec.  See Java Formatter class for details on the specification string
|formatdate(pattern)|Formats the date attribute according to the pattern parameter.  See <<date-format-expression>> for details on the date format specification.
|getAttributeValueByName(attributeName)|Gets the value of the specified attribute.  Could be used to reference a different attribute from within the current one.
|integer()|Converts the attribute to an integer
|left(length)|Truncates the attribute to the left-most length digits
|lower()|Converts the attribute to lower case
|lpad(padChar,length)|Left pads the attribute with length padChar characters
|messageParameter(paramterName)|Replaces the attribute with the given message parameter specified by name parameterName
|nullvalue()|Replaces the attribute with the null value
|nvl(substituteForNull)|Conditionally replaces the attribute with substituteForNull value if the attribute is null
|parseAndFormatDate(parsePattern,formatPattern)|Parses the string attribute into a date based on the parsePattern and then formats it into a date time string according to the formatPattern parameter
|parseBigDecimal()|Parses the string attribute into a Big Decimal
|parseDouble()|Parses the string attribute into a Double
|parseInt()|Parses the string attribute into an Integer
|parseLong()|Parses the string attribute into a Long
|parsedate(pattern)|Parses the string into a Date based on the pattern.  See <<date-format-expression>> for details on the parse pattern
|replace(searchString,replacement)|Find and replace searchString with replacement. Tip: You can chain replace() calls. e.g., replace("1","a").replace("2","b")
|right(length)|Truncates the attribute to the right-most length digits
|rpad(padChar,length)|Right pads the attribute with length padChar characters
|sequence(seedValue, incrementValue, breakField)|Increments a sequence by incrementValue starting at seedValue.  Restarts at seedValue when breakField changes.  breakField is an attribute name for the current entity record.
|substr(start,end)|Returns the substring of the value beginning with the character at the specified start index and extends to the end index - 1
|stringConstant(value)|Replaces the attribute value with the constant value provided
|trim()|Removes leading and trailing whitespace
|upper()|Converts the attribute to uppercase

|===========================================

[id="date-format-expression",reftext="Date Format Expression"]
Format and parse expressions for dates can be defined using the following designations.

[options="header" cols="10%,30%,30%,30%"]
|============================================================================
|Letter|Date or Time Component|Presentation|Examples
|G|Era designator|Text|AD
|y|Year|Year|1996; 96
|M|Month in year|Month|July; Jul; 07
|w|Week in year|Number|27
|W|Week in month|Number|2
|D|Day in year|Number|189
|d|Day in month|Number|10
|F|Day of week in month|Number|2
|E|Day in week|Text|Tuesday; Tue
|a|Am/pm marker|Text|PM
|H|Hour in day (0-23)|Number|0
|k|Hour in day (1-24)|Number|24
|K|Hour in am/pm (0-11)|Number|0
|h|Hour in am/pm (1-12)|Number|12
|m|Minute in hour|Number|30
|s|Second in minute|Number|55
|S|Millisecond|Number|978
|z|Time zone|General time zone|Pacific Standard Time; PST; GMT-08:00
|Z|Time zone|RFC 822 time zone|-0800
|============================================================================
[id="union",reftext="Union"]
===== Union
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-merger-48x48-color.png[]
|Use When|Inbound messages need to be combined into a single message and then sent to downstream components.
|Samples|
|Description|The union component combines all inbound messages from each input link and then sends one combined message to the downstream
components.  The combination of messages continues until all messages have been processed and the unit of work boundary is reached.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
[id="web",reftext="Web"]
===== Web Request
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-web-48x48-color.png[]
|Use When|A web service (either REST or WSDL) needs to be called to perform an action such as retrieving or writing data
|Samples|WSDL Service Call, DB Request to WSDL to DB results
|Description|The web component allows REST or WSDL based web services to be called, sending the response
data to downstream components.

The Web component can be used to call either WSDL or REST based services.  The Web component takes a text
based message as input.  This text based input message is formatted XML that represents the web service request.
Thus, the web component is frequently used with the XML Formatter component which takes entity data and formats
it into an xml formatted text message.


|Inbound Message Type|Text Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Resource|An Http resource where the service resides and can be called
|<<enabled,Enabled>>|
|<<run-when,Run When>>|When providing the web service request message set this to PER UNIT OF WORK.  When the request message is received from a prior component set this to PER MESSAGE
|Append to URL|Text that will be appended to to the Connection url forming the base url for the web service call
|Body From|Either 'Message' or 'Provided'.  If Message the body of the web service request message will come from
the inbound message.  If 'Provided" the body of the web service request message will come from the Body Text parameter
|Body Text|If the Body From parameter is 'Provided', this parameter is used to specify the body of the text message for the
web service request
|Http Headers|A list of line feed separated Http Headers that will be passed in the web request.  The name value pairs should be separated by a colon (:).
|Http Parameters|A list of line feed separated Http Parameters that will be passed on the URL of the web request.  The name value pairs should be separated by a colon (:).
|Parameter replacement|Enables token replacement for the Command.  Message headers and flow parameters can be token replaced.  The following token formats can be used:  $(HeaderKey) or $(FlowParameter).   Note that you can link:Stamp entity and text message values in a message header if you want to use message values.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
===== XML Formatter
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-xml-formatter-out-48x48-color.png[]
|Description|The XML Formatter component takes an inbound model based message and formats into an xml formatted text message.

|Inbound Message Type|Model Based Message
|Output Message Type|Text Based Message
|===========================================
[id="xslt-processor",reftext="XSLT Processor"]
===== XSLT Processor
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-xslt-48x48-color.png[]
|Use When|XSLT needs applied to a set of input data in order to transform it into a given output format
|Samples|
|Description|The XSLT Processor component applies XSLT to a set of input data and sends the output of the XSLT processing as text
messgaes to downstream components.

|Inbound Message Type|Model Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Output all attributes|
|XML Format|
|Parameter replacement|Enables token replacement for the Command.  Message headers and flow parameters can be token replaced.  The following token formats can be used:  $(HeaderKey) or $(FlowParameter).   Note that you can link:Stamp entity and text message values in a message header if you want to use message values.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

==== Writers
[id="binary-file-writer",reftext="Binary File Writer"]
===== Binary File Writer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-binary-writer-48x48-color.png[]
|Use When|Data needs written to a binary based file
|Samples|
|Description|The Binary File Writer is used to write data to a binary based file.

The Binary File Writer can write to one output file either locally or to a remote destination.

|Inbound Message Type|Binary Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The file based resource (either local or remote) on which the file will be written.
|<<enabled,Enabled>>|
|Path and File|The relative path and file of the file to be written.  The relative path and file are concatenated with the
absolute path specified in the File based resource
|Must Exist|Requires the file to exist prior to writing
|Append|Add input data to the end of an existing file
|Get File Name From Message|If selected, the path and file name to be written will be determined by the payload of the incoming message.
If this setting is not selected, the path and file to be read is based on the File Path property.
|Param Name For Msg Based File Name|If 'Get File Name From Message' property is checked, this is the name of parameter to pull the filename from.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
[id="email-writer",reftext="Email Writer"]
===== Email Writer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-email-48x48-color.png[]
|Use When|An email needs to be sent
|Samples|You have a situation or business event where someone needs to be notified via email.
|Description|The Email Writer is used to send one or multiple emails to one or multiple recipients.

The Email Writer can take one or more input messages.  The writer will be executed according to the 'Run When' setting.  Depending on the setting and the type of input message the email settings (subject, body, to ...) that have tokens will be filled out.

|Inbound Message Type|Any
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|A mail server session.  If this is left blank, then the global mail server session will be used
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Subject|The subject line for the email.  This can be templated
|Body|The body text for the email.  This can be templated
|FROM:|The from line for the email.  If this is not set, the value will be used that is configured in the resource
|TO:|The TO line for the email.  This can be templated.  The TO values can also be provided from a source step
|CC:|The CC line for the email.  This can be templated.  The CC values can also be provided from a source step
|BCC:|The BCC line for the email.  This can be templated.  The BCC values can also be provided from a source step
|Source Step For Email Addresses|This is optional.  If the recipients come from a database or source file this can be used to feed them to the component
|Source Step Email Addresses Type|How to use the source recipients.  Values can be TO, CC, and BCC
|One Email per Recipient|Setting that indicates if multiple emails should be sent or just one
|Run When|When this component should send email
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|
|===========================================
[id="excel-file-writer",reftext="Excel File Writer"]
===== Excel File Writer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-excel-writer-48x48-color.png[]
|Use When|Data needs written to an Excel based file
|Samples|
|Description|The Excel File Writer is used to write data to an Excel based file.

The Excel File Writer can write to one output file either locally or to a remote destination.  Excel files that
are written can be in either Microsoft Excel (.xls) or Microsoft Excel XML (.xlsx) format.

|Inbound Message Type|Entity Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The file based resource (either local or remote) on which the file will be written.
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Excel Output Type|The type of Excel file to generate.  (.xls or .xlsx)
|Must Exist|Requires the file to exist prior to writing
|Empty File On No Input|Output an empty file if no input messages occur
|Path and File|The relative path and file of the file to be written.  The relative path and file are concatenated with the
absolute path specified in the File based resource.  Note, the extension must match the output type selected or flow will
error.  Also, if no extension is provided the component will automatically add the proper extension based on the type.
|Sheet Name|String to use as the Excel worksheet (tab) name.  Must follow Excel guidelines for naming.  If invalid characters
are identified they will be replaced with spaces.  Not providing will use the default of Sheet1.
|Include Header Row|Whether to add a header row with attribute names as column headings in the output file.
|Get File Name From Message|If selected, the path and file name to be read will be determined by the payload of the
incoming message.  If this setting is not selected,
the path and file to be read is based on the File Path property.
|Param Name For Msg Based File Name|Name of the parameter containing the file to be read.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|
|Notes|General notes about this component.

|===========================================
NOTE: The output rows/columns will start in cell A1.
[id="rdbms-writer",reftext="RDBMS Writer"]
===== RDBMS Writer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-writer-48x48-color.png[]
|Use When|Data needs written to a relational database
|Samples|Flat File to Relational Database, DB request to WSDL to DB results
|Description|The RDBMS Writer is used to write data to a SQL compliant relational database by specifying an input model.

The RDBMS Writer can take one or more input messages.  The writer will be executed (i.e. the update and/or insert sql query run) for every
entity record within an input message that it receives.

By default, input records will attempt to be inserted into the defined Input Model.  Parameters can be selected that will allow an update only
or a combination of an update and insert in either order depending on selections.

|Inbound Message Type|Model Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The SQL Database Resource on which the SQL query should be run to write the data.
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Catalog|The catalog where the target table resides
|Schema|The schema where the target table resides
|Replace rows if they exist|Whether to execute an update sql if a unique key exception occurs on the default insert sql attempt
|Update rows first instead of insert|Whether to execute an update sql first
|Fallback to insert if no rows updated|Whether to execute an insert sql if at first the update sql updates no rows
|Quote table and column names|Whether or not to include table and column names in quotes
|Trim character data to fit within column|Whether to trim the character field data if the incoming values length is larger than the target column size
|Continue on Error|Whether to continue processing records on a failure and count the error record in the ignore count
|Batch Mode|Whether to execute the insert and/or update as a transaction allowing rollback capability upon failure (assuming the RDBMS is not an auto
commit system)
|Create table if missing|Whether to create the target table if it does not exist on the database.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
IMPORTANT: Specifying both a Catalog and Schema, the flow will only use the Catalog value and ignore the Schema
Specifying 'Fallback to insert...' has no affect unless 'Update rows first...' is also selected
When no options are checked the default is to attempt an insert only
[id="sql-executor",reftext="SQL Executor"]
===== Sql Executor
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-writer-48x48-color.png[]
|Use When|Data needs written to or deleted from a database using a specific sql statement
|Samples|
|Description|The Sql Executor is used to write data to a SQL compliant relational database by specifying a sql query.

The Sql Executor can take one or more input messages.  The sql will be executed based on the 'Run When' option, either Per Message -
one time per message received, Per Entity - for every entity record in an input message or On Success - one time at flow completion.

|Inbound Message Type|Model Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The SQL Database Resource on which the SQL query should be run to write the data.
|<<input-model,Input Model>>|
|<<enabled,Enabled>>|
|Run When|Timing of when to run the defined sql.  Whether the sql executes after each message received, once per entity record in a message or on flow
completion.
|Sql|Sql statement(s) to run.  Could be an insert, update or delete statement.  Multiple statements can be executed by including a semi-colon (;) between
sql statements.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
[id="text-file-writer",reftext="Text File Writer"]
===== Text File Writer
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-text-writer-48x48-color.png[]
|Use When|Data needs written to a text based file
|Samples|
|Description|The Text File Writer is used to write data to a text based file.

The Text File Writer can write to one output file either locally or to a remote destination.  Text files that
are written can be in any format (fixed length, comma delimited, etc.) and can be formatted by an upstream
processing component like the Format Delimited and Format Fixed components.

|Inbound Message Type|Text Based Message
|Output Message Type|Text Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Resource|The file based resource (either local or remote) on which the file will be written.
|<<enabled,Enabled>>|
|Path and File|The relative path and file of the file to be written.  The relative path and file are concatenated with the
absolute path specified in the File based resource
|Must Exist|Requires the file to exist prior to writing
|Append|Add input data to the end of an existing file
|Line Terminator|String to append to the end of each line written to the file
|<<encoding,Encoding>>|
|Get File Name From Message|If selected, the path and file name to be read will be determined by the payload of the
incoming message.  If this setting is not selected,
the path and file to be read is based on the File Path property.
|Param Name For Msg Based File Name|Name of the parameter containing the file to be read.
|Empty File On No Input|Output an empty file if no input messages occur
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================
IMPORTANT: Specifying None on the Action on Error does NOT stop the flow from failing when a file cannot be written.  It simply
specifies what should be done with the file when the error condition does occur.
[id="zip",reftext="Zip"]
===== Zip
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-zip-48x48-color.png[]
|Use When|Data needs to be put in a compressed file
|Samples|
|Description|The Zip component is used to generate a compressed file of data provided to it.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Source Directory Resource|The file based resource (either local or remote) on which the source files reside.
|Target Directory Resource|The file based resource (either local or remote) on which the compressed file will be written.
|<<enabled,Enabled>>|
|File Path|The relative path and file of the zip file to be written.  The relative path and file are concatenated with the
absolute path specified in the File based resource.
|Delete Source Files|Whether to delete the source files upon completion of creating the compressed file
|Encoding|File encoding type, defaults to UTF-8
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

==== Services
[id="http-request",reftext="Http Request"]
===== Http Request
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-web-48x48-color.png[]
|Use When|A RESTful service needs to be created and hosted by the Metl server
|Samples|Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons
|Description|The HTTP Request component is used to define the expected inbound HTTP Request for a RESTful service that will be hosted by the Metl server
|Inbound Message Type|None
|Output Message Type|Any
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Output Model|If an output model is specified, Metl will attempt to automatically unmarshal the payload of the inbound message (either xml or json)
into the structure specified by the output model definition.  See <<http-request-example-1>> below.
|<<enabled,Enabled>>|
|HTTP Method|The http method for which http requests will be directed to this component.  Either, GET, PUT, POST or DELETE.
|Path|The path definition for which http requests will be directed to this component.  This is a partial path that will be used in conjunction
with the base metl path.  I.E. if Metl is accessed at http://myserver:42000/metl and the path specIfied is /customer/listAll, then HTTP requests to
http://myserver:42000/metl/api/ws/customer/listAll will be directed to this component and flow.

Parameters for a service request can be specified in one of two ways.  The first is with a standard http request parameter.  For example, with a path of
/customer/getById and a base Metl url of http://myserver:42000/metl, http requests to http://myserver/api/ws/customer/getById?customerId=1234 will pass
http requests to the component with customerId available as a parameter.  Parameters can be accessed by enclosing them in $().  I.E. $(customerId).

The second way to specify a parameter is within the url itself.  I.E. with the following path /customer/getById/{customerId} and base metl url of
http://myserver:42000/metl, requets to http://myserver/api/ws/customer/getById/1234 will also pass http requests to the component with customerId available
as a parameter.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

[id="http-request-example-1",reftext="Example 1"]
====== Example 1.  Automatic unmarshelling of xml or json in the http request component

As discussed above, the http request component will attempt to automatically unmarshal the payload of the inbound message (either xml or json) if the
Output Model is specified.  The format of the xml or json must look as follows to be successfully unmarshalled.

XML Example

JSON Example

[id="http-response",reftext="Http Response"]
===== Http Response
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-web-48x48-color.png[]
|Use When|A RESTful service needs to be created and hosted by the Metl server
|Samples|Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons
|Description|The HTTP Response component is used to define the expected outbound HTTP Response for a RESTful service that will be
hosted by the Metl server
|Inbound Message Type|None
|Output Message Type|Any
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|Input Model|If an input model is specified, Metl will attempt to automatically marshal the payload of the entity based inbound message
to the http response component into a json or xml outbound message  See <<http-response-example-1>> below.
|<<enabled,Enabled>>|
|Content Type|The content type of the output.  If an input model is specified, and the http response component is automatically
marshalling the output, the content type will be set automatically based on the accept parameter of the request.  If no input model
is specified and the flow is formatting the outbound message as part of the flow, the content type should be set as approproiate.
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

[id="http-response-example-1",reftext="Example 1"]
====== Example 1.  Automatic marshelling of xml or json in the http response component

As discussed above, if the Inbound Model is specified, the http response component will attempt to automatically marshal
the payload of the inbound entity message into either an xml or json outbound payload.
The format of the xml or json will look as follows:

XML Example

JSON Example

If an input model is NOT specified, the flow will need to format the output in either xml or json format accordingly.
[id="subscriber",reftext="Subscriber"]
===== Subscriber
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-subscriber-48x48-color.png[]
|Use When|You need to consume messages from a Pub/Sub system (like JMS)
|Samples|
|Description|This component requires a resource that supports subscriptions.  When enabled, it will subscribe with the resource.  Each message it receives results in an execution of a flow.
|Inbound Message Type|None
|Output Message Type|Text
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<enabled,Enabled>>|
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================


==== Controls
[id="gate",reftext="Gate"]
===== Gate
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-gate-48x48-color.png[]
|Use When|A component(s) needs to wait until another component has completely finished
|Samples|Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons
|Description|The Gate component passes all incoming messages except for messages from the control source component to a downstream component once the gate component
receives the final unit of work control message from the control source.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<output-model,Output Model>>|
|<<enabled,Enabled>>|
|Gate Control Source|The source component that controls the opening of the gate
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================

[id="last-unit-of-work",reftext="Last Unit of Work"]
===== Last Unit of Work
[cols="<25%h,<75%"]
|===========================================
|Icon|image:images/icons/metl-delay-48x48-color.png[]
|Use When|A component needs to wait until one or more other components complete prior to continuing
|Samples|
|Description|The Last Unit of Work component waits until all end of work control messages are received from all input components before a
message is forwarded to the remaining downstream components.

|Inbound Message Type|Model Based Message
|Output Message Type|Model Based Message
|===========================================

====== Properties

[options="header" cols="<25%,<75%"]
|===========================================
|Name|Description
|<<input-model,Input Model>>|
|<<output-model,Output Model>>|
|<<enabled,Enabled>>|
|<<log-input,Log Input>>|
|<<log-output,Log Output>>|
|<<inbound-queue-capacity,Inbound Queue Capacity>>|

|===========================================


==== Common Properties
[[action-on-error]]
====== Action on Error

Either None, Archive or Delete.  If Delete, the input file will be deleted if there is an error when reading.
If Archive, the input file will be archived to the Archive on Error Path property location if there is an error when reading.  If None, the
input file will be left in its original position if there is an error when reading.

[[achive-on-error-path]]
====== Archive on Error Path

If Action on Error is set to Archive, this property sets the archive location where the input file should
be archived if there is an error when reading.

[[action-on-success]]
====== Action on Success

Either None, Archive or Delete.  If Delete, the input file will be deleted after it is successfully read.
If Archive the input file will be archived to the Archive on Success Path property location.  If None, the input file will be
left it its original location once read.

[[archive-on-success-path]]
====== Archive On Success Path

If Action on Success is set to Archive, this property sets the archive location where the input file
should be archived once successfully read.

[[cancel-on-no-files]]
====== Cancel On No Files

If there are not files to process then cancel the entire flow (versus putting the flow in ERROR status.)

[[enabled]]
====== Enabled

Whether the component is enabled for the given flow.

[[encoding]]
====== Encoding

The encoding to use when reading or writing of text data.


[[inbound-queue-capacity]]
====== Inbound Queue Capacity

The number of pending inbound messages this component can store as it processes other requests.

[[input-model]]
====== Input Model

The input <<models,model>> that describes input data sent to this component.

[[log-input]]
====== Log Input

Whether to the log input messages to this component to the log file.

[[log-input]]
====== Log Input

Whether to the log output messages from this component to the log file.

[[output-model]]
====== Output Model

The output <<models,model>> that describes data sent from this component.

[[rows-per-message]]
====== Rows/Msg

The number of entity rows that will be packaged into a single message.

[run-when]
====== Run When

Indicates when the component will run.  Most components have options of:

|===========================================
|*PER UNIT OF WORK*|The component will run when it receives a control message.
|*PER MESSAGE*|The component will run when it receives a content message.
|===========================================
[[shared]]
====== Shared

Whether this component definition is shared across multiple flows.


[id="models",reftext="Models"]
=== Models

Data integrated with Metl can be structured or unstructured data. When dealing with Structured data, the structured data
can be defined by modeling the data. Metl Models allow you to describe that structured data. Models can either be relational or
hierarchical in nature.  All structured data must be modeled in Metl.  For example, a delimited file may exist that looks as follows:

This file represents people including their last name, first name, gender and date of birth.  In order to use this file in a structured
way, you must model the data contents of the file.  I.E. define an entity named something like Person, and then define attributes for that person
(lastName, firstName, gender, dateOfBirth).

To create a new model, click on the "Models" folder in the navigation pane, and then click "New" - "Model" as shown below.

image:images/screenshots/design/models-new-model.png[width=300]

Once the model has been created, double click the model in the navigation pane in order to edit the model.  From the content editor pane,
there are buttons for adding and entities and attributes as well as positing those attributes within the model.
Highlighting an entity and clicking the "Add Attribute" button will add an attribute to the given entity.

image:images/screenshots/design/models-edit-model.png[]

[discrete]
==== Importing Models from a Relational Database

Similarly when dealing with data in a relational database, there is an inherent model defined in the relational database itself.  I.E.
tables and columns equate to entities and attributes.  Metl models for relational databases can be automatically imported from the database
structure itself, or they can be maintained manually.  In order to import a model from a relational database, click the "Import..." button.  The
following screen will be displayed which displays all database resources.  Drilling into a database resource allows the user to multi-select
one or more tables in order for those tables to be imported into the model.

image:images/screenshots/design/models-import-model.png[width=400]

[id="resources",reftext="Resources"]
=== Resources

Resources represent connections to physical end-points where data is
read from or written to.  A resource is configurable per environment (agent). This allows a flow to be tested against a
resource configured for testing and then deployed to an agent with resources configured for production.


The following resource types are available in Metl:

[options="header"]
|===========================================
|Rsource Type|Description
2+|Database
|<<sql-database,Database>>|A JDBC connection to a JDBC compliant relational database
2+|Directory
|<<localfilesystem,Local File System>>|A connection to a local file system
|<<ftp,FTP>>|An FTP connection to a file system
|<<sftp,SFTP>>|An SFTP connection to a file system
|<<smb,SMB>>|An SMB connetion to a server
|<<jms,JMS>>|A JMS connection to a specified queue and topic
2+|HTTP(S)
|<<webresources,Web Resources>>|An HTTP(S) connection to REST or SOAP based services
2+|Mail Session
|<<mailsession,Mail Session>>|An SMTP connection to a Mail server
2+|Subscribe
|<<jms,JMS>>|A JMS connection to a specified queue or topic that can be subscribed to
|===========================================

Resources are defined on a project by project basis.  In order to create a new resource, Click on the "Resources" folder
in the Navigation pane, click the "New" menu item, and then select the type of resource to be created.

image:images/screenshots/design/resources-new.png[width=300]

If resources will be shared
across multiple projects, a project can be defined that only contains resources, and then that project can be used as a Project Dependency
for other projects that will leverage the shared resources.

When a flow is deployed, its associated resources are automatically
deployed with it.  A flow, and thus its associated resources, can be deployed to one or more Metl agents.  Each deployment
of that resource on a given agent can have different settings.  For example, companies typically have several environments for a
given system (i.e. development, test, user acceptance test, production).  A flow and its resources can be deployed to the
development agent/environment with settings specific to the development environment (i.e. server names, ids, passwords, etc.)  When
that same flow is deployed to the test agent/environment the resources on the test environment can be changed for the specifics
of that environment.  The flow itself stays the same regardless of the environment in which it is running.  This allows seamless
migrations from environment to environment.

To edit the settings of a resource, double click the resource in the navigation pane, and the settings details will be displayed
in the content pane.

[[sql-database]]
==== Database

The following settings are available for the Database resource

[options="header" cols="<25%,<75%"]
|===========================================
|Setting|Description
|Driver|The fully qualified Java class name of the JDBC driver to be used
|Url|The connection URL to be passed to the JDBC driver to establish a connection
|User|The connection username to be passed to the JDBC driver to establish a connection
|Password|The connection password to be passed to our JDBC driver to establish a connection
|Validation Query|The SQL query that will be used to validate connections from this pool before returning them to the caller.
If specified, this query MUST be an SQL SELECT statement that returns at least one row. If not specified, connections will be
validated by calling the isValid() method
|Initial Size|The initial number of connections that are created when the pool is started
|Max Active|The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit
|Max Idle|The maximum number of connections that can remain idle in the pool, without extra ones being released, or negative for no limit
|Min Idle|The minimum number of connections that can remain idle in the pool, without extra ones being created, or zero to create none
|Wait Time (ms)|The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be
returned before throwing an exception, or -1 to wait indefinitely
|Evict Time (ms)|The minimum amount of time an object may sit idle in the pool before it is eligable for eviction by the idle object evictor (if any)
|Test on Borrow|The indication of whether objects will be validated before being borrowed from the pool. If the object fails to validate, it will be
dropped from the pool, and we will attempt to borrow another
|Test on Return|The indication of whether objects will be validated before being returned to the pool
|Test while Idle|The indication of whether objects will be validated by the idle object evictor (if any). If an object fails to validate, it will be
dropped from the pool
|Init Sql|An SQL statement that will be sent to the jdbc driver upon each connection to the database.  Can be used for custom initialization
|Fetch Size|The number of rows to fetch at a time for a sql select statement
Query timeout (seconds)|The allowable time a query can run without completing before being cancelled
Connection Properties|The connection properties that will be sent to our JDBC driver when establishing new connections.
Format of the string must be [propertyName=property;]
|===========================================

===== Adding a JDBC Driver .jar file

Metl comes pre-packaged with the following jdbc drivers:

* SQL Server jTDS jdbc driver
* H2 jdbc driver
* Postgres jdbc driver

For other databases such as Oracle, jdbc drivers can be downloaded from the respective vendor and added to the classpath
of Metl.  Adding the jdbc driver to the classpath depends on whether Metl is being run as a service.

If Metl is being run as a service:

* Download the jdbc driver and place it on the local file system
* Edit the Metl metl_service.conf file and change the wrapper.java.classpath line in the file.

TIP: The separator for the classpath shown above depends on the operating system.  For OSX and Linux, use ":", for Windows use ";"

If Metl is NOT being run as a service, set the classpath to include the additional jdbc drivers as well as the metl.war and launch metl by specifying the class name within the .war file as shown below.

===== Example database drivers and urls

===== In Memory H2 Database

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|org.h2.Driver
|Url|jdbc:h2:mem:mydatabasename
|===========================================

===== File based H2 Database (linux, osx)

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|org.h2.Driver
|Url|jdbc:h2:~/foldername/databasename
|===========================================

===== File based H2 Database (windows)

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|org.h2.Driver
|Url|jdbc:h2:C:/foldername/databasename
|===========================================

===== SQL Server Using JTDS driver

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|net.sourceforge.jtds.jdbc.Driver
|Url|jdbc:jtds:sqlserver:/servername.domainname:1433;databaseName=databasename;
|===========================================

===== Oracle

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|oracle.jdbc.OracleDriver
|Url|jdbc:oracle:thin:@//servername.domainname:1521/servicename
|===========================================

===== Postgres

[options="header" cols="<25%,<75%"]
|===========================================
|parameter|value
|Driver|org.postgresql.Driver
|Url|jdbc:postgresql://servername:5432/databasename
|===========================================

===== MySql

==== Local File System

The following settings are available for the Local File Resource

[options="header" cols="<25%,<75%"]
|===========================================
|Setting|Description
|Path|The base path that should be used when accessing files from this resource
|===========================================

==== FTP

The following settings are available for the FTP resource

[options="header" cols="<25%,<75%"]
|===========================================
|Setting|Description
|Server|The name of the FTP server including domain name.  i.e. servername.domainname
|Port|The TCP port over which to communicate.  Default FTP port is 21
|User|The user name that should be used to connect to the FTP resource
|Password|The password that should be used to connect to the FTP resource
|Base Path|The base path that should be used once the connection has been made
(i.e. the connection will default to this base directory on the server for subsequent requests)
|Connection Timeout|The amount of time to wait for connection before canceling the connection
|===========================================

==== SFTP

[options="header" cols="<25%,<75%"]
|===========================================
|Setting|Description
|Server|The name of the SFTP server including domain name.  i.e. servername.domainname
|Port|The TCP port over which to communicate.  Default SFTP port is 22
|User|The user name that should be used to connect to the SFTP resource
|Password|The password that should be used to connect to the SFTP resource (if password is used to authenticate)
|Key File Location|The full path and file name of a key file that should be used for authentication (if public key authentication is used to authenticate)
|Base Path|The base path that should be used once the connection has been made
(i.e. the connection will default to this base directory on the server for subsequent requests)
|Connection Timeout|The amount of time to wait for connection before canceling the connection
|===========================================

==== Web Resources

The following settings are available for Web Resources

[options="header" cols="<25%,<75%"]
|===========================================
|Setting|Description
|Url|The URL of the web resource to be utilized
|Http Method|The Http method t be sent for requests sent to this resource, either GET, PUT or POST
|Http Timeout|The max time to wait for a response from the request before canceling the request
|Content Type|The content type for the body for requests sent to this resource
|Security|Either None, Basic Auth, OAuth 1.0 or Token
|User|If Security is set to Basic Auth, the user name to be used to authenticate requests sent to this resource
|Password|If Security is set to Basic Auth, the password to be used to authenticate requests sent to this resource
|Token|If Security is set to Token, this is the token provided by the web service provider to be used to authenticate requests sent to this resource
|===========================================
[id="release",reftext="Release"]
== Release
[id="deploy",reftext="Deploy"]
== Deploy

Once flows have been designed and configured, they can be deployed to a Metl agent to be run.  When a flow is deployed to an
agent, its associated resources are also deployed to that agent and can be configured differently for each agent deployment.
In a typical scenario, a company might have application environments for development, system test, user acceptance test and
production.  Similarly with Metl, you can have a development, system test, user acceptance test and production Metl agents.
After a flow has been designed and configured, it can be deployed to the development agent for dev testing.  The resources
that are deployed on the Metl development agent can point to the external systems development resources.  When the flow is ready
for system testing, it can be deployed to a Metl system test agent, and the associated resources on that agent can point to
the external systems system test resources, etc.

Each running instance of the Metl application can have one or more Metl agents running underneath it.  As an example,
development, test and production agents could all be run under a single running instance of the Metl application.  More commonly
each agent is deployed on a different physical server or virtual machine.

When a flow is run from the design screen, Metl uses a pre-configured agent on the current running instance of Metl called
the design-time agent.  This agent is like any other Metl agent except that it is created automatically.

Similar to the design screen, the deploy screen is split into different panels.  The navigation panel show below,
allows viewing of the Metl agents that are configured and running for the current instance of the Metl application.

image:images/screenshots/deploy/navigation-panel.png[width=270]

Agents are organized within folders.  To create a new folder, click "New" - "Folder", and type the new name of the folder.

image:images/screenshots/deploy/new-folder.png[width=270]

Similarly, to create a new Metl Agnet, click on the folder under which you would like the agent organized, click "New" - "Agent",
and type the name of the new agent.

image:images/screenshots/deploy/new-folder.png[width=270]

Double clicking on an agent in the navigation panel, or selecting an agent and clicking "Edit" - "Open" will open the agent
details in the content editor panel as shown below.

image:images/screenshots/deploy/agent-details.png[]

The Start Mode for each agent can be "AUTO" or "MANUAL".  When in "AUTO" mode, all agents defined in this instance of the Metl application
will be started automatically when the Metl software is run.  The Hostname lists the computer's hostname on which the agent is being run.

Agent specific parameters can be set on the Agent screen.  Clicking the "Parameters" button displays a dialog box where these parameters
can be maintained.  These parameters are available to flow components and can be used when things (in addition to resources) need to be
specified on an environment by enironment basis.

image:images/screenshots/deploy/agent-parameters.png[width=570]

The main list on the content editor panel of the deploy screen displays the flows and resources that have been deployed
to a given agent.

image:images/screenshots/deploy/agent-deploys.png[]

[options="header" cols="<25%,<75%"]
|===========================================
|Column Name|Description
|Project Name|The name of the project to which the deployed item belongs
|Deployment|The name of the item that has been deployed to this agent.  This can be a resource name or a flow name
|Type|The type of the item that has been deployed.  Either "Resource" or "Flow"
|Status|The status of the deployed item.  Either "DISABLED" or "ENABLED".  If "ENABLED" the flow will can be run on a
scheduled or manual basis from the agent.  If "DISABLED" the job must be enabled before it can be run.  This column is only
applicable for a flow.
|Log Level|The log level for the deployed item.  Either "DEBUG", "INFO", "WARN", or "ERROR".  Determines the log level that will
be output for the flow when the flow is run on the agent.  This column is only appicable for a flow.
|Start Type|Determines how the flow will be started or run on the agent.  Either "MANUAL", "ON_DEPLOY", or "SCHEDULED_CRON".  If
"MANUAL" the flow will not be invoked automatically by the system, but can be run manually by a user from the deploy screen.  If
"ON_DEPLOY", the flow will be run immediately after the flow has been deployed and enabled.  If "SCHEDULED_CRON", the flow will be
run on a scheduled cron basis that can be maintained by double clicking the flow in the list.
|Start Expression|If Start Type is set to "SCHEDULED_CRON", this is the cron expression that will determine the run time of the job
|===========================================

[discrete]
=== Deploying a flow

To deploy a flow to an agent, click on the "Add Deployment" button and the Add deployment screen will be displayed as show below:

image:images/screenshots/deploy/agent-deploy-flow.png[width=520]

All projects and flows will be displayed.  Selecting a flow, and clicking the "select" button will deploy the flow to the Metl agent.
When a flow is first deployed to an agent, its status will be "Disabled."  To enable the flow for the agent, select the flow, and
click the "Enable" button.  The flow status will be changed to "REQUEST_DEPLOY" and once deployed will be changed to "DEPLOYED."

[discrete]
=== Running a flow

Once a flow has been deployed to an agent and enabled, the flow can be run from the deploy screen.  Select the flow to be run, and
click the "RUN" button.  After the flow has run, the flow's run statistics will be displayed as follows.

[options="header" cols="<25%,<75%"]
|===========================================
|Column Name|Description
|Component Name|The name of the component in the flow
|Thread|The number of threads that were used when running the component
|Status|The completion status of the component.  Either "DONE", "ERROR", or "CANCEL".  If "DONE", the component completed all of its
work successfully.  If "ERROR", the component finished in error.  If "CANCEL" the component was cancelled before it could complete its work
|Msgs Rcvd|The number of messages this component received during the run of the flow.
|Msgs Sent|The number of messages this component sent during the run of the flow
|Entities Prcd|The number of entities processed (if the component takes a model based inbound message)
|Start|The start run time of the component
|End|The end run time of the component
|===========================================

Selecting the component in the top pane will result in run information for that component being display in the bottom pane.  Run information
for the component can be filtered by typing a filter expression in the Filter box.

[id="enterprise-scheduler",reftext="Running a flow from an enterprise scheduler"]
[discrete]
=== Running a flow from an enterprise scheduler

Metl has a REST API that can be used to call a deployed flow from an external enterprise scheduler.  Information about the Rest
API can be found here <<rest-api>>.  The Rest service allows an HTTP GET request to run a flow and get a response back
that provides the status of the flow run.

One way to initiate the REST API is to create a .bat file using cURL as shown in the example below.

[id="manage",reftext="Manage"]
== Manage
The Manage screen allows viewing run information for flows.  The manage screen is shown below:

image:images/screenshots/manage/manage-home.png[]

The Manage screen is split into two panels, the navigation panel on the left hand side of the screen and the content panel on the right hand side of the screen.  The navigation panel allows selection of the flows to be viewed.  The following options are available.

[options="header"]
|===========================================
|Option|Description
|_Currently Running_|Displays all flows that are currently executing on any Metl run-time agent in this Metl instance
|In Error|Displays all flows that are currently in an error status on any Metl run-time agent in this Metl instance
|Agents|Allows selection of flows to be viewed based on the run-time agent to which they are deployed.  When a specific run-time agent is selected in the navigation panel, all flows that have been run on that agent will be displayed in the content panel.
|Flows|Allows selection of flows to be viewed based on flow name.  When a flow name is selected, every run of the given flow will be listed in the navigation panel.
|===========================================

Double clicking a row/run in the content panel will display the run statistics for that run of the flow.  There are two modes for displaying run statistics for a flow, _table_ and _graphical_.  If the _Show Diagram_ checkbox is checked, a graphical representation of statistics for the flow will be displayed.  If it is not checked, a tabular representation of the stats will be displayed.  Both representations are shown below.

Tabular

image:images/screenshots/deploy/agent-run.png[]

Graphical

image:images/screenshots/deploy/agent-run-graphical.png[]

In the tabular representation, the following data is available:

[options="header" cols="<25%,<75%"]
|===========================================
|Column Name|Description
|Component Name|The name of the component in the flow
|Thread|The number of threads that were used when running the component
|Status|The completion status of the component.  Either "DONE", "ERROR", or "CANCEL".  If "DONE", the component completed all of its
work successfully.  If "ERROR", the component finished in error.  If "CANCEL" the component was cancelled before it could complete its work
|Msgs Rcvd|The number of messages this component received during the run of the flow
|Payload Rcvd|The number of content messages this component received during the run of the flow
|Msgs Sent|The number of messages this component sent during the run of the flow
|Payload Sent|The number of content messages this component sent during the run of the flow
|Entities Prcd|The number of entities processed (if the component takes a model based inbound message)
|Start|The start run time of the component
|End|The end run time of the component
|===========================================

Selecting the component in the top pane will result in run information for that component being display in the bottom pane.  Run information
for the component can be filtered by typing a filter expression in the Filter box.

== Explore

The explore screen is for users to explore deployed resources.  Metl currently supports
exploring Sql Datasources via a _Sql Explorer_ and Directory based resources via a _Directory Explorer_.

== Admin

The administration screens allow setup of admin data including:

* <<users,Users>> - Managing users
* <<groups,Groups>> - Managing User groups
* <<rest,REST>> - Displaying information about the REST API
* <<generalsettings,General Settings>> - Managing purge, auto backup and password settings
* <<pluginrepositories,Plugin Repositories>> - Managing maven repositories from which plugins are downloaded
* <<mailserver,Mail Server>> - Configuring the SMTP mail server
* <<notifications,Notifications>> - Configuring notifications
* <<activeusers,Active Users>> - Displaying active users
* <<auditevents,Audit Events>> - Displaying system events such as logins, restarts, config imports, etc.
* <<logging,Logging>> - Displaying Metl log information
* <<about,About>> - Displaying information about the running Metl system


=== Users

Selecting "Users" in the Admin navigation panel displays a list of Metl users in the content panel as shown below.

image:images/screenshots/admin/admin-users.png[]

Clicking on the "New" button allows creation of a new user, while clicking on the "Edit" button allows editing of an existing
user.  In either case the edit user dialog will be display as follows:

image:images/screenshots/admin/admin-edit-users.png[width=550]

[options="header" cols="<25%,<75%"]
|===========================================
|Field|Description
|Login ID|The ID the user will use to log into the Metl user interface
|Full Name|The full name for the user.  For documentation purposes only
|Password|The password the user will use to log into the Metl user interface
|Available Groups|The available groups configured in the "Groups" section that are available to be assigned to the user
|Selected Groups|The groups to which this user currently belongs
|===========================================

=== Groups

Selecting "Groups" in the Admin navigation panel displays a list of Metl groups in the content panel as shown below.

image:images/screenshots/admin/admin-groups.png[]

Clicking on the "New" button allows creation of a new group, while clicking on the "Edit" button allows editing of an existing
group.  In either case the edit user dialog will be display as follows:

image:images/screenshots/admin/admin-edit-group.png[width=550]

[options="header" cols="<25%,<75%"]
|===========================================
|Field|Description
|Group Name|The name of the group that will be availble to assign to users
|Available Privileges|The available privileges in the system.  The available privileges are "DESIGN", "DEPLOY", "MANAGE", "EXPLORE",
and "ADMIN".  Each of these privileges allows the user to access the the corresponding high level screens in Metl and perform the
actions on those screens.  For example, design privileges allow the user to design new flows.  Deploy allws users to deploy existing
flows to Metl agents.  "Manage" allows users to look at run time data for flows running on the given agent, etc.
|Selected Privileges|The selected privileges the user group will have.  The default user group has access to all privileges.
|===========================================

[id="rest-api",reftext="Rest API"]
=== REST

Selecting "REST" in the Admin navigation panel displays a list of Metl Rest Services in the content panel as shown below.

image:images/screenshots/admin/admin-rest.png[]

The current REST service that is available for Metl is a service to allow flows deployed on an agent to be run by invoking
the REST service.  This is used frequently when scheduling a Metl flow with an existing enterprise scheduler.  Documentation
for the service includes:

* HTTP Method - GET
* URL - /api/agents/{agentName}/deployments/{deploymentName}/invoke.  {agentName} should be replaced with the agent that hosts
the flow to be run, and {deploymentName} should be replaced with the flow name to be run.
* Parameters - agentName and deploymentName from above
* Example Response - The response that is sent by the REST service after invoked.

See <<enterprise-scheduler>> for an example.

=== Mail Server

Metl allows notifications to be sent to an email address or distribution list for certain flow run statuses.  See <<notifications>>
for details.  In order for notifications to be sent, the mail server must be configured.  Selecting "Mail Server" in the Admin
navigation panel allows the mail server to be configured in the content panel as shown below.

image:images/screenshots/admin/admin-mail-server.png[width=550]

[options="header" cols="<25%,<75%"]
|===========================================
|Field|Description
|Host Name|The server name on which the mail server resides
|Transport|The transport over which to connect to the mail server.  Either "SMTP" or "SMTPS".  SMTPS is secure.
|Port|The port on which to connect to the mail server
|From Address|The email address notification emails should be sent FROM
|Use TLS|Whether to use transport layer security protocol over SMTP to provide certificate based authentication.
|Use Authentication|Whether the mail server requires authentication
|Username|The username to be used to connect to the mail server if authentication is required
|Password|The password to be used to connect to the mail server if authentication is required
|===========================================
[id="notifications",reftext="Notifications"]
=== Notifications

Metl allows notifications to be sent to an email address or distribution list for certain flow run statuses.  In order for
notifications to be sent, the notifications must be configured.  Selecting "Notifications" in the Admin
navigation panel shows a list of configured notifications in the content panel as shown below.

image:images/screenshots/admin/admin-notifications.png[]

Clicking on the "New" button allows creation of a new notification, while clicking on the "Edit" button allows editing of an existing
notification.  In either case the edit notification dialog will be display as follows:

image:images/screenshots/admin/admin-edit-notification.png[width=550]

[options="header" cols="<25%,<75%"]
|===========================================
|Field|Description
|Level|The level at which this notification applies.  Either "GLOBAL", "AGENT", or "DEPLOYMENT".  If "GLOBAL" the notification
will be sent for all flows and all agents in this instance of Metl.  If "AGENT", the notification will be sent for the agent
specified by the "Linked To" field, and if "DEPLOYMENT", the notification will be sent for the flow and specified agent in the
"Linked To" field.
|Linked To|If "AGENT" or "DEPLOYMENT" is specified for the "Level" field, this field specifies for which agent or deployment (agent and
flow) the notification should fire
|Event|The event for which the flow should fire.  Either "FLOW_START", "FLOW_END" or "FLOW_ERROR".  This field is used in conjunction
with the "Level" and "Linked To" fields to determine when to fire events.  "FLOW_START" indicates a notification should be fired
when the flow begins, "FLOW_END" when the flow finishes (either successfully or on error), and "FLOW_ERROR" when the flow finishes
with an error condition.
|Name|The name of the notification
|Recipients|The recipient list.  One or more email addresses may be entered
|Subject|The subject message that should be displayed for the email notification
|Message|The message body for the email notification
|Enabled|Whether this notification is enabled or not.  If not enabled, the notification will not be sent
|===========================================

[discrete]
==== Notification variables that can be used in Subject or Message

There are several variables that can be used in the notification subject and message.  These are specifically helpful when the
notification Level is at the Agent or Global level as details can be added that specify which flow fired the notification.

* $(_flowName) - The name of the flow that fired the notification message
* $(_agentName) - The name of the agent that fired the notification message
* $(_date) - The date the notification message was fired
* $(_time) - The time the notification message was fired


=== Logging

Selecting "Logging" in the Admin navigation panel shows Metl's log file as shown below.

image:images/screenshots/admin/admin-logging.png[]

The log file can be filtered or downloaded by using the appropriate fields on the logging screen.

Unresolved directive in user-guide.asciidoc - include::examples/examples.asciidoc[]
=== Creating RESTful Services
Metl can be used to create and host RESTful services.  Creating a RESTful service in Metl is as simple as creating
a flow and deploying it to one of the configured agents.  Metl comes with several sample flows that show the creation
of RESTful services.  The samples can be reviewed and run from the Samples project.

The core of Metl's service capabilities come from the <<http-request,Http Request>> and <<http-response,Http Response>>
components.

Following are instructions on how to create and host a REST service that lists all persons from a relational database.
Metl REST services are created by configuring a Metl flow to accept the Http request, using other components to complete
the logic of the flow, and then using the Http response component to return the results to the caller.

The first step in creating the sample REST service is to create the resource that will be used to gather the person data from the database.
For this sample, an in memory H2 database will be used and is configured as follows:

image:images/screenshots/examples/sample-database-resource.png[width=350]

Next, a model must be created to represent a person.  In this example, the structure of the data
of a person will be the same in the relational database source and the response of the REST service, so the same model will be
used for both.  Different models for each could be used and mapped within the flow, but for this sample, we'll keep things simple.
The person model looks as follows:

image:images/screenshots/examples/person-model.png[]

Now that our source resource and model are defined, the flow can be created that handles the serivce logic.  The completed
flow looks as follows.

image:images/screenshots/examples/flow-get-all-persons.png[]

First, let's go through the non-service aspects of the flow.  The SQL Executor labeled "Setup Person Database" is used in the
sample flow to create the person table and populate it with data.  The Gate component labeled "Ensure DB Setup Completes"
is used to control the order of execution of the flow when an HTTP response triggers the flow to begin.  It simply ensures
SQL Executor "Setup Person Database" completes before reading the data from the database itself.  In most scenarios, these two steps
would not be necessary as the database where the data resides most likely already exists.  These two steps are included in the sample
simply to make it run stand-alone without the need for predefined database resources.

Next, let's look at the Http Request component and its associated properties.

image:images/screenshots/examples/http-request-properties.png[width=350]

Each property above is described in the <<http-request,Http Request>> component definition.  For this example, the "HTTP Method" of
type "GET" and the "Path" of "/person/getAll" tells the flow to listen for HTTP GET requests to a URL of http://myserver:port/metl/api/ws/person/getAll,
where "myserver" and "port" are the server and port on which Metl is running.  If Metl is running locally on its standard port of 42000,
HTTP GET requests to http://localhost:42000/metl/api/ws/person/getAll would be handled by the flow.

Once the request hits the flow, the "Setup Person" SQL Executor would run and setup the database.  The "Ensure DB Setup Completes" Gate would ensure
the SQL Executor is finished setting up the database before sending the HTTP request to the RDBMS Reader labeled "Read From Database".  The RDBMS Reader
component is set up as follows:

image:images/screenshots/examples/rdbms-reader-person-getall.png[width=350]

Note the output model of "Person" and the Sql of `select * from person`.  The RDBMS Reader will simply read all rows from the person table and return
the results based on the Person model.  Once the data has been read from the database, it will be forwarded to the <<serializer,Serializer>> component configured
as follows:

image:images/screenshots/examples/serializer-properties.png[width=350]

Now that the response has been formatted it send to the <<http-response,Http Response>> configured as follows:

image:images/screenshots/examples/http-response-properties.png[width=350]

Each property above is described in the <<http-response,Http Response>> component definition.

The following shows the results of a GET request to the service with an `Accept` header of `application/xml`.

With an `Accept` header of `application/json`, the result would look as follows:

If different XML or JSON formatting is desired, the flow can be configured to format entity based data into XML or JSON using
the XML Formatter or custom scripts and the resulting text messages from the formatters can be passed to the HTTP Response component.
The HTTP Response component will return the pre-formatted text (XML, JSON, or other) accordingly.

TIP: If the flow formats its own output, the HTTP Response component should be configured to communicate the content-type to the caller.
This can be done by setting the HTTP Response "Content Type" accordingly (i.e. application/xml, etc.)

In order for a REST service flow to be used (run), it must be deployed to an existing Metl agent.  See the <<Deploy,Deploy>> section
for details on creating agents and deploying flows.

To test the flow in the design mode, use the Run button and the flow will be auto-deployed to the design agent and a user interface will be presented for testing the web services.

image:images/screenshots/examples/test-webservices.png[]