Difference between revisions of "Datainterchange DSL"

Revision as of 15:03, 11 September 2017

A general introduction and some high-level concepts for OSBP DSLs can be found in OSBP DSL Documentation.

Please note that you can only make use of the Datainterchange DSL by downloading its packages (shown below on both Figure 1 and 2) from the download section of our website.

Purpose

The Data Interchange DSL (datainterchange for short) is made for defining data exchange models that can be used to import data from various formats (CSV, XML, EDI, etc.), map the data to entities, store them into database, or export them back into other formats.

You only need to define the relationship between the file and the bean, not the import / export process themselves. Once defined, these models can be used in e.g. action DSL to define actions which, when triggered, execute the actual import / export process, which are generated automatically by the OSBP based on the model.

Overview

The Datainterchange DSL is currently to be found under both packages org.eclipse.osbp.datainterchange.api and org.osbee.datainterchange. While all the basic interfaces are defined inside the API package, in the latter one are all commercial features and components implemented such as configuration files and artifacts based on the Smooks framework.

Figure 1: - Data Interchange API Project Structure.

Figure 2: - Data Interchange CORE Project Structure.

As shown in Figure 3, the [DSL Inferrer] will generate various views and In/Export component according to model described by datainterchange DSL (and action DSL, in the case of ActionButtons). The action buttons, when clicked, will trigger their corresponding In/Export processes by putting WorkerThread (Runnable) jobs into the executor job pool within the TriggerView (prefixed with datainterchang name), buttons (and toolbar / menus containing them) are further included in the perspective.

Figure 3: - Data Interchange Structure.

Data Interchange Grammar Definition

Figure 4: - Syntax Graph Data Interchange DSL

Data Interchange Model File

Datainterchange DSL model files end with the .data extension. Data Interchange models may be split into several .data files, as long as they have the same package declaration.

Reserved Keywords

In the following we’ll dive deeper into the description and the usage of Datainterchange related and reserved keywords.

import

In the import section are all entities to be found - as full qualified names – that are currently used in the DSL.

import ns net.osbee.sample.foodmart.entities.Mstore
import ns net.osbee.sample.foodmart.entities.Mwarehouse
import ns net.osbee.sample.foodmart.entities.Mregion

Note that ns is a mandatory keyword (stands for "namespace") that comes after import for distinguishing the OS.bee internal namespaces and Java library namespaces. Wildcards are not supported, all names should be imported separately. Note that the import section will be imported/added automatically if they are used in the package, so you don't have to manually manage this section.

package

Datainterchange DSL model files must start with a package declaration. Packages are the root element of the DSL and should be defined as <ApplicationName>.datainterchange, for example:

package net.osbee.sample.foodmart.datainterchanges { }

Data Interchange models may be split into several .data files, as long as they have the same package declaration, the interchanges will be available under this package name.

titel

With the keyword title you can give a name to the corresponding TriggerView dialog inside your application. For example, the definition of the same datainterchanges package from above with title "Data Interchange Example":

package net.osbee.sample.foodmart.datainterchanges title "Data Interchange Example" {}

This title will be translated based on locale.

You can get more details about the TriggerView in the section below.

interchange

The interchange keyword defines interchange units for entities in the package, it should be in the form of:

interchange <InterchangeUnitName> [describedBy <description>] <EntityManagerMode> file <FileType> [<FileDetails>] beans {
  ...
}

Where InterchangeUnitName being the name of the interchange unit; describedBy (optional) can be used to provide a short description string. EntityManagerMode being how the file should be handled (see section "persist, merge, remove" below), and after file keyword, you should specify the type of the source / target data file you would like to import from or export to, then the file path, and other details depends on the type.

The following example specifies an interchange that reads a CSV file under the specified path (note the forward slash as the path separator), delimited by semicolon, skip one line (the header), and treat the content as encoded in UTF-8:

interchange CurrencyNames persist file
CSV "C:/data/ISOCurrencyCode20170101.csv" delimiter ";" skipLines 1 encoding "UTF-8"
beans {
  ...
}

The beans keyword comes after all the file specification and starts a block of entity definitions, which will be covered in section "entity".

describedBy

With this keyword you can the optional description of an interchange unit as shown below.

interchange Currencies describedBy "european central bank currency exchange rates based on euro" persist … {

}

persist, merge, remove

These keywords define the purpose of the datainterchange unit and has a similar meaning as in the JPA's EntityManager class. Basically, persist will insert the data records into database, merge will update existing data record, or insert new one if necessary, remove will remove the record if it could be found in database.

Example persist:

interchange EDIOrders describedBy "Orders" persist file
XML "C:/data/orders.xml"
beans {
  ...
}

Example merge:

interchange Stores merge file
XML "C:/data/stores.xml"
beans {
  ...
}

Example remove:

interchange Storesremove remove file
XML "C:/data/stores_remove.xml"
beans {
  ...
}

file

With the keyword file you are able to set the file format of the files you intent to process with you interchange unit.

interchange <SampleInterchangeUnitName> <EntityManagerMode> file <FileNameFormat> {}

The current supported file formats are CSV, EDI and XML followed by the name of the file you want to process, given its full path location in the system.

interchange SampleInterchangeUnit1 merge file CSV "C:/temp/testFile.csv" {}

interchange SampleInterchangeUnit2 persist file XML "C:/temp/testFile.xml" {}

interchange SampleInterchangeUnit3 merge file EDI "C:/temp/testFile.edi" {}

After choosing the file format you can either give the file name as a String value in a double quote "..." as shown here above, or press Ctrl+Space to get via the content assist the option of opening a File Chooser/Picker to specify the file you want to work with.

Figure 5: File Chooser

Please note that you can also change the path(?) of the file to process on runtime by selecting a new file.

mapByAttribute

The mapByAttribute keyword is an XML-specific keyword that turns on the automatic attribute mapping. When enabled, datainterchange will detect if the value being mapped is from an attribute. For example, the 'USD' value in the following XML file comes from the attribute 'currency' of the element 'Cube':

<Cube currency='USD' rate='1.3759'/>

while in the following XML file, the same value is encapsulated in the element 'currency':

<Cube>
 <currency>USD</currency>
 <rate>1.3759</rate>
</Cube>

When mapByAttribute is present, datainterchange will automatically decide that a query in the form like '/Cube/currency' will also catch the value from attribute. Without it, the query will have to put an '@' symbol in front of the attribute name, i.e. '/Cube/@currency'.

elementSize

With the keyword elementSize followed by an integer, the user can set the estimated average size of the elements in bytes. Since the underlying API can not know the size of an element before it is processed, this value can be supplied as a guide value to be used for estimating the import / output progress based on how much bytes have been processed.

Example:

interchange EDIOrders describedBy "edi orders" persist elementSize 50 file EDI "C:/data/orders.edi" beans {
  ...
}

Defines an average estimated element size of 50 bytes.

delimiter

The delimiter is a CSV format-specific keyword, which defines the character to use in the CSV file to set the delimiter/separation character between different values. Default value is "," (comma).

Example:

interchange CurrencyNames persist file
CSV "C:/data/ISOCurrencyCodes081507.csv" delimiter ";" ... {
  ...
}

will set the delimiter of the CSV file to ";".

skipLines

This is a CSV format-specific keyword. Using skipLines followed by an integer, the user can specify the number of lines to be skipped from the beginning in the processing of the selected file, this can be used to skip headers.

Example:

interchange CurrencyNames persist file CSV "C:/data/ISOCurrencyCodes081507.csv" ... skipLines 1 encoding "UTF-8" {
   ...
}

will skip the first line of the CSV file.

report

If a datainterchange unit is defined with the report keyword, a report will be generated for data conversions. The report file is generated by Smooks, lies under /smooks output directory of the datainterchange bundle, and has a name in the form like <DataInterChangeName>-input.xml for input processes, and <DataInterChangeName>-output.xml for output processes. Note: turn on report will have an impact performance.

indent

The indent is a CSV-format specified keyword which adds indentation character data to the generated event stream. This simply makes the generated event stream easier to read in its serialized form and generally should only be used in testing.

quoteCharacter

The quoteCharacter is a CSV format-specific keyword, which defines the character to use in the CSV file to identify values.

encoding

With the keyword enconding followed by the encoding name as a string value you are able to specify the valid encoding of the file content.

interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "UTF-8"{}

interchange SampleInterchangeUnitName remove elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "GB18030"{}

interchange SampleInterchangeUnitName persistelementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "ISO-2022-JP"{}

beans

The keyword beans starts the series of entities, see example below. The name "bean" comes from the internal entities called JavaBean, which act as data containers.

entity

With the keyword entity the user can specify the entity on which the interchange unit is based on.

Example:

interchange Stores merge elementSize 83 file
XML "C:/data/stores.xml" beans {
	entity Mregion
	entity Mstore
	entity Mwarehouse
}

For more information about the entities, review the corresponding Entity DSL documentation.

Entity

With the entity keyword followed by a fully qualified name, the user can specify the projection of a source data file to a data-containing entity. The behavior of the projection can be further fine-tuned with the keywords discussed below. These keywords, while can theoretically be combined all together and create very complicated behavior, are normally used in a simple and straight forward way.

The general form of entity keywords is:

entity <ID>
    [nodeName <node-name>]
    [createOn <element-map>]
    [marker <property-name>]
    [expression '{' <expressions> '}'] 
    [lookup '{' <lookup-rules> '}']
    [format '{' <formats> '}']
    [mapping '{' <mappings> '}']
    [keys '{' <lookup-keys> '}']
;

The order of the optional keywords are irrelevant, they can appear in any order.

nodeName

The nodeName keyword followed by a string specifies the name (alias) of its corresponding element inside an (XML) input/output configuration file. This name is used to identify entities within an XML file using the NodeModel of Freemarker instead of using the standard Java Object Model name.

For example:

interchange Stores remove vectorName "stores" elementSize 83 file
	XML "C:/.../net.osbee.sample.foodmart/net.osbee.sample.foodmart.datainterchange/smooks-resources/stores.xml" beans {
		entity Mregion nodeName    "region"
		entity Mstore nodeName     "store"
		entity Mwarehouse nodeName "warehouse" 
	}

will create a Smooks configuration file in which the structure of the output data file (XML) to be generated will be the same as declared inside the beans expression, but also will also contain the string sprecified after the keywords.

Figure 6: Store-import.xml with NodeModel

Figure 7: Store-export.xml with NodeModel

The use of this keyword gives you the flexibility of naming entities using alias in order to match each third part system descriptions. The main difference here with the latter is that the names of the entities are used inside both configuration files.

Figure 8: Store-import.xml (normal) with Java Object Model

Figure 9: Store-export.xml with Java Object Model

Please note that the order of declaration of each node names matters and is compliant with the designated entities structures.

createOn

The createOn keyword followed by a string specifies on which input element should an entity to be created. For example:

entity Mcurrency createOn "/Envelope/Cube/Cube/Cube" ... {
    ...
}

will create an Mcurrency entity when encountering /Envelope/Cube/Cube in the source data file.

marker

With marker keyword optionally followed by a property-name, the user can set the data to be imported to have an additional property under the given name whose value set to 1, while updating the existing data to have this property having value 0. This could be useful to identify the most recent import of some periodically updated data. For example:

entity McurrencyStream createOn "/Envelope/Cube" marker latest expression {
	...
}

will make the entity McurrencyStream to have a property "latest", which is to be set to 1 for the most recently imported data.

expression

With a expression { ... } block, the user can define expressions that assign certain value to an entity property. This could have two forms:

  assign <id> with <value> as <type>

this will assign the property id with the value of value, in type of type. The value may be one of the following:

• NowDate: the date of now (the time point of action)
• StartDate: the date when the process started
• UniversallyUniqueIdentifier: an UUID

and type may be one of the following:

• Date: the value of NowDate or StartDate will be in date format
• Milliseconds: the value of NowDate or StartDate will be in millisecond
• Nanoseconds: the value of NowDate or StartDate will be in nanosconds
• Random: the value of UUID will be random
• ExecuteContext: the value of UUID will be unique for the execute context

Note that the Random and ExecuteContext should be only combined with UniversallyUniqueIdentifier, while the other types should be only combined with NowDate or StartDate.

Example:

  entity McurrencyStream createOn "/Envelope/Cube" marker latest expression {
      assign importDate with NowDate as Date
  }

will assign the importDate as the current date in Date format.

The other type of expression has the form:

  copy <target-property> from <entity-name> property <from-property>

which will copy the value of from-property from the entity-name to target-property.

Example:

  entity Mcurrency createOn "/Envelope/Cube/Cube/Cube" expression {
      copy currencyDate from McurrencyDay property ratingDate
  }

will copy the value of ratingDate from McurrencyDay entity to currencyDate.

format -> for...coding...locale

keys -> key

mapping -> map...to

lookup -> for...on...createOn...with...cacheSize...mapTo

marker -> makerPath -> markerEntity | markedBy

Enums

Enums are an abstraction of the Java enum. They compile to enum classes and can be used as properties in entities and beans.

Figure 10: Defining enums allows using them as variables in entities and beans.

The Datainterchange DSL provides 3 Enum types that can be used: PredefinedBeanEnum, PredefinedBeanTypeEnum, ProgessBarStyleEnum and the EntityManagerMode.

PredefinedBeanEnum

Contains the definition of the values now(NowDate) | start(StartDate) | UUID(UniversallyUniqueIdentifier) used for ...

PredefinedBeanTypeEnum

Contains the definition of the values Date(Date) | Millis(Milliseconds) | Nanos(Nanoseconds) | random(Random) | execContext(ExecuteContext) used for ...

ProgessBarStyleEnum

Contains the definition of the values none(none) | normal(normal) | important(important) used for setting the style of a progressbar.

EntityManagerMode

Contains the definition of the values persist(persist) | merge(merge) | remove(remove) used for specifying the Entity Manager Mode in order to decide how to manage data inside a database before committing any changes.

Configuration Settings / Smooks Framework (A small introduction)

The OS.bee implementation of Data Interchange interface is based on Smooks. Smooks is a Java framework for processing XML and non XML data (CSV, EDI, Java etc) by mapping data to JavaBeans, which can later be persisted, enriched (merge with existing data from other source), or converted and exported into other formats.

Smooks relies on a proper configuration file for the import / export processes. These configuration files are generated by the Datainterchange DSL automatically. Here is a brief introduction of how the generated Smooks configuration files work.

When a data import / export Smooks instance is initiated, it will be supplied with the generated config file. This file defines the actions to be performed upon certain events during the SAX parsing process. Here is an example:

 1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
 2 <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd" xmlns:dao="http://www.milyn.org/xsd/smooks/persistence-1.2.xsd" xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd">
 3     <jb:bean beanId="McurrencyNameStream" class="net.osbee.sample.foodmart.entities.McurrencyNameStream" createOnElement="/csv-set">
 4         <jb:expression property="importDate">PTIME.nowDate</jb:expression>
 5         <jb:wiring beanIdRef="McurrencyName" setterMethod="addToCurrencyNames"/>
 6     </jb:bean>
 7     <jb:bean beanId="McurrencyName" class="net.osbee.sample.foodmart.entities.McurrencyName" createOnElement="/csv-set/csv-record">
 8         <jb:value data="/csv-set/csv-record/isoCode" decoder="String" property="isoCode"/>
 9         <jb:value data="/csv-set/csv-record/name" decoder="String" property="name"/>
10         <jb:value data="/csv-set/csv-record/countries" decoder="String" property="countries"/>
11     </jb:bean>
12     <csv:reader fields="isoCode,name,countries" indent="false" separator=";" skipLines="1"/>
13     <params>
14         <param name="stream.filter.type">SAX</param>
15         <param name="inputType">input.csv</param>
16         <param name="smooks.visitors.sort">false</param>
17         <param name="input.csv" type="input.type.actived">C:/git/net.osbee.sample.foodmart/net.osbee.sample.foodmart.datainterchange/smooks-resources/ISOCurrencyCodes081507.csv</param>
18     </params>
19 </smooks-resource-list>

Further informations on the Smoooks framework are available and can be reviewed in its documentation you will find on the official website.

TriggerView & Executorservice

Annotations

Comments

Copyright Notice

All rights are reserved by Compex Systemhaus GmbH. In particular, duplications, translations, microfilming, saving and processing in electronic systems are protected by copyright. Use of this manual is only authorized with the permission of Compex Systemhaus GmbH. Infringements of the law shall be punished in accordance with civil and penal laws. We have taken utmost care in putting together texts and images. Nevertheless, the possibility of errors cannot be completely ruled out. The Figures and information in this manual are only given as approximations unless expressly indicated as binding. Amendments to the manual due to amendments to the standard software remain reserved. Please note that the latest amendments to the manual can be accessed through our helpdesk at any time. The contractually agreed regulations of the licensing and maintenance of the standard software shall apply with regard to liability for any errors in the documentation. Guarantees, particularly guarantees of quality or durability can only be assumed for the manual insofar as its quality or durability are expressly stipulated as guaranteed. If you would like to make a suggestion, the Compex Team would be very pleased to hear from you.

(c) 2016-2024 Compex Systemhaus GmbH