Difference between revisions of "Datainterchange DSL"

From OS.bee documentation
Jump to: navigation, search
(→‎Overview)
(→‎Data Interchange Model File)
Line 44: Line 44:
 
Datainterchange DSL model files must start with a package declaration. Packages are the root element of the DSL grammar and must be defined as follow package <PackageName> for example
 
Datainterchange DSL model files must start with a package declaration. Packages are the root element of the DSL grammar and must be defined as follow package <PackageName> for example
  
−
<syntaxhighlight lang="java">
+
<syntaxhighlight lang="java">package net.osbee.sample.ApplicationName.datainterchanges { }</syntaxhighlight>
−
package net.osbee.sample.ApplicationName.datainterchanges { }
+
−
</syntaxhighlight>
+
  
 
Data Interchange models may be split into several .data files, as long as they have the same package declaration.  
 
Data Interchange models may be split into several .data files, as long as they have the same package declaration.  
Line 65: Line 63:
 
For example the definition of the same datainterchanges package from above with the title DataInterchange
 
For example the definition of the same datainterchanges package from above with the title DataInterchange
  
−
<syntaxhighlight lang="java">
+
<syntaxhighlight lang="java">package net.osbee.sample.foodmart.datainterchanges title "DataInterchange" {}</syntaxhighlight>
−
package net.osbee.sample.foodmart.datainterchanges title "DataInterchange" {}
+
−
</syntaxhighlight>
+
  
 
You can get more details about the TriggerView in the section below.
 
You can get more details about the TriggerView in the section below.
Line 74: Line 70:
 
All interchange units have to be defined in the package followed by the Entity they are referencing/applied to.
 
All interchange units have to be defined in the package followed by the Entity they are referencing/applied to.
  
−
<syntaxhighlight lang="java">
+
<syntaxhighlight lang="java">interchange <SampleInterchangeUnitName> <EntityManagerMode> <FileFormat> {}</syntaxhighlight>
−
interchange <SampleInterchangeUnitName> <EntityManagerMode> <FileFormat> {}
+
−
</syntaxhighlight>
+
  
 
====describedBy====
 
====describedBy====
 
With this keyword you get to set the optional description of an interchange unit as shown below.  
 
With this keyword you get to set the optional description of an interchange unit as shown below.  
  
−
<syntaxhighlight lang="java">
+
<syntaxhighlight lang="java">interchange <SampleInterchangeUnitName> describedBy "This is a description of the Datainterchange DSL Unit EntityName" … {}</syntaxhighlight>
−
interchange <SampleInterchangeUnitName> describedBy "This is a description of the Datainterchange DSL Unit EntityName" … {}
+
−
</syntaxhighlight>
+
  
 
====persist, merge, remove====
 
====persist, merge, remove====
 
With those keywords defined inside the Enum Entity Manager Mode, you set how the data have to be process by inserting then into a database.
 
With those keywords defined inside the Enum Entity Manager Mode, you set how the data have to be process by inserting then into a database.
−
<syntaxhighlight lang="java">
+
<syntaxhighlight lang="java">interchange <SampleInterchangeUnitName> <EntityManagerMode> … {}</syntaxhighlight>
−
interchange <SampleInterchangeUnitName> <EntityManagerMode> … {}
+
 
−
</syntaxhighlight>
+
 
Example 1:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge … {}</syntaxhighlight>
 
Example 1:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge … {}</syntaxhighlight>
 
Example 2:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit2 persist … {}</syntaxhighlight>
 
Example 2:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit2 persist … {}</syntaxhighlight>
Line 101: Line 92:
  
 
The current supported file formats are CSV, EDI and XML followed by the name of the file you want to process, given its full path location in the system.
 
The current supported file formats are CSV, EDI and XML followed by the name of the file you want to process, given its full path location in the system.
−
Example 1:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge file CSV "C:/temp/testFile.csv” {}</syntaxhighlight>
+
Example 1:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge file CSV "C:/temp/testFile.csv" {}</syntaxhighlight>
−
Example 2:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit2 persist file XML "C:/temp/testFile.xml” {}</syntaxhighlight>
+
Example 2:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit2 persist file XML "C:/temp/testFile.xml" {}</syntaxhighlight>
−
Example 3:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit3 merge file EDI "C:/temp/testFile.edi” {}</syntaxhighlight>
+
Example 3:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit3 merge file EDI "C:/temp/testFile.edi" {}</syntaxhighlight>
  
 
After choosing the file format you can either give the file name as a String value in a double quote "..." as shown here above, or press Ctrl+Space to get via the content assist the option of opening a File Chooser/Picker to specify the file you want to work with.
 
After choosing the file format you can either give the file name as a String value in a double quote "..." as shown here above, or press Ctrl+Space to get via the content assist the option of opening a File Chooser/Picker to specify the file you want to work with.
Line 114: Line 105:
 
====elementSize====
 
====elementSize====
 
With the keyword elementSize followed by a number between 0 and 100 ?  you set the average size of the file (or set of data).
 
With the keyword elementSize followed by a number between 0 and 100 ?  you set the average size of the file (or set of data).
−
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv” {}</syntaxhighlight>
+
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" {}</syntaxhighlight>
  
 
====delimiter====
 
====delimiter====
 
With the keyword delimiter you set the delimiter/separation character of the file.
 
With the keyword delimiter you set the delimiter/separation character of the file.
−
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv” delimiter ";" {}</syntaxhighlight>
+
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" {}</syntaxhighlight>
  
  
 
====skipLines====
 
====skipLines====
−
With the keyword skipLines followed by a number you specify the number of line to be skipped or not to be considered in the processing of the selected file.
+
With the keyword '''skipLines''' followed by a number you specify the number of line to be skipped or not to be considered in the processing of the selected file.
−
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv” delimiter ";" skipLines 1 {}</syntaxhighlight>
+
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 {}</syntaxhighlight>
  
 
====report====
 
====report====
−
 
−
 
  
 
====indent====
 
====indent====
  
 
====quoteCharacter====
 
====quoteCharacter====
−
 
  
 
====strict====
 
====strict====
 +
 +
====validateHeader====
 +
 +
====encoding====
 +
With the keyword '''enconding''' followed by the encoding name as a string value you are able to specify the valid encoding of the file content.
 +
 +
Example 1:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "UTF-8"{}</syntaxhighlight>
 +
Example 2:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName remove elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "GB18030"{}</syntaxhighlight>
 +
Example 3:  <syntaxhighlight lang="java">interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "ISO-2022-JP"{}</syntaxhighlight>
  
 
====beans====
 
====beans====
Line 141: Line 138:
 
With the keyword '''entity''' you specify the Entity on what the interchange unit is based on.
 
With the keyword '''entity''' you specify the Entity on what the interchange unit is based on.
  
−
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge elementSize 50 file CSV "C:/temp/testFile.csv” delimiter ";" skipLines 1 {
+
Example:  <syntaxhighlight lang="java">interchange SampleInterchangeUnit1 merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 {
 
entity SampleEntity1 … {}  
 
entity SampleEntity1 … {}  
 
}</syntaxhighlight>
 
}</syntaxhighlight>

Revision as of 09:51, 10 August 2017

A general introduction and some high-level concepts for OSBP DSLs can be found in OSBP DSL Documentation.

Please note that you can only make use of the Datainterchange DSL by downloading its packages (shown below on both Figure 1 and 2) from the download section of our website.

Purpose

The Data Interchange DSL (datainterchange for short) is made for defining data exchange models that can be used to import data from various formats (CSV, XML, EDI, etc.), map the data to entities, store them into database, or export them back into other formats.

You only need to define the relationship between the file and the bean, not the import / export process themselves. Once defined, these models can be used in e.g. action DSL to define actions which, when triggered, execute the actual import / export process, which are generated automatically by the OSBP based on the model.

Overview

The Datainterchange DSL is currently to be found under both packages org.eclipse.osbp.datainterchange.api and org.osbee.datainterchange. While all the basic interfaces are defined inside the API package, in the latter one are all commercial features and components implemented such as configuration files and artifacts based on the Smooks framework.

Figure 1: - Data Interchange API Project Structure.
Figure 2: - Data Interchange CORE Project Structure.


As shown in Figure 3, the DSL inferrer will generate various views and In/Export component according to model described by datainterchange DSL (and action DSL, in the case of ActionButtons). The action buttons, when clicked, will trigger their corresponding In/Export processes by putting WorkerThreadRunnable jobs into the executor job pool within the TriggerView (prefixed with datainterchang name), buttons (and toolbar / menus containing them) are further included in the perspective.


Figure 3: - Data Interchange Structure.

Data Interchange Model File

Datainterchange DSL model files end with the .data extension. Data Interchange models may be split into several .data files, as long as they have the same package declaration.


Reserved Keywords

In the following we’ll dive deeper into the description and the usage of Datainterchange related and reserved keywords.

import

In the import section are all entities to be found - as full qualified names – that are currently used in the DSL and automatically imported/added (OXImportDeclaration) by editing it. 

import ns net.osbee.sample.<ApplicationName>.entities.SampleEntity1
import ns net.osbee.sample.<ApplicationName>.entities.SampleEntity2
import ns net.osbee.sample.<ApplicationName>.entities.SampleEntity3

package

Datainterchange DSL model files must start with a package declaration. Packages are the root element of the DSL grammar and must be defined as follow package <PackageName> for example 

package net.osbee.sample.ApplicationName.datainterchanges { }

Data Interchange models may be split into several .data files, as long as they have the same package declaration. 


package net.osbee.sample.<applicationname>.datainterchanges title "<titletext>" {
    interchange <interchangename> persist file
    <fileformat> "<filepath>" [<further specifications>]
    beans {
        <entity relationships>
    }
}

titel

With the keyword title you can give a name to the corresponding TriggerView dialog inside your application. For example the definition of the same datainterchanges package from above with the title DataInterchange 

package net.osbee.sample.foodmart.datainterchanges title "DataInterchange" {}

You can get more details about the TriggerView in the section below.

interchange

All interchange units have to be defined in the package followed by the Entity they are referencing/applied to. 

interchange <SampleInterchangeUnitName> <EntityManagerMode> <FileFormat> {}

describedBy

With this keyword you get to set the optional description of an interchange unit as shown below. 

interchange <SampleInterchangeUnitName> describedBy "This is a description of the Datainterchange DSL Unit EntityName" … {}

persist, merge, remove

With those keywords defined inside the Enum Entity Manager Mode, you set how the data have to be process by inserting then into a database. 

interchange <SampleInterchangeUnitName> <EntityManagerMode> … {}
Example 1: 
interchange SampleInterchangeUnit1 merge … {}
Example 2: 
interchange SampleInterchangeUnit2 persist … {}
Example 3: 
interchange SampleInterchangeUnit3 remove … {}

file

With the keyword file you are able to set the file format of the files you intent to process with you interchange unit. 

interchange <SampleInterchangeUnitName> <EntityManagerMode> file <FileNameFormat> {}


The current supported file formats are CSV, EDI and XML followed by the name of the file you want to process, given its full path location in the system.

Example 1: 
interchange SampleInterchangeUnit1 merge file CSV "C:/temp/testFile.csv" {}
Example 2: 
interchange SampleInterchangeUnit2 persist file XML "C:/temp/testFile.xml" {}
Example 3: 
interchange SampleInterchangeUnit3 merge file EDI "C:/temp/testFile.edi" {}

After choosing the file format you can either give the file name as a String value in a double quote "..." as shown here above, or press Ctrl+Space to get via the content assist the option of opening a File Chooser/Picker to specify the file you want to work with.

Figure 4: File Chooser


Please note that you can also change the path(?) of the file to process on runtime by selecting a new file.

elementSize

With the keyword elementSize followed by a number between 0 and 100 ? you set the average size of the file (or set of data).

Example: 
interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" {}

delimiter

With the keyword delimiter you set the delimiter/separation character of the file.

Example: 
interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" {}


skipLines

With the keyword skipLines followed by a number you specify the number of line to be skipped or not to be considered in the processing of the selected file.

Example: 
interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 {}

report

indent

quoteCharacter

strict

validateHeader

encoding

With the keyword enconding followed by the encoding name as a string value you are able to specify the valid encoding of the file content.

Example 1: 
interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "UTF-8"{}
Example 2: 
interchange SampleInterchangeUnitName remove elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "GB18030"{}
Example 3: 
interchange SampleInterchangeUnitName merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 encoding "ISO-2022-JP"{}

beans

entity

With the keyword entity you specify the Entity on what the interchange unit is based on.

Example: 
interchange SampleInterchangeUnit1 merge elementSize 50 file CSV "C:/temp/testFile.csv" delimiter ";" skipLines 1 {
entity SampleEntity1 … {} 
}

For more information about Entities you can review the corresponding Entity DSL documentation.

createOn

element

expression -> assign | copy | with...as

format

keys -> key

list

mapping -> map...to

lookup

marker

Enums

Enums are an abstraction of the Java enum. They compile to enum classes and can be used as properties in entities and beans.

Figure 5: Defining enums allows using them as variables in entities and beans.

The Datainterchange DSL provides 3 Enum types that can be used: PredefinedBeanEnum, PredefinedBeanTypeEnum, ProgessBarStyleEnum and the EntityManagerMode.


PredefinedBeanEnum

Contains the definition of the values now(NowDate) | start(StartDate) | UUID(UniversallyUniqueIdentifier) used for ...

PredefinedBeanTypeEnum

Contains the definition of the values Date(Date) | Millis(Milliseconds) | Nanos(Nanoseconds) | random(Random) | execContext(ExecuteContext) used for ...

ProgessBarStyleEnum

Contains the definition of the values none(none) | normal(normal) | important(important) used for setting the style of a progressbar.

EntityManagerMode

Contains the definition of the values persist(persist) | merge(merge) | remove(remove) used for specifying the Entity Manager Mode in order to decide how to manage data inside a database before committing any changes.

Configuration Settings / Smooks Framework (A small introduction)

The OS.bee implementation of Data Interchange interface is based on Smooks. Smooks is a Java framework for processing XML and non XML data (CSV, EDI, Java etc) by mapping data to JavaBeans, which can later be persisted, enriched (merge with existing data from other source), or converted and exported into other formats.

Smooks relies on a proper configuration file for the import / export processes. These configuration files are generated by the Datainterchange DSL automatically. Here is a brief introduction of how the generated Smooks configuration files work.

When a data import / export Smooks instance is initiated, it will be supplied with the generated config file. This file defines the actions to be performed upon certain events during the SAX parsing process. Here is an example: 

 1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
 2 <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd" xmlns:dao="http://www.milyn.org/xsd/smooks/persistence-1.2.xsd" xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd">
 3     <jb:bean beanId="McurrencyNameStream" class="net.osbee.sample.foodmart.entities.McurrencyNameStream" createOnElement="/csv-set">
 4         <jb:expression property="importDate">PTIME.nowDate</jb:expression>
 5         <jb:wiring beanIdRef="McurrencyName" setterMethod="addToCurrencyNames"/>
 6     </jb:bean>
 7     <jb:bean beanId="McurrencyName" class="net.osbee.sample.foodmart.entities.McurrencyName" createOnElement="/csv-set/csv-record">
 8         <jb:value data="/csv-set/csv-record/isoCode" decoder="String" property="isoCode"/>
 9         <jb:value data="/csv-set/csv-record/name" decoder="String" property="name"/>
10         <jb:value data="/csv-set/csv-record/countries" decoder="String" property="countries"/>
11     </jb:bean>
12     <csv:reader fields="isoCode,name,countries" indent="false" separator=";" skipLines="1"/>
13     <params>
14         <param name="stream.filter.type">SAX</param>
15         <param name="inputType">input.csv</param>
16         <param name="smooks.visitors.sort">false</param>
17         <param name="input.csv" type="input.type.actived">C:/git/net.osbee.sample.foodmart/net.osbee.sample.foodmart.datainterchange/smooks-resources/ISOCurrencyCodes081507.csv</param>
18     </params>
19 </smooks-resource-list>

Annotations

Comments

Smooks (Extended)

just links???

WorkerThread (Runnable)

Javadoc

TriggerView & Executorservice

DSL Inferrer

DSL Scope Provider

DSL Proposal Provider

DSL Validator

Copyright Notice

All rights are reserved by Compex Systemhaus GmbH. In particular, duplications, translations, microfilming, saving and processing in electronic systems are protected by copyright. Use of this manual is only authorized with the permission of Compex Systemhaus GmbH. Infringements of the law shall be punished in accordance with civil and penal laws. We have taken utmost care in putting together texts and images. Nevertheless, the possibility of errors cannot be completely ruled out. The Figures and information in this manual are only given as approximations unless expressly indicated as binding. Amendments to the manual due to amendments to the standard software remain reserved. Please note that the latest amendments to the manual can be accessed through our helpdesk at any time. The contractually agreed regulations of the licensing and maintenance of the standard software shall apply with regard to liability for any errors in the documentation. Guarantees, particularly guarantees of quality or durability can only be assumed for the manual insofar as its quality or durability are expressly stipulated as guaranteed. If you would like to make a suggestion, the Compex Team would be very pleased to hear from you.

(c) 2016-2024 Compex Systemhaus GmbH

Retrieved from "https://download.osbee.org/documentation/index.php?title=Datainterchange_DSL&oldid=2099"