Difference between revisions of "Signal DSL"

Latest revision as of 12:34, 12 November 2018

Purpose

POS master data updates over signal watchers, to execute a list of tasks (for now only import and/or export via datainterchange) depending on newly created files. This so called trigger-file will be deleted after the execution of all the tasks (or actions) weither they are be executed sequentially or parallel.

A client wishes to export a certain amount of data (sales, reports, master data...) on a specific working day (or not) at a certain time, either once or several time on this date. Therefore, it should now be possible to define such characteristic inside a model.

SignalDSL

The SingalDSL defines the java functions for Os.bee and the functions will be used in other Os.bee models. The main semantic elements of the SingalDSL are:

• package - The root element that contains all the other elements. (now in one file only one package could be defined, the grammar could be altered to support several packages in one file, if needed)
• import declarations - Used to import external models or other Java classes.
• watcher - define a watcher which monitors a directory.
• filemask - define a group of target files to watch and all detail about the watcher.
• filename - define a target file to watch and detail about the watcher.
• scheduler - define the simple or complex schedulers for executing tens, hundreds, or even tens-of-thousands of jobs.

package defintion

With the keyword package followed by a qualified name, you define the root element that contains all the other elements. A model can contain multiple packages.

► Syntax:

package <package name>[{ 
	(
	  watcher <watcher name>
		  filemask <file mask> . . .
		| filename <file name> . . .

	| scheduler <scheduler name> . . .
	)* 
}]

• one or more watcher/scheduler can be defined in the same package.
• 2 types of watcher can be defined, they are file mask watcher and file name watcher.
• The trigger policy defines how the execution of its list of tasks be triggered: either file based (the creation of a file) or scheduled plan based.
  • Currently we only support the export and import functions of existing Datainterchange Units from the DatainterchangeDSL as the possible watcher’s/scheduler’s tasks to be executed.
  • The properties of DatainterchangeDSL are saved in the Datainterchange Configuration Files which are generated with the DatainterchangeDSL.
  • The Datainterchange Configuration Files are default saved in C:\Users\<user name>\.osbee. Per Datainterchange Group, a <Datainterchange Group name>Config.xml file will be generated automatically. This file path can be changed in Eclipse--> Preferences--> OSBP Application Configuration -->External Data Source --> Datainterchange Setting.
  • The corresponding watcher's and scheduler's properties defined in SingalDSL will be added as new entries into the corresponding Datainterchange Configuration File (<Datainterchange Group name>Config.xml) automatically after saving the SignalDSL file, like <entry key="<package name>.WithMaskWatcher.filemask"><file mask></entry> , or <entry key="<package name>.WithMaskWatcher.filename"><file name></entry> , or <entry key="<package name>.<scheduler name>Scheduler.<scheduler type>.<scheduler property>"><property value></entry> . The <scheduler type> can be hourlyscheduler,dailyscheduler,monthlyscheduler,weeklyscheduler and cronscheduler; the <scheduler property> can be minute, hour, dayofweek,dayofmonth, and expression.
  • In cases the needed datainterchange properties are not available in the Datainterchange Configuration File, or the Datainterchange Configuration File doesn't exist, the default definition of the properties from the DatainterchangeDSL file will be used.
  • Currently it is not possible to monitor changes on a (web)url-based directory (destination). You may however define a watcher with a url-based directory and set schedulers to import data from it.
  • Watcher/scheduler generates watcher/scheduler jobs, they will be stored into a queue and processed by the watcher/scheduler job handler. Keyword sequential and parallel determinate how the jobs will be executed. As same as the literally meaning, Keyword sequential means all tasks/jobs will be sequentially executed; Keyword parallel means all tasks/jobs will be parallel executed. If it is not defined, sequential will be take as the default value.

► Following are the example predefined datainterchange definition and the corresponding configuration files which will be used in the SingalDSL definition example in this documentation.

Datainterchange Group in DatainterchangeDSL

group Brands {
	interchange Brandowner merge deleteFileAfterImport file CSV "C:/datatransfers/BrandOwner.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brandowner
	}

	interchange Brandtype merge deleteFileAfterImport file CSV "C:/datatransfers/BrandType.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brandtype
	}

	interchange Brand merge deleteFileAfterImport file CSV "C:/datatransfers/Brand.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brand lookup {
			for brandtypeid in Brandtype createOn "/csv-set/csv-record" mapFrom "id" mapTo id for brandownerid in Brandowner createOn "/csv-set/csv-record" mapFrom "id" mapTo id
			}
		mapping {
			map brandtypecd to "brandtypecd" map bsin to "bsin" mapBlob bsinmedia to "bsin" extension "jpg" path "C:/POD/brands/brand" mimeType jpg map brandnm to "brandnm" map brandlink to "brandlink" map id to "id"
		} 
	}

	interchange Gtin merge deleteFileAfterImport file CSV "C:/datatransfers/Gtin.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Gtin lookup {
			for brandid in Brand createOn "/csv-set/csv-record" mapFrom "id" allowNoResult mapTo id
		}
	}

	interchange Pkgtype merge deleteFileAfterImport file CSV "C:/datatransfers/PkgType.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Pkgtype
	}
}

the Datainterchange Group name is Brands, and the Datainterchange Units are Brandowner,Brandtype,Brand,Gtin and Pkgtype.

Datainterchange configuration file BrandsConfig.xml for this group 

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>dataInterchange file URLs</comment>
<entry key="Pkgtype-export">C:/datatransfers/PkgType.csv</entry>
<entry key="Max-Parallel-Threads-Count">3</entry>
<entry key="Brand-export">C:/datatransfers/Brand.csv</entry>
<entry key="Pkgtype-import">C:/datatransfers/PkgType.csv</entry>
<entry key="Brandtype-export">C:/datatransfers/BrandType.csv</entry>
<entry key="Brand-import">C:/datatransfers/Brand.csv</entry>
<entry key="Brandtype-import">C:/datatransfers/BrandType.csv</entry>
<entry key="Brandowner-export">C:/datatransfers/BrandOwner.csv</entry>
<entry key="Gtin-export">C:/datatransfers/Gtin.csv</entry>
<entry key="Brandowner-import">C:/datatransfers/BrandOwner.csv</entry>
<entry key="Gtin-import">C:/datatransfers/Gtin.csv</entry>
</properties>

2 entries of import/export directories are generated for each Datainterchange Unit and an entry of Max-Parallel-Threads-Count value is generated for this Datainterchange Group. All values in this file can be modified manual.

Watcher

Watcher jobs are generated by a watcher, they will be stored into a queue and processed by the watcher job handler.

► Syntax:

watcher <watcher name>
	filemask <file mask>
		sequential | parallel
		from <DataInterchange Group>
	{
		import|export <DataInterchange Unit> [applyon]
	}

	| 
	filename <file name>
		sequential | parallel
		from <DataInterchangeGroup>
	{
		(import|export <DataInterchange Unit> [applyon])*
	}

• 2 types of watcher can be defined, they are:
  • file mask watcher

    ► Example:

    watcher WithMask filemask "Gtin*.csv" parallel from Brands {
    	import Gtin
    }
    

  • file name watcher

    ► Example:

    watcher WithName filename "brands.csv" parallel from Brands {
    	import Brandowner
    	import Brandtype
    	import Brand
    	import Pkgtype
    	import Gtin applyon
    	export Brand
    }
    

Target File

A target file is the file which is generated inside the monitored directory, whose filename (+extension) matches the file mask or the file name.

The datainterchange file used for import processes will be deleted at the end of a successful import, if it is explicitly set in its datainterchange definition. See Datainterchange DSL for more information.

In case the target file is at the same time used in datainterchange as the import file, it will be deleted at the end of the execution of the watcher job (parallel or sequential), even if it’s not set in the datainterchange definition.

If the import process is failed, at the end of the import process, the corresponding import file will be renamed with form FAILEDIMPORT+< actual date formated >+Original file name.LOCKED (as the definition in the following blog post).

Keyword applyon

The keyword applyon has been introduced in order to define per watcher, which directory has to be monitored for the triggering events: creations of target files in this directory, which match the given file mask or file name.

The watcher-monitored directory corresponds to the import-path ( in configuration file after keyword entry key ="<datainterchange uinit >-import" ) of the Datainterchange Unit marked with the keyword applyon as the watcher tasks in the SignalDSL file.

If the DatainterchangeDSL Configuration File (per Datainterchange group a file, which contains also the SignalDSL properties) exists, the needed import-path property of the Datainterchange Unit exists and its value is valid, then this file path will be used to monitor. Otherwise, the default path of the Datainterchange Unit defined in the corresponding DatainterchangeDSL definition will be used as the monitored directory.

Watcher Job

The watcher job queue is used to secure the sequential execution of each watcher jobs (task/list of tasks).

The jobs generated at the same time from the same file mask watcher can be called as a job group. A grouping of jobs from the same watcher is actually taking place in order to better make use of multi-threading functionalities for the watchers which are allowed a parallel execution of their jobs (only one for a file mask watcher).

Grouping the execution of jobs from the same watcher is mostly useful in the case of parallel execution of the watcher jobs. Currently this concerns watcher with file mask definition, whose jobs execute only one task. The file mask watcher can be used for importing a huge amount of data (independently / orthogonally) split over numerous file into os.bee software. Those file could be processed parallel with a same file mask watcher. The amount of parallel executed jobs is limited and determined by the property Max-Parallel-Threads-Count, which can be found and set in the corresponding DataInterchange Configuration File.

Please note that the watcher job from a file name watcher defined with sequential execution type will process its list of tasks in only one dedicated Thread instance; the watcher job from a file name watcher with parallel execution type can execute its tasks in several available threads (not exceeding the maximum thread count).

The maximum thread allowed to be parallel processed (max-parallel-thread-count) is defined in the Datainterchange Configuration File, after keyword entry key="Max-Parallel-Threads-Count" for each Datainterchange Group. In the run time, the maximum value of all max-parallel-thread-count from all Datainterchange Groups will be used to set the runtime max-parallel-thread-count.

The current allowed maximum value is 15, and default value is 5. If this value could not be found from any Datainterchange Configuration File for any Datainterchange Groups, the default count will be used to set the runtime max-parallel-thread-count.

2 watcher types

• file mask watcher:
  • We provide the possibility to set a file mask to extend the further specify (and identify) on a file name basis, the defined task will be executed when the matched files creations occur in the monitored directory.
  • Keyword file mask is used to define the identification criteria (<filename>.<extension> or with regular expression), you define the name of files, which have to be created and match the given file mask, as trigger event.The file mask corresponds to widely used glob-pattern, which is actually used in most (if not all) operating systems for user to make a file search using the wildcard. More information can be found here: Glob-syntax or here
  • One or more matched target-files can be found in the same monitored directory each time.
  • The directory to monitor is determined by the only one Datainterchange Unit in file mask watcher.
  • One or more watcher jobs can be found in the same job group of watcher job queue.
  • For each watcher job, only one watcher task can be defined.
  • The watcher job group will be executing one after another, and
    • for parallel execution : at most max-parallel-thread-count threads of jobs in the same job group can be executing at the same time.
    • for sequential execution : the jobs in the same job group will be executing one by one.

► Example:

watcher WithMask filemask "Gtin*.csv" parallel from Brands {
	import Gtin
}

the file mask watcher WithMask monitors the import directory of Datainterchange Unit Gtin from Datainterchange Configuration File, when the file names match the mask "Gtin*.csv" (Gtin123.csv, Gtinabcd.csv...) occur, the watcher jobs with the same task import Gtin will be added to the same watcher job group into watcher job queue and the jobs in the same job group will be executed parallel with at most max-parallel-thread-count threads.

• file name watcher:
  • A specified file name can be defined as the target file, and a list of tasks will be executed when the matched file creation occurs in the monitored directory.
  • Keyword file name is used to define the identification criteria (<filename>.<extension>), you define the name of file, which have to be created and match the given file name, as trigger event.
  • Only one matched target-file will be found in the monitored directory each time, so only one watcher job can be found in the same group of watcher job queue.
  • The directory to monitor is determined by the Datainterchange Unit which marked with the keyword applyon in filename watcher.
  • For each watcher job, one or more watcher tasks can be defined.
  • The job will be executing one after another, and
    • for parallel execution : at most max-parallel-thread-count threads of tasks in the same job can be executing at the same time.
    • for sequential execution : the tasks will be executing one by one.

► Example:

watcher WithName filename "brands.csv" parallel from Brands {
	import Brandowner
	import Brandtype
	import Brand
	import Pkgtype
	import Gtin applyon
	export Brand
}

the file name watcher WithName monitors the import directory of Datainterchange Unit Gtin from Datainterchange Configuration File, when the file named "brands.csv" occurs, the watcher job with task list import Brandowner ... export Brand will be added to the watcher job queue, the tasks of the job from the same task list will be executed parallel with at most max-parallel-thread-count threads.

Scheduler

The scheduler functionalities are built upon the Quartz Job Scheduler Framework.

Scheduler jobs are timely executed at run-time based upon their scheduler definition.

► Syntax:

scheduler <scheduler name>
	 <scheduler expression>
	[sequential|parallel]
	from <DataInterchange Group>
{
	(import|export <DataInterchange Unit>)*
}

scheduler expression

You are able to define an hourly, a daily, a weekly or a monthly schedule plan, to set when to execute the list of tasks of a Scheduler. Additionally, you can also be more straightforward and define a more complex schedule plan with cron-expressions.

More information about how to build cron-expressions can be find here.

hourly scheduler

hourlyat <minute>

• The minute entry has to be between 0 and 59.

► Example:

scheduler HourlyJob hourlyat 55 from Brands {
	export Brandowner
}

the scheduler HourlyJob will be executed every hour at the 55th minute.

daily scheduler

dailyat <hour> : <minute>

• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler DailyJob dailyat 18 : 10 from Brands {
	export Gtin
}

the scheduler DailyJob will be executed every day at 18:10.

weekly scheduler

weeklyon <Day Of Week Enum> at <hour> : <minute>

• Day Of Week Enum is including sunday, monday, tuesday, wednesday, thursday, friday and saturday.
• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler WeeklyJob weeklyon saturday at 6 : 15 from Brands {
	export Brandtype
}

the scheduler WeeklyJob will be executed on every saturday at 6:15.

monthly scheduler

monthlyon <day of month> at <hour> : <minute>

• The day of month entry has to be between 1 and 31.
• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler MonthlyJob monthlyon 28 at 20 : 45 from Brands {
	export Brand
}

the scheduler MonthlyJob will be executed on every month the 28th day at 20:45.

cron scheduler

cron <cron expression>

• the format of cron expression is defined as a string like "<second> <minute> <hour> <day of month> <month> <day of week>".
  • The value of second entry has to be between 0 and 59.
  • The value of minute entry has to be between 0 and 59.
  • The value of hour entry has to be between 0 and 23, or * for every hour.
  • The value of day of month entry has to be between 1 and 31, or * for every day.
  • The value of month entry has to be between 1 and 12, or * for every month.
  • The value of day of week entry has to be a value of SUN, MON, TUE, WED, THU, FRI, and SAT, or ? for every week day, MON-SUN for every workday.
• some simple examples:
  • "0 45/5 * 1/1 * ?" | "0 45/5 * * * ?":

    Do the job every 5 minutes from the 45th minute of every hour

  • "0 0 0/2 1/1 * ?" | "0 0 0/2 * * ?":

    Do the job on every 2 hours from 0:00

  • "0 5 8 1/4 * ?" :

    Do the job on every 4 days at 8:05

  • "0 5 12 ? * MON,TUE" :

    Do the job on every Monday & Tuesday at 12:05

  • "0 0 12 5 6 ?" :

    Do the job every year on June 5th at 12:00

  • "0 0 6 1 8/3 ?" :

    Do the job on the first day of month every 3 months (count from August to December) at 6:00

  • "0 0 17 ? 1/1 FRI#2" | "0 0 17 ? * FRI#2":

    Do the job on the second Friday every month at 17:00

  • "0 53/5 10/5 29/1 * ?" :

    Do the job every day (count from every month the 29th till the end of the month), every 5 hours (count from every day 10 o’clock till the end of the day), every 5 minutes (count from the 53th minutes of every hour till the end of the hour).

► Example:

scheduler CronJob cron "0 0/5 * * * ?" from Brands {
	import Brandowner
	import Brand
	export Gtin
}

this scheduler CronJob will be executed every 5 minutes.

Copyright Notice

All rights are reserved by Compex Systemhaus GmbH. In particular, duplications, translations, microfilming, saving and processing in electronic systems are protected by copyright. Use of this manual is only authorized with the permission of Compex Systemhaus GmbH. Infringements of the law shall be punished in accordance with civil and penal laws. We have taken utmost care in putting together texts and images. Nevertheless, the possibility of errors cannot be completely ruled out. The Figures and information in this manual are only given as approximations unless expressly indicated as binding. Amendments to the manual due to amendments to the standard software remain reserved. Please note that the latest amendments to the manual can be accessed through our helpdesk at any time. The contractually agreed regulations of the licensing and maintenance of the standard software shall apply with regard to liability for any errors in the documentation. Guarantees, particularly guarantees of quality or durability can only be assumed for the manual insofar as its quality or durability are expressly stipulated as guaranteed. If you would like to make a suggestion, the Compex Team would be very pleased to hear from you.

(c) 2016-2024 Compex Systemhaus GmbH