Difference between revisions of "Signal DSL"

Revision as of 15:26, 9 November 2018

Purpose

POS master data updates over signal watchers, to execute a list of tasks (for now only import and/or export via datainterchange) depending on newly created files. This so called trigger-file will be deleted after the execution of all the tasks (or actions) weither they are be executed sequentially or parallel.

A client wishes to export a certain amount of data (sales, reports, master data...) on a specific working day (or not) at a certain time, either once or several time on this date. Therefore, it should now be possible to define such characteristic inside a model.

SignalDSL

The SingalDSL defines the java functions for Os.bee and the functions will be used in other Os.bee models. The main semantic elements of the SingalDSL are:

• package - The root element that contains all the other elements. (now in one file only one package could be defined, the grammar could be altered to support several packages in one file, if needed)
• import declarations - Used to import external models or other Java classes.
• watcher - define a watcher which monitors a directory.
• filemask - define a group of target files to watch and all detail about the watcher.
• filename - define a target file to watch and detail about the watcher.
• scheduler - define the simple or complex schedulers for executing tens, hundreds, or even tens-of-thousands of jobs.

package defintion

With the keyword package followed by a qualified name, you define the root element that contains all the other elements. A model can contain multiple packages.

► Syntax:

package <package name>[{ 
	(
	  watcher <watcher name>
		  filemask <file mask> . . .
		| filename <file name> . . .

	| scheduler <scheduler name> . . .
	)* 
}]

• one or more watcher/scheduler can be defined in the same package.
• 2 types of watcher can be defined, they are file mask watcher and file name watcher.
• The trigger policy defines how the execution of its list of tasks be triggered: either file based (the creation of a file) or scheduled plan based.
  • Currently we only support the export and import functions of existing datainterchange units from the datainterchangeDSL as the possible watcher’s/scheduler’s tasks to be executed.
  • The properties of datainterchangeDSL are saved in the datainterchange configuration files which are generated with the datainterchangeDSL.
  • The datainterchange configuration files are default saved in C:\Users\<user name>\.osbee. Per datainterchange group, a <groupname>Config.xml file will be generated automatically. This file path can be changed in Eclipse--> Preferences--> OSBP Application Configuration -->External Data Source --> Datainterchange Setting.
  • The corresponding watcher's and scheduler's properties defined in SingalDSL will be added into the corresponding datainterchange-group-configuration files after saving the SignalDSL file.
  • In cases the needed datainterchange properties are not available in the configuration file, or the configuration file doesn't exist, the default definition of the properties from the datainterchangeDSL file will be used.
  • Currently it is not possible to monitor changes on a (web)url-based directory (destination). You may however define a watcher with a url-based directory and set schedulers to import data from it.
  • Watcher/scheduler generates watcher/scheduler jobs, they will be stored into a queue and processed by the watcher/scheduler job handler. Keyword sequential and parallel determinate how the jobs will be executed. As same as the literally meaning, Keyword sequential means all tasks/jobs will be sequentially executed; Keyword parallel means all tasks/jobs will be parallel executed.

► Datainterchange Group:

group Brands {
	interchange Brandowner merge deleteFileAfterImport file CSV "C:/datatransfers/BrandOwner.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brandowner
	}

	interchange Brandtype merge deleteFileAfterImport file CSV "C:/datatransfers/BrandType.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brandtype
	}

	interchange Brand merge deleteFileAfterImport file CSV "C:/datatransfers/Brand.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Brand lookup {
			for brandtypeid in Brandtype createOn "/csv-set/csv-record" mapFrom "id" mapTo id for brandownerid in Brandowner createOn "/csv-set/csv-record" mapFrom "id" mapTo id
			}
		mapping {
			map brandtypecd to "brandtypecd" map bsin to "bsin" mapBlob bsinmedia to "bsin" extension "jpg" path "C:/POD/brands/brand" mimeType jpg map brandnm to "brandnm" map brandlink to "brandlink" map id to "id"
		} 
	}

	interchange Gtin merge deleteFileAfterImport file CSV "C:/datatransfers/Gtin.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Gtin lookup {
			for brandid in Brand createOn "/csv-set/csv-record" mapFrom "id" allowNoResult mapTo id
		}
	}

	interchange Pkgtype merge deleteFileAfterImport file CSV "C:/datatransfers/PkgType.csv" delimiter ";" quoteCharacter """ skipLines 1 path {
		entity Pkgtype
	}
}

► BrandsConfig.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>dataInterchange file URLs</comment>
<entry key="Pkgtype-export">C:/datatransfers/PkgType.csv</entry>
<entry key="Max-Parallel-Threads-Count">3</entry>
<entry key="Brand-export">C:/datatransfers/Brand.csv</entry>
<entry key="Pkgtype-import">C:/datatransfers/PkgType.csv</entry>
<entry key="Brandtype-export">C:/datatransfers/BrandType.csv</entry>
<entry key="Brand-import">C:/datatransfers/Brand.csv</entry>
<entry key="Brandtype-import">C:/datatransfers/BrandType.csv</entry>
<entry key="Brandowner-export">C:/datatransfers/BrandOwner.csv</entry>
<entry key="Gtin-export">C:/datatransfers/Gtin.csv</entry>
<entry key="Brandowner-import">C:/datatransfers/BrandOwner.csv</entry>
<entry key="Gtin-import">C:/datatransfers/Gtin.csv</entry>
</properties>

Watcher

Watcher jobs are generated by a watcher, they will be stored into a queue and processed by the watcher job handler.

► Syntax:

watcher <watcher name>
	filemask <file mask>
		sequential | parallel
		from <DataInterchange Group>
	{
		import|export <DataInterchange Unit> [applyon]
	}

	| 
	filename <file name>
		sequential | parallel
		from <DataInterchangeGroup>
	{
		(import|export <DataInterchange Unit> [applyon])*
	}

• 2 types of watcher can be defined, they are:
  • file mask watcher

    ► Example:

    watcher WithMask filemask "Gtin*.csv" parallel from Brands {
    	import Gtin
    }
    

  • file name watcher

    ► Example:

    watcher WithName filename "brands.csv" parallel from Brands {
    	import Brandowner
    	import Brandtype
    	import Brand
    	import Pkgtype
    	import Gtin applyon
    	export Brand
    }
    

Target File

A target file is the file which is generated inside the monitored directory, whose filename (+extension) matches the file mask or the file name.

The datainterchange file used for import processes will be deleted at the end of a successful import, if it is explicitly set in its datainterchange definition. See [Datainterchange DSL documentation page] for more information.

In case the target file is at the same time used in datainterchange as the import file, it will be deleted at the end of the execution of the watcher job (parallel or sequential), even if it’s not set in the datainterchange definition.

If the import process is failed, at the end of the import process, the corresponding import file will be renamed with form "FAILEDIMPORT" +< actual date formated >+ "Original file name.LOCKED" (as the definition in the following blog post).

Keyword applyon

The keyword applyon has been introduced in order to define per watcher, which directory has to be monitored for the triggering events: creations of target files in this directory, which match the given file mask or file name. The watcher-monitored directory corresponds to the import-path ( in configuration file after keyword entry key ="<datainterchange uinit >-import" ) of the Datainterchange Unit marked with the keyword applyon as the watcher tasks in the SignalDSL file. If a configuration file (the DatainterchangeDSL configuration file, which also contains the SignalDSL properties) exists, the needed import-path property of the Datainterchange Unit exists and its value is valid, then this file path will be used to monitor. Otherwise, the default path of the Datainterchange Unit defined in the corresponding DatainterchangeDSL file will be used as the monitored directory.

Watcher Job

The watcher job queue is used to secure the sequential execution of each watcher jobs (task/list of tasks). Responsible for the sequential handling of each queued watcher jobs. The jobs generated at the same time from the same file mask watcher can be called as a job group. A grouping of jobs from the same watcher is actually taking place in order to better make use of multi-threading functionalities for the watchers which are allowed a parallel execution of their jobs (only one for a file mask watcher). Grouping the execution of jobs from the same watcher is mostly useful in the case of parallel execution of the watcher jobs. Currently this concerns watcher with file mask definition, whose jobs execute only one task. The file mask watcher can be used for importing a huge amount of data (independently / orthogonally) split over numerous files into os.bee software. Those files could be processed parallel with a same file mask watcher. The amount of parallel executed jobs is limited and determined by the property Max-Parallel-Threads-Count, which can be found and set in the corresponding DataInterchangeDSL configuration file. Please note that the watcher job from a file name watcher defined with sequential execution type will process its list of tasks in only one dedicated Thread instance; the watcher job from a file name watcher with parallel execution type can execute its tasks in several available threads (not exceeding the maximum thread count). The maximum thread allowed to be parallel processed (max-parallel-thread-count) is defined in the datainterchange configuration file, after keyword entry key="Max-Parallel-Threads-Count" for each Datainterchange Group. In the running time, the maximum value of all max-parallel-thread-count from all Datainterchange Groups will be used to set the runtime max-parallel-thread-count. The current allowed maximum value is 15, and default value is 5. If this value could not be found from any datainterchange groups, the default count will be used to set the runtime max-parallel-thread-count.

2 watcher types

• file mask watcher:
  • We provide the possibility to set a file mask to extend the further specify (and identify) on a file name basis, the defined task will be executed when the matched files creations occur in the monitored directory.
  • Keyword file mask is used to define the identification criteria (<filename>.<extension> or with regular expression), you define the name of files, which have to be created and match the given file mask, as trigger event.The file mask corresponds to widely used glob-pattern, which is actually used in most (if not all) operating systems for user to make a file search using the wildcard. More information can be found here: Glob-syntax or here
  • One or more matched target-files can be found in the same monitored directory each time.
  • The directory to monitor is determined by the only one Datainterchange Unit in file mask watcher.
  • One or more watcher jobs can be found in the same job group of watcher job queue.
  • For each watcher job, only one watcher task can be defined.
  • The watcher job group will be executing one after another, and
    • for parallel execution : at most max-parallel-thread-count threads of jobs in the same job group can be executing at the same time.
    • for sequential execution : the jobs in the same job group will be executing one by one.
• file name watcher:
  • A specified file name can be defined as the target file, and a list of tasks will be executed when the matched file creation occurs in the monitored directory.
  • Keyword file name is used to define the identification criteria (<filename>.<extension>), you define the name of file, which have to be created and match the given file name, as trigger event.
  • Only one matched target-file will be found in the monitored directory each time, so only one watcher job can be found in the same group of watcher job queue.
  • The directory to monitor is determined by the Datainterchange Unit which marked with the keyword applyon in filename watcher.
  • For each watcher job, one or more watcher tasks can be defined.
  • The job will be executing one after another, and
    • for parallel execution : at most max-parallel-thread-count threads of tasks in the same job can be executing at the same time.
    • for sequential execution : the tasks will be executing one by one.

Scheduler

The scheduler functionalities are built upon the Quartz Job Scheduler Framework.

Scheduler jobs are timely executed at run-time based upon their scheduler definition.

► Syntax:

scheduler <scheduler name>
	 <scheduler expression>
	[sequential|parallel]
	from <DataInterchange Group>
{
	(import|export <DataInterchange Unit>)*
}

scheduler expression

You are able to define an hourly, a daily, a weekly or a monthly schedule plan, to set when to execute the list of tasks of a Scheduler. Additionally, you can also be more straightforward and define a more complex schedule plan with cron-expressions.

More information about how to build cron-expressions can be find here.

hourly scheduler

hourlyat <minute>

• The minute entry has to be between 0 and 59.

► Example:

scheduler HourlyJob hourlyat 55 from Brands {
	export Brandowner
}

daily scheduler

dailyat <hour> : <minute>

• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler DailyJob dailyat 18 : 10 from Brands {
	export Gtin
}

weekly scheduler

weeklyon <Day Of Week Enum> at <hour> : <minute>

• Day Of Week Enum is including sunday, monday, tuesday, wednesday, thursday, friday and saturday.
• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler WeeklyJob weeklyon saturday at 6 : 15 from Brands {
	export Brandtype
}

monthly scheduler

monthlyon <day of month> at <hour> : <minute>

• The day of month entry has to be between 1 and 31.
• The hour entry has to be between 0 and 23.
• The minute entry has to be between 0 and 59.

► Example:

scheduler MonthlyJob monthlyon 28 at 20 : 45 from Brands {
	export Brand
}

cron scheduler

cron <cron expression>

• the format of cron expression is defined as a string like <second> <minute> <hour> <day of month> <month> <day of week>.
  • The second entry has to be between 0 and 59.
  • The minute entry has to be between 0 and 59.
  • The hour entry has to be between 0 and 23.
  • The day of month entry has to be between 1 and 31.
  • The month entry has to be between 1 and 12.
  • The day of week entry has to be a value of SUN, MON, TUE, WED, THU, FRI, and SAT.

► Example:

scheduler CronJob cron "0 0/5 * * * ?" from Brands {
	import Brandowner
	import Brand
	export Gtin
}

Copyright Notice

All rights are reserved by Compex Systemhaus GmbH. In particular, duplications, translations, microfilming, saving and processing in electronic systems are protected by copyright. Use of this manual is only authorized with the permission of Compex Systemhaus GmbH. Infringements of the law shall be punished in accordance with civil and penal laws. We have taken utmost care in putting together texts and images. Nevertheless, the possibility of errors cannot be completely ruled out. The Figures and information in this manual are only given as approximations unless expressly indicated as binding. Amendments to the manual due to amendments to the standard software remain reserved. Please note that the latest amendments to the manual can be accessed through our helpdesk at any time. The contractually agreed regulations of the licensing and maintenance of the standard software shall apply with regard to liability for any errors in the documentation. Guarantees, particularly guarantees of quality or durability can only be assumed for the manual insofar as its quality or durability are expressly stipulated as guaranteed. If you would like to make a suggestion, the Compex Team would be very pleased to hear from you.

(c) 2016-2024 Compex Systemhaus GmbH