Jobs

Jobs

Jobs in DATAmaestro Lake execute an action at a given frequency. The following types of jobs are available at DATAmaestro Lake. 

From the jobs page, all jobs can be created, edited or deleted. There is also information on a job’s status: executing (indicates job is currently running), pending (waiting for next execution) or error (an error message will be provided).

Create Jobs

From this menu, create jobs for computed tags or exports to DATAmaestro analytics. 

1. Click Manage > Jobs  in the menu.  

2. Click + New job button on top right.

3. Choose a New Job Type. 

4. Click Create. 

On General tab: 

1. Enter Job id:  /folder/Job Name. Example: /demo/Automotive/Computed Tag/PercentageValue.

2. Select Scheduling.

   1. Standard:  Job will be executed a first time after the initial delay and then execution will occur at regular intervals. This regular interval is defined by the period.

   2. Every day:  Job will be executed every day at the same time.

3. Enter Initial delay, if the initial delay is equal to zero the job starts as soon as the Save button is clicked.

4. Put Period, time interval between each job execution. Important remark: it is necessary to put the units, example, for a period of 10 minutes: 10m. 

   Letter	Description
   d	day
   h	hour
   m	minute
   s	second

   Period for a Computed tag: The period does not influence the frequency or number of output values written by the Computed Tag job itself, only when the calculation will run. The frequency and number of output values written by the Computed Tag job depend on the input variable(s) frequency and number. For example, a job with a Period set to 24h will run every 24 hours. If the data uses input tags sampled every minute, then it will write values every minute. Each time the job runs, it looks to see the last output value and continues the calculation from that point. Therefore, if the job last ran 24 hours ago, it will write an output value every minute (for minute sampled data) for the last 24 hours.  The resulting calculation does not depend on the period. 

5. Put the Class name, this indicates the type of job:

   1. Computed tags: be.pepite.pepito.data.historian.tm.jobs.ComputedTagTask

   2. Exports to Analytics: be.pepite.pepito.data.historian.tm.jobs.ExportToAnalyticsTask

On Properties tab (depend on the Job Type): 

Create computed tags

A computed tag calculates functions based on other tags within the Lake.

Two steps to creating a computed tag in DATAmaestro Lake:

1. Create new tag “shell”.

2. Create a job to calculate values to fill new tag “shell”.

Introduction to Lake Access rights:

• All jobs within DATAmaestro Lake must be saved in a folder that has access to the data they will require to make their calculation.

• If a job is created at “Cement Data…” it has access to all tags in the folder “Cement Data” AND any sub folders within Cement Data (i.e., Computed).

• However, if a job is created at “Cement Data/Computed”, it ONLY has access to tags in the sub folder “Computed”.

Note: A forward slash “/” denotes the next level in the folder structure.

To create new computed tags:

This step creates a new empty tag within the DMlake, this is the first step required to create computed tags.

From this menu, create jobs for computed tags or exports to DATAmaestro analytics. 

1. Click Manage > Jobs  in the menu.  

2. Click + New job button on top right.

3. Choose a New Job Type: ComputedTagTask 

4. Click Create. 

On General tab: 

1. Enter Job id. Note: The job must be in a Folder that has access to the necessary data to be executed. 

   1. Click + New Folder 

   2. Select Base Folder. Choose the same Base folder as the tags used for the computed tag executing. 

   3. Enter New Folder name. Example: computed. 

1. Create a New Folder to put all computed tags in the same folder. 

2. Example of Job Id: /Site A/Line 1/computed/Delta (predict - real)

2. Select Scheduling.

   1. Enter At time (hours) : (minutes) 

1. Standard:  Job will be executed a first time after the initial delay and then execution will occur at regular intervals. This regular interval is defined by the period.

2. Every day:  Job will be executed every day at the same time.

3. If scheduling Standard, enter Initial delay, if the initial delay is equal to zero the job starts as soon as the Save button is clicked.

4. Put Period, time interval between each job execution. Important remark: it is necessary to put the units, example, for each 10 minutes : 10m. Period for a Computed tag: The period does not influence the frequency or number of output values written by the Computed Tag job itself, only when the calculation will run. The frequency and number of output values written by the Computed Tag job depend on the input variable(s) frequency and number. For example, a job with a Period set to 24h will run every 24 hours. If the data uses input tags sampled every minute, then it will write values every minute. Each time the job runs, it looks to see the last output value and continues the calculation from that point. Therefore, if the job last ran 24 hours ago, it will write an output value every minute (for minute sampled data) for the last 24 hours.  The resulting calculation does not depend on the period. 

5. Class name indicates the type of job. This field is  automatically filled. 

1. Computed tags: be.pepite.pepito.data.historian.tm.jobs.ComputedTagTask

On Properties tab: 

1. Before creating a computed tag, first you need to create a New Tag. The new tag created is empty of data and the calculation performed by the computed tag will fill this tag with data.  

1. Click + New Tag button. 

2. Enter Folder, path is automatically filled. 

3. Enter tag Name. 

4. Enter Title, if required.  

5. Enter Type, Numeric or Symbolic. 

6. Enter Unit, if required. 

7. Enter Description, if required. 

8. Click Save. 

2. Select Computed Tag. Note: You can only select tags in the folder or subfolders of the job. 

3. Select Input tags. Note: You can only select tags in the folder or subfolders of the job. 

4. Compute latest value checkbox is checked by default. Lake allows the insertion of several values at a same timestamp. By default, the computed tag is not going to calculate a new value for the last input since it is not sure whether it is going to have another value at the same timestamp. Therefore the computed tag can have one late value. If Compute latest value equals to true it is possible to indicate to the computed tag that it can calculate the last input even if there is a new value at the same timestamp that can be inserted later. 

5. Write the script. The script must be written in Javascript. 

6. Click Save. 

7. Click Truncate to remove data of the computed tags from a particular date onwards. 

In the example below, the new computed tag Percentage of bad units simply transforms the variable Ratio of bad units into a percentage value by multiplying it by 100. 

On Advanced properties tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For computed tags the properties that can be edited are: 

• output tags

• input tags

• script function

• Compute latest value = false 

• output tags

• input tags

• script function

• Compute latest value = true (check the image below, still is Optimistic when true)

For advanced usage: 

Apply tag validity

If for example, you would like to calculate a computed tag with data that is uploaded every minute, however, there is a laboratory measurement that is uploaded only once per day. In this case, you can decide to save this same value during 24 hours (tag validity = 24h). In Advanced properties write the script below. In this example, there is a 3 days validity. Warning: This option should only be used with highly rare data measurements. Please contact PEPITe support for more information. 

{
  "outputTag": "demo/tagAB",
  "tagValidityMap": {
    "/demo/tagA": 259200000,
    "/demo/tagB": 259200000
  },

Create delayed tags

From this menu, shifts a tag backwards or forward in time. 

1. Click Manage > Jobs  in the menu.  

2. Click + New job button on top right.

3. Choose a New Job Type: DelayedTagTask.

4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. In Class name indicates the type of job. This field is  automatically filled. 

1. Delayed tags: be.pepite.pepito.data.historian.tm.jobs.DelayedTagTask

On Advanced properties tab: 

1. Write script. Note that the script must contain: 

• The time shift in ms. Example: for 1h write 3600000. If positive or no signal the tag shifted forward (advanced of 1 hour). If negative the tag is shifted backwards ( delayed by 1 hour). 
• Suffix. Example : _ delayed
• Input tags with path Name. 
• Is Optimistic : true/false. (Compute latest value)

{

  "delay": -3600000, // delayed 1 h

  "suffix": "_delayed",

  "tags": [

    "/folder/site/Tag Name"

  ],

  "isOptimistic": true  

}

Export to Analytics 

From this menu, create jobs to export data to DATAmaestro analytics. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: ExportToAnalyticsTask.
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. In Class name indicates the type of job. This field is  automatically filled. 

1. Delayed tags: be.pepite.pepito.data.historian.tm.jobs.ExportToAnalyticsTask

On Properties tab: 

1. Select Tags. Note: You can only select tags in the folder or subfolders of the job. 
2. Enter Last period, export with a rolling window. For example, the last 72 hours. If defined, it remplaces Start date and End date, therefore they are not employable. Export data is defined as from now  until Last period duration. 
3. Select Start date, set date followed by hour and minutes (Optional). Export data after Start date. Unused if Last period is defined. 
4. Select End date, set date followed by hour and minutes (Optional). Export all data between Start date and End date. If the End date is undefined, export all data after Start date. Unused if Last period is defined.
5. Select Method. To have more details about the methods, please check Export to Analytics. 
6. Select Sampling period.  It defines desired data sampling frequency, for example, a database which should have one value every hour should enter “1h” in this field. 

7. Enter DMFF file name, example : nameDMFFfile_rawextract.dmff. Hint: Search the DMFF file name in Data sources of your DATAmaestro project. 

8. To enter a project first click on + Add project. In Project Id select the project (folder/project name.dom.project) by clicking on the folder button. As Task id select dmff-2 from the list. This information is placed at  the end part of the url pathname of the DMFF file in the data sources. To remove the selected project click on the trash icon button. You can add more than one project for the projects’ list. 
9. Enter the DMFF file name, as example : name_rawextract.dmff. 

On Advanced tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For export to DATAmaestro the properties that can be edited are: 

• tags 
• file name (DMFF) 
• method
• last period 
• project id
• task id 

Explore Imported Model created in DATAmaestro Analytics project

From this menu, create an imported model created in Analytics and exported in JSON format. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: ImportedModelTagTask
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Class name indicates the type of job. This field is  automatically filled. 
1. Model ISHM: be.pepite.pepito.data.historian.tm.jobs.ImportedModelTagTask

On Properties tab: 

1. Enter Model Path. 
   1. In the ISHM model page, click on More Actions. 
   2. Click Export model to JSON. 
   3. The .json file is automatically downloaded on your computer
1. Open the Model Path folder in the ISHM Job in DATAmaestro Lake and click "Upload model file" and select the .json file from your computer.
2. Enter Input tags (Optional). It is a list of tags used to fill model inputs. Use this field if the tag names are not the same as those used in the DATAmaestro project or the tag location needs to be specified. If left empty, the model will try to find input tags in the working folder which is a combination of Job Folder and Input Folder, the latter, if assigned. 
3. Enter Output tags (Optional). The list of tags as a result of the ISHM model. It can be an extensive list therefore it is advisable to let the ISHM Job create it in a specific folder.
4. Select Start Time. The model results will be calculated from this point onwards.

Advanced

1. Enter Input Folder. It is an optional, used to relativize input tags’ path. A relative path is a path which doesn’t start with the root element. 
2. Enter Output Folder. It is an optional, used to relativize output tags’ path. Hint: Enter the New Folder created in step 1 - General tab - example:  /demo/Chemicals/computed_ISHM. 
3. Compute latest value checkbox.  If it is equal to true it is possible to indicate to the computed tag that it can calculate the last input even if there is a new value at the same timestamp that can be inserted later. 

On Advanced tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For model tag task the properties that can be edited are: 

• sample method 
• model path 
• input or output folder 
• start time 

Explore a Model Predict Job created in DATAmaestro Analytics project

From this menu, create a model predict job of a model created in Analytics. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: ModelPredictTagTask
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Class name indicates the type of job. This field is  automatically filled. 
1. Model predict tag: be.pepite.pepito.data.historian.tm.jobs.ModelPredictTagTask. 

On Properties tab: 

1. Enter Project Id which is the path name of  DATAmaestro Analytics project, example : /folder/ProjectName.dm.project. 
2. Enter Task Id, open the task in DATAmaestro, for example linear regression, the last part of the URL is the Task Id.  Example: linear-regression-505.
3. Input tags field only if Input Map is not used. When it is used, it disables the Input map. List of tags used to fill model inputs. Note that the sequence (order) of the tags is important. If left empty, the model will try to find input tags in the working folder. Working folder is a combination of Job Folder and Input Folder, the latter, only if assigned.  

Advanced

1. Output Map is the model output. If the model has more than one output ? If left empty, the model will try to find the output tag in the working folder. Working folder is a combination of a Job Folder and Output Folder, the latter, only if assigned. 
2. Enter Input Folder. It is an optional, used to relativize input tags’ path. A relative path is a path which doesn’t start with the root element. 
3. Enter Output Folder. It is an optional, used to relativize output tags’ path. Hint: Enter the New folder used in step 1 - General tab - example:  /Site A/Line 1/computed_models
4. Compute latest values checkbox.  If it equals to true it is possible to indicate to the computed tag that it can calculate the last input even if there is a new value at the same timestamp that can be inserted later. 
5. Input tags field is used only if you don't want to use the Input map. When it is used, it disables the Input map. List of tags used to fill the model inputs. Note that the sequence (order) of the tags is important. If left empty, the model will try to find input tags in the working folder. Working folder is a combination of Job Folder and Input Folder, the latter, only if assigned. 
6. Output tags field is use only if you don't want to use the Output map. When it is used, it disables the Output map. List of tags that will contain model results (if more than one ouput). Note that the sequence (order) of the tags is important. If left empty, model will create output tags in the working folder. Working folder is a combination of Job Folder and Output Folder, the latter, only if assigned.

On Advanced tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For model predict task the properties that can be edited are: 

• sample method (definition) always snap? 
• input tags’ map 
• project id 
• task id 

Moving Average Tag 

From this menu, create a simple moving average calculated by taking the average of a set of values over a defined period length. It is a technique used to smooth a curve, filter out the noise. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: MovingAverageTagTask.
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Put the Class name, this indicates the type of job:
1. Moving average: be.pepite.pepito.data.historian.tm.jobs.MovingAverageTagTask

On Properties tab: 

1. Select Input Tags. Note: You can only select tags in folders or subfolders of the job. 
2. Enter the time window From To specify the period length for the moving average calculation. 
3. Enter Prefix for the tag name, if required, e.g.: MovingAVG_XXX
4. Enter Suffix for the tag name, e.g.: _AVG. 

On Advanced: 

1. Enter Input Folder. It is an optional, used to relativize input tags’ path. A relative path is a path which doesn’t start with the root element. 
2. Enter Output Folder. Used to define output tags' path inside the job folder.. Hint: Enter the New Folder created in step 1 - General tab - example:  /Site A/Line 1/computed_MovingAverage. 
3. Select Output Tags. It is Optional. Only useful if you don't want to use a prefix and / or suffix mechanism. Output tags will be defined following this list and won't be created based on Prefix and Suffix.
4. Check Compute latest values checkbox. 

In Advanced properties tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For moving average tag task the properties that can be edited are: 

• output tags  
• from to 
• input tags

Script 

From this menu, create a script tag. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: ScriptTagTask.
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Class name indicates the type of job. This field is  automatically filled. 
1. Script tags: be.pepite.pepito.data.historian.tm.jobs.ScriptTagTask

On Properties tab: 

1. Select language. Options: R, javascript or Python. 
2. Write the script in the page editor. 

On Advanced properties tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. 

Note: Examples of script tags will be added soon. 

Synchronizer  

From this menu, copy tags from a remote Lake. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: SynchronizerTask. 
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Class name indicates the type of job. This field is  automatically filled. 

Synchronizer Task: be.pepite.pepito.data.historian.tm.jobs.SynchronizerTask

On Advanced properties tab: 

1. Write script. 

The script must contain: 

• User and Password for the remote Lake 
• Remote path, where the tags are located. 
• Recurse option = true/false.  
• Server URL location.  
• Activate or deactivate the synchronization with parameter Disabled. If true synchronization is deactivated. 

{

  "password": "admin",

  "remotePath": [

    "/demo/computed2"

  ],

  "recurse": false,

  "serverURL": "http://localhost:8888",

  "disabled": false,

  "user": "admin"

} 

Tag Replay

From this menu, it is possible to repeat the tags in another period of time. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: TagReplayTask.
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Put the Class name, this indicates the type of job:
1. Replay tags: be.pepite.pepito.data.historian.tm.jobs.TagReplayTask

On Advanced properties tab: 

1. Write the script. 

{

  "endDate": "2005-08-19T10:00:00Z",  // original end date 

  "replayDate": "2018-11-01T00:00:00Z",  // start date used to replay date

  "startDate": "2005-08-18T13:00:00Z"  // original start date 

}

Mail Alert 

From this menu, send an email alert based on a condition. 

1. Click Manage > Jobs  in the menu.  
2. Click + New job button on top right.
3. Choose a New Job Type: MailAlertTagTask. 
4. Click Create. 

On General tab (please check the general tab explannation in Create Job):

1. Put the Class name, this indicates the type of job:
1. Replay tags: be.pepite.pepito.data.historian.tm.jobs.MailAlertTagTask

On Properties tab: 

1. Enter Alert title, e.g. Gas Pressure Alert. 
2. Enter Alert Duration. It is the minimum duration for a threshold state to be active before an alert is sent. For example, the tag value must be above the threshold during 5 minutes before the alert is sent.
3. Enter Deactivation Duration. It is the minimum duration for a threshold state to return to an acceptable range before an alert state is deactivated. For example, the tag value must return below the threshold during 1 minute before the alert state is deactivated.
4. Enter Mail Recipients. Note: one email address per line. 
5. Enter Mail Throttling Period. It is the minimum time period between two alert emails. For example, send maximum 1 email per day. 
6. Click + Add Condition. 
   1. Select tag(s). Note: you can only select tags in the folder or subfolders of the job. 
   2. Select condition. E.g.: Tag Outside 850 1800, an email will be sent when the numerical tag is outside the range (850-1800). Tag Not Member of GOOD, if the symbolic tag is not equal to GOOD an email will be sent. 
   3. Enter value(s). Note: Member of or Not Member of is the only condition that can be used for symbolic tags. 
      

On Advanced tab: 

In this tab it is possible to edit all the properties of the job in a JSON format. For mail alert task the properties that can be edited are: 

• Condition (Tags, condition and value) 
• Recipients (mails)
• Alert duration 
• Alert title 
• Throttling period 

Edit Jobs

1. Click Admin > Tasks (Jobs)  in the menu.  

2. Click Edit icon.

3. Make changes.

4. Click Save.