For more information, see the online learning platform Once the data is loaded from the Lake into your Analytics project, you can select the sampled data with Variable Sets and Record Sets: - Variable Sets: selection of variables (column selection). Once your Variable Set is created, you can use this as a filter each time you must select some variables.
For example, you can create a Variable Set "temperature" with all your temperature variables, so that the temperatures are easier to select when needed. - Record Sets: selection of rows (timestamps or batches) based on some rules (row selection). For example, you can filter the data before a given time or when the production rate is above a given value.
What is a Record?A record is simply an indexed numerical value that identifies a specific instance, data point, in a database. The identifiers are established based on the row index of the table. Record SetsFor more information, see the online learning platform After a file is uploaded into the DATAmaestro environment, you can select operators to define the rules for the record sets you want to create. Info |
---|
| A record set does not delete or change the data, simply filters out specific rows for graphs or models. |
To create a record set:- Click Select > Record set in the menu.
- Change the data source, if required.
- Enter a name for the new record set, for example: Clean Data Set.
- Click Add to open the rules definition.
- Select an operator, see table below, and complete the rule, see Record Set Rules.
- Add more rules, as required.
- Click Compute.
Tip |
---|
| You cannot reorder the rules applied to the set of records. Try to plan the order before you create a series of rules. To remove a specific rule, click Clear, or Clear all to remove them all. |
Tip |
---|
title | Compounding record sets |
---|
| It can be a good idea to work with “compounding” Record Sets to avoid defining the same rules multiple times in multiple different Record Sets. Typically, we recommend defining a Record Set for Clean-Steady State operations and using this as a starting point for other Record Sets (use the rule “Intersect” to begin with another record set). |
Edit a Record SetTo access the Record Set editor: - Click Record Sets on the sidebar to view a list of the saved record sets.
- Click the edit icon (
) for the record set you want to edit.
Tip |
---|
| A record set is a set of data points, specific instances, records or rows in a database. Record sets can be created based on a series of rules (First, Last, Random, Intersect, Filter, etc) or via rulers on all visualization graphs. |
Record Set Rules Name of Operator | What to Enter | How It Is Used | First | Number | Indicates records selected from the front of the current record set. For example, “First 100” will select the first 100 records (or rows) within the selected data set (or record set if combining record set rules). | Last | Number | Indicates records selected from the end of the current record set. For example, “last 100” will select the last 100 records (or rows) within the selected data set (or record set if combining record set rules). | Random | Number | Indicates records selected randomly. For example, “random 100” will randomly select 100 records (or rows) within the selected data set (or record set if combining record set rules). | Subseq | Numbers | Record set rules that span from row n to row m within the selected data set (or record set if combining record set rules). | Not-in | Record set | Record set rule that excludes all data contained within a specified record set. | Union | Record set | Record set rule that allows the combination of an existing record set with additional rules (or with additional record sets). When combining two (or more) record sets using union, this is equivalent to keeping data points that are in either record set 1 “OR” record set 2. | Intersect | Record set | Record set rule that allows the combination of an existing record set with additional rules (or with additional record sets). When combining two (or more) record sets using intersect, this is equivalent to keeping data points that are in both record set 1 “AND” record set 2. | Filter | Variable, control and number | Method for creating record sets based on filtering a particular variable (numerical or symbolic) based on the given filter rules (less than, greater than, etc.). | Filter missing | Variable set | Create rules that removes records (rows), for a given variable set, that have at least one missing value for the variables. NB: Data sets with high proportions of missing data may result in empty record sets. If a value is missing for one variable, the whole record line will be removed from the selection ( = from the record set) ![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/imageX.png?version=1&modificationDate=1672930825822&cacheVersion=1&api=v2&width=500)
Be aware that such a rule: will remove all rows where "Profit/hr" is below 3000, including any rows where “Profit/hr” is missing and that this impacts all variables (Record sets filter entire rows). | Cyclic | Number | Method for creating record sets that Keep or Skip rows of a dataset. This can be used to create learning and testing sets systematically, not completely random. If the record set keep 100 and skip 10, for example, 100 rows are kept in the record set and the next 10 are skipped, then the next 100 are kept and 10 skipped, repeatedly until the end of the dataset. It is useful for orderly records such as time series. | Script filter | Script rules | Method for creating record sets based on scripting rules. Rules can be scripted in Javascript, Python or R. |
Example of record set: ![](https://pepite.atlassian.net/wiki/download/attachments/154566683/image2022-11-24_14-47-35.png?version=1&modificationDate=1669297657690&cacheVersion=1&api=v2)
Info |
---|
title | Rorder or Clone rules |
---|
| Click ( ) to clone rule E.g. To duplicate the selected filter for a specific variable and then add a threshold. Drag and drop to change the order of rules ( ). |
![](https://pepite.atlassian.net/wiki/download/attachments/154566683/image2022-11-24_14-48-31.png?version=1&modificationDate=1669297712922&cacheVersion=1&api=v2)
How to read the results: Initially there are 15867 rows or records in the data set. After filtering rows to keep only >= 250, there are 15830. Finally, after filtering rows to keep only <= 550, there are 15781 rows.
Example of record set using a Script filter: You can add a rule using a script filter. Select the language you are going to use, there are three options : Javascript, R and Phyton. Write the script in the area.
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/image2021-9-2_8-53-43.png?version=1&modificationDate=1630583625323&cacheVersion=1&api=v2&height=250)
Code Block |
---|
language | js |
---|
title | Record set script filter example |
---|
linenumbers | true |
---|
collapse | true |
---|
| val("variable1") <1000 || ( val("variable2") >=80 && val("variable3") == "ON" )
/* Value of variable1 < 1000 or [ value of variable2 >= 80 and value of symbolic variable3 equal My_Symbol ] */ |
Info |
---|
| For the rules First, Last and Random it is possible to select the percentage of the dataset as well as the number of records (Rows). For example, “random percentage 75” will randomly select 75% of the data set. |
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Screen%20Shot%202021-05-11%20at%2012.12.28.png?version=1&modificationDate=1620727962278&cacheVersion=1&api=v2&width=700)
Tip |
---|
title | Find your Record sets |
---|
| Find your record sets more easily with a new filter options. |
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/image2021-5-12_10-15-43.png?version=1&modificationDate=1620807344920&cacheVersion=1&api=v2&width=700)
Tip |
---|
| Depending on the type of analysis and type of outliers, they may or may not need to be removed. For example, if the outlier represents measurement error, it is best to remove. Alternatively, if the outlier represents process upsets that you would like to investigate, they should be left in the data set. Remove outliers by defining data filtering rules with “Record Sets”. |
Info |
---|
title | Export Record Set to Function |
---|
| To create a function from any record set. - Export Record Sets to a function
- Set names for points within or outside the Record Set
- View the result on charts to understand which points are within the Record Set
|
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture%20training%20set.PNG?version=1&modificationDate=1698065368622&cacheVersion=1&api=v2&width=700)
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture%20record%20set.PNG?version=1&modificationDate=1698065368135&cacheVersion=1&api=v2&width=700)
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture%20fv.PNG?version=1&modificationDate=1698065367576&cacheVersion=1&api=v2&width=700)
What is a Variable?A variable is a property or characteristic of a record (for example, the weight of a mechanical piece, the time at which an event occurred or the eye color of a person) that varies from record to record. - numerical: its value is an integer or real number. Such values can obviously be numerically ordered and compared.
- symbolic: its value is a string or symbol. It is qualitative and generally cannot be ordered (except for symbolic variables such that low medium high implying an intuitive order).
Variable Sets Anchor |
---|
| Variable Sets |
---|
| Variable Sets |
---|
|
For more information, see the online learning platform To build models, you may decide to first select a set of variables to use as inputs to the modeling methods. These variables are usually called candidate variables. You may also want to create subsets of variables to represent specific groups of variables (for example, ambient physical characteristics, process parameters, or quality-related variables). To create a variable set:- Click Select > Variable set in the menu.
- Change the data source, if required.
- Enter a name for the new variable set, for example: My Candidates.
Select variables from the list, and click the arrow to add them to the set. - Click Save.
Tip |
---|
| To select multiple variables from the Variable List, use Shift+Click for adjoining variables, and Ctrl+Click to include singles. |
Info |
---|
| On all Editor pages (Charts, models, etc), user preferences will be saved in the browser including: - Resizing column widths to fit information,
- Changing column order in function of importance,
- Hiding empty columns.
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/image2022-11-24_9-50-54.png?version=1&modificationDate=1669279856172&cacheVersion=1&api=v2&height=400)
|
Info |
---|
title | Improve Variable Set selection |
---|
| When a Variable Set is selected, all variables within that set are highlighted: - One Variable Set has been selected, which contains 22 variables
- The 22 variables have been highlighted to indicate that they have been selected via the Variable Set
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture%20vs.PNG?version=1&modificationDate=1698066330636&cacheVersion=1&api=v2&width=750)
|
Edit a Variable setTo access the Variable Set editor: - Click Variable Sets on the sidebar to view a list of the saved variable sets.
- Click the edit icon (
) for the variable set you want to edit.
Tip |
---|
If you want to see what's in a Variable Set and Record Set, you can click on "Reports>Data Export" and select the variables and apply a “Record Set” to filter the data. In this way, you will see a list with all the values from your selection, filtered by your “Record set”. |
Classify VariablesFor more information, see the online learning platform You may want to classify the variables to describe if a variable is manipulable or a disturbance, measure or a set point, reliable or unreliable. Classifiers keep track of key information about variables. Classifiers pre-defined in DATAmaestro Analytics are: - Parameters: characteristic that classifies a type variable in a dataset. E.g.: temperature, pressure, flow, etc.
- Location: classifies a place or equipment. E.g.: Plant 1, etc.
- Signal type: defines the type of signal. E.g.: measurement, setpoint , specification, etc.
- Classification: defines a category for the variable. E.g.: manipulable, disturbance, output, etc.
- Frequency: defines the rate the variable is collected. E.g.: minute, seconds, etc.
- Accuracy: classifies the precision of the variable. E.g.: 0-1%, reliable, unreliable, etc.
- Min/Max: define the minimum and maximum values of the variable.
Info |
---|
| Classifiers are stored with the data source. |
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/capture4.png?version=1&modificationDate=1690881773425&cacheVersion=1&api=v2&width=250)
To classify variables:- Click Select > Classify variables in the menu.
- In variable list enter information for the different variable, for example, in Title: My Title, in Classification : Manipulable.
- Select more than one variable and edit them all at once by clicking on Bulk Edit.
- Click Save.
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture1.PNG?version=1&modificationDate=1690813074605&cacheVersion=1&api=v2&width=600)
To Move, Edit, Hide, Resize and Remove Classifiers (directly from the column header): It is possible to add, edit and remove a classifier directly from the column header: - To move the column header, pass the mouse over the three vertical points icon and then, with the grabbing hand cursor change the column position. You can drop it after you see a blue line indicating the new position.
- To hide a classifier click on - .
- To resize a header column, use | .
- To remove a classifier, click on x.
- To edit a classifier click a cell and select the information.
Tip |
---|
| Remove all the filter by using the "Trash" icon on the top right side of the table. |
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture2.PNG?version=1&modificationDate=1690813334493&cacheVersion=1&api=v2&width=600)
To Add, Edit, Hide and Remove Classifiers (using "Edit classifiers"):- Click on “Edit Classifiers”.
- The “Edit Classifiers” windows is composed of 5 columns allowing to modify classifiers set:
- Change the Name of a classifier
- Change the Description of a classifier
- Change the predefined Values that a classifier can take
- Hide a classifier
- Delete a classifier
- The “+ Add Classifier” icon allows to create a new classifier. A line appears at the end of the list.
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/Capture3.PNG?version=1&modificationDate=1690870305845&cacheVersion=1&api=v2&width=650)
In Script tab: - Select the Language to be used to write the script. There are three options: Javascript, R and Phyton.
- Write the script in the area.
- Click Run to launch the script. The message Done, appears beside the Run button, once the script is finished. If there are errors at the script, you may find an error message is this area. Note that the script in not saved.
- Check the results in the Classify tab.
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/image2021-9-2_11-24-48.png?version=1&modificationDate=1630592690876&cacheVersion=1&api=v2&height=250)
![](https://pepite.atlassian.net/wiki/download/thumbnails/154566683/image2021-9-2_11-26-41.png?version=1&modificationDate=1630592803467&cacheVersion=1&api=v2&height=250)
Code Block |
---|
language | js |
---|
title | Classify variables |
---|
linenumbers | true |
---|
collapse | true |
---|
| In this example, the variables that contains "Temp" in their variable names is classified as "Temperature". Note that the variable name is case sensitive, only those with "Temp" (capital T) are classified and those with "temp" (small t) are not.
It is also possible to replace "contains" with "startsWith".
var attributes = inputs.attributes;
var result = output.createArrayResult('attributes');
for (var i=0; i < attributes.length; i++)
{
var attribute = attributes[i];
if (attribute.name.contains('Temperature') || attribute.name.contains('temperature'))
{
var resultAttribute = {id:i};
resultAttribute.classifiers = {Parameters:'Temperature'};
result.push(resultAttribute);
}
} |
Process Flow This feature helps create a block flow diagram, assign variables to different process steps and then account for time delays and buffers across the process with shifting or moving average calculations. The example below shows a PVC foam seal extrusion process. The idea is to translate this information into a diagram containing the general flow of the process (Input, Output, Equipment, Connection and Flow).
Image Added
To create a process flow diagram:- Click Select > Process Flow in the menu. You can also click on OTHER, on the left bar, and click New. Select Process Flow and then click Create.
The Process Flow Editor will appear. On the Build Process Flow tab, enter a name for the Process flow. The icons are going to be used to create the process flow steps. The process flow diagram available in DATAmaestro Analytics project of a unit process include a series of icons. - Drag and drop icons into the grey shared area below the icons to build the process flow. For this example, add the following process steps:
- 1 x Input
- 2 x Equipment (Equipment and Equipment 2)
- 1 x Output
- 4 x Flow arrows. Note that the flow is only a visual connection. Arrows: To change the “Flow” arrow direction, select an arrow and drag it onto the icon you want it to point at. ”Flow” arrows can point up, down, left or right.
To save the diagram, click on Save. Note: No selection means that no icon is selected. - On the Build Process Flow tab, if you click on the process step (the border of the icon area changes color), you can enter a new label name in Label.
- Input as for example “Natural Gas”.
- Equipment 1: Extruder
- Equipment 2: Winding Motor
- Output: Seal Diameter
- Select an icon and then click on Delete Node, to delete the icon.
Image Added Image Added
Map Tags helps assign tags to different process steps (Optional feature). On the Map Tags tab, start the tag mapping by clicking on a process step and selecting the tags (or variables) that are measured at that process step. Click on a process node, e.g., “Natural Gas”. - The list of tags appears on the empty area above the diagram. Click the tags related to the input variables e.g., ’Gas Flow’ and ‘Gas pressure. Click on the other icons to map the tags of the following process steps.
- For the “Extruder”, select the tags: ‘Zone temperatures’ from 02 to 10 (Zone 01 temperature is empty).
- For the “Winding Motor”, select ‘Motor velocity ‘ and ‘Drum velocity’.
- For the “Seal Diameter”, select ‘Seal Diameter by laser’.
Image Added
Timeshift Tags helps offset process steps and all associated tags in time to account for different time delays and buffers along the process (Optional feature). This feature creates new variables in the Analytics project with the shift applied. On the Timeshift Tags tab, select the time shift methods. Click on the process step in the diagram area. - Select the Shift type.
- Select No shift if no time shift is required. No shift means that no new time shifted variable(s) will be created in this process step.
- Select Fixed shift if it is required. This will create, for each variable, a new variable which is offset in time compared to the original variable(s). The window is specified with Past and Future row counts.
- Enter the value in Shift rows field.
- If the shift row is positive, e.g., 10, the rows for the variables related to the process step are shifted up ten lines compared to the original variable.
- If the shift row is negative, e.g., -10, the rows for the variables related to the process step are shifted down ten lines compared to the original variable.
- Enter Variable suffix that will be added to the original tag names to create the new shifted variables.
- Select Moving average shift if it is required. This will create, for each variable, a new variable which is offset in time compared to the original variable(s). The window is specified with Past and Future row counts.
- Enter the value in Shift rows field.
- If the shift row is positive, e.g., 2, the rows for the variables related to the process step are shifted up one line compared to the original variable
- If the shift row is negative, e.g., -2, the rows for the variables related to the process step are shifted down one line compared to the original variable.
Enter Past row count and Future row count (can be positive or negative values). Past row count specifies the number of previous rows to include in the moving average calculation for each given record. - Future row count specifies the number of next rows to include in the moving average calculation for each given record.
1.Enter Variable suffix that will be added to the original tag names to create the new shifted variables.
- Click Save.
Image Added Image Added Image Added
Info |
---|
title | Process flow step and Classifiers |
---|
| Once tags have been assigned to a process step using the Process Flow tool, the Process Steps classifier will appear on all editor pages to help quickly sort and select variables.
Image Added
|
|