Once the data is loaded from the Lake into your Analytics project, you can select the sampled data with Variable Sets and Record Sets:

• Variable Sets: selection of variables (column selection). Once your Variable Set is created, you can use this as a filter each time you must select some variables.
  For example, you can create a Variable Set "temperature" with all your temperature variables, so that the temperatures are easier to select when needed.
• Record Sets: selection of rows (timestamps or batches) based on some rules (row selection). For example, you can filter the data before a given time or when the production rate is above a given value. 

What is a Record?

A record is simply an indexed numerical value that identifies a specific instance, data point, in a database. The identifiers are established based on the row index of the table.

Record Sets

Anchor
Record Sets Record Sets

After a file is uploaded into the DATAmaestro environment, you can select operators to define the rules for the record sets you want to create.

To create a record set:

1. Click Select > Record set in the menu.
2. Change the data source, if required.
3. Enter a name for the new record set, for example: Clean Data Set.
4. Click Add to open the rules definition.
5. Select an operator, see table below, and complete the rule, see Record Set Rules.
6. Add more rules, as required.
7. Click Compute.

Edit a Record Set

To access the Record Set editor:

1. Click Record Sets on the sidebar to view a list of the saved record sets.
2. Click the edit icon () for the record set you want to edit.

Record Set Rules

Example of record set: 

How to read the results: 

Initially there are 15867 rows or records in the data set.

After filtering rows to keep only >= 250, there are 15830.

Finally, after filtering rows to keep only <= 550, there are 15781 rows.

Example of record set using a Script filter: 

You can add a rule using a script filter. Select the language you are going to use, there are three options : Javascript, R and Phyton. Write the script in the area. 

val("variable1") <1000 || (  val("variable2") >=80 &&  val("variable3") == "ON"   )  
    /* Value of variable1 < 1000 or [ value of variable2 >= 80 and value of symbolic variable3  equal My_Symbol ] */ 

What is a Variable?

A variable is a property or characteristic of a record (for example, the weight of a mechanical piece, the time at which an event occurred or the eye color of a person) that varies from record to record.

• numerical: its value is an integer or real number. Such values can obviously be numerically ordered and compared.
• symbolic: its value is a string or symbol. It is qualitative and generally cannot be ordered (except for symbolic variables such that low medium high implying an intuitive order).

Variable Sets

Anchor
Variable Sets Variable Sets

To build models, you may decide to first select a set of variables to use as inputs to the modeling methods. These variables are usually called candidate variables.

You may also want to create subsets of variables to represent specific groups of variables (for example, ambient physical characteristics, process parameters, or quality-related variables).

To create a variable set:

1. Click Select > Variable set in the menu.
2. Change the data source, if required.
3. Enter a name for the new variable set, for example: My Candidates.

4. Select variables from the list, and click the arrow to add them to the set.

5. Click Save.

Edit a Variable set

To access the Variable Set editor:

1. Click Variable Sets on the sidebar to view a list of the saved variable sets.
2. Click the edit icon () for the variable set you want to edit.

Classify Variables

You may want to classify the variables to describe if a variable is manipulable or a disturbance, measure or a set point, reliable or unreliable. 

Classifiers pre-defined in DATAmaestro Analytics are:

• Parameters: characteristic that classifies a type variable in a dataset. E.g.: temperature, pressure, flow, etc.
• Location: classifies a place or equipment. E.g.: Plant 1, etc.
• Signal type: defines the type of signal. E.g.: measurement, setpoint , specification, etc.
• Classification: defines a category for the variable. E.g.: manipulable, disturbance, output, etc.
• Frequency: defines the rate the variable is collected. E.g.: minute, seconds, etc.
• Accuracy: classifies the precision of the variable. E.g.: 0-1%, reliable, unreliable, etc.
• Min/Max: define the minimum and maximum values of the variable.

To classify variables:

1. Click Select > Classify variables in the menu.
2. In variable list enter information for the different variable, for example, in Title: My Title, in Classification : Manipulable. 
3. Click Save.

To Add, Edit, Hide and Remove Classifiers (directly from the column header): 

It is possible to add, edit and remove a classifier directly from the column header:

1. To edit a classifier Name, click on the header (it turns light green), edit the Name and click “Enter”. 
2. To hide a classifier, you can click on - , directly from the header.
3. To remove a classifier, click on x.
4. To add a New classifier, click on +  (last column). In tab, New Custom Classifier, enter Name, Description (Optional) and Values. Click “Add”.
5. Hidden Classifiers tab, is where stand the classifiers that were hidden. In case, you want to display them click on the checkbox. Click “Add”.

To Add, Edit, Hide and Remove Classifiers (using "Edit classifiers"):

1. Click on “Edit Classifiers”.
2. The “Edit Classifiers” windows is composed of 5 columns allowing to modify classifiers set:
   1. Change the Name of a classifier
   2. Change the Description of a classifier
   3. Change the predefined Values that a classifier can take
   4. Hide a classifier
   5. Delete a classifier
3. The “+ Add Classifier” icon allows to create a new classifier. A line appears at the end of the list.

In Script tab: 

1. Select the Language to be used to write the script. There are three options: Javascript, R and Phyton. 
2. Write the script in the area. 
3. Click Run to launch the script. The message Done, appears beside the Run button, once the script is finished. If there are errors at the script, you may find an error message is this area. Note that the script in not saved. 
4. Check the results in the Classify tab. 

In this example, the variables that contains "Temp" in their variable names is classified as  "Temperature". Note that the variable name is case sensitive, only those with "Temp" (capital T) are classified and those with "temp" (small t) are not. 
It is also possible to replace "contains" with "startsWith". 

var attributes = inputs.attributes;
var result = output.createArrayResult('attributes');
for (var i=0; i < attributes.length; i++)
{
   var attribute = attributes[i];
   if (attribute.name.contains('Temperature') || attribute.name.contains('temperature'))
   {
      var resultAttribute = {id:i};
      resultAttribute.classifiers = {Parameters:'Temperature'};
      result.push(resultAttribute);
   }
}