Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

English us

This features allows the user to export desired data from a specific DATAmaestro Lake folder to any pre-existing DATAmaestro project A .dmff file is created with the desired subset of data (period, frequency, variables) stored in the DATAlake and uploaded (export) it to a specific DATAmaestro project or onto the user’s local drive.

Remark: Data from different DATAlake directories can be merged into a single export .dmff file and thus be used simultaneously for analysis in a DATAmaestro project.

To export :

  1. Click Export in the main menu.
  2. Select From, set date followed by hour and minutes.
  3. Select To, set date followed by hour and minutes. For each time step the window changes proposing new choices.
  4. Enter Interval, it defines desired data sampling frequency or data intervals, for example, a database which should have one value every hour should enter “1h” in this field. Other common period examples include:


    Range (Shortcuts)

    Description

    1s

    Extract data at 1 second intervals

    5m

    Extract data at 5 minute intervals

    1h

    Extract data at 1 hour intervals

    7d

    Extract data at 7 day intervals


  5. Select Method, options: Snap, Average, Interpolation, Most probable, Minimum and Maximum values.

    Method DescriptionMethod type 
    Average

    The average method calculates the weighted arithmetic mean which is based on the value and the duration of each value that spans the given period or time window. For more information on Method average please check below. The formula is: 

    which expands to 

     

    where wi are the weights (or durations) and xi are the values. 

    (1)  Source: Average Wikipedia 

    Info
    titleHint
    Use Average for numerical variables like pressures, temperatures and flows. 


    Forward 

    Interpolation

    Linear interpolation is a type of interpolation which generates new values based on an existing set of values. It is achieved geometrically by drawing a straight line between two adjacent points. If the known points are given by the coordinates (x0,y0) and (x1,y1) , the linear interpolant is a line between these points. For a value x in the interval (x0,x1), the value of y along the straight line is given from the equation of slopes (1)It is useful for continuous values. Moreover the interpolation method is only calculated if there is no recorded value at a time stamp.


    (1)  Source: Wikipedia - Linear Interpolation


    Info

    Use Interpolate for numerical variables like pressures, temperatures and flows.


    Centered 
    Longest record

    Selects the single recorded value or class that has the longest duration for each Period or time window. If the same value or symbol appears twice, the durations are not accumulated.

    Info

    We often use “Longest Record” for symbolic (or numerical) variables like product names or machine states, where it doesn’t make sense to interpolate or average those variables. 


    Forward 
    MinimumSelects the lowest/minimum recorded value for each Period or time window. If there is no record in the time window, last value is returned.Forward 
    Maximum

    Selects the highest/maximum recorded value for each Period or time window. If there is no record in the time window, last value is returned. 


    Info

    You may choose to work with “Min” or “Max” if you are searching for particular events in your data. For example, if you want to know if a valve opened (0 or 1) even once during a time window, you could use “Max”.


    Forward 
    Raw
    Exports raw values. For each timestamp with a value for at least one tag, a row will be created at that timestamp. If no value is available for a tag at a given timestamp, the previous recorded value before that timestamp is applied. 


    Info

    Normally, for batch-series data, you would use the "Raw" method.


    -
    Last

    Selects the last recorded value at or before a given time.



    Info

    You may choose “Last” for a lab product quality measurement, given that the last known value for quality was this last lab measurement. You may also choose “Last” for any variables that are measured “on-change” where the last known value is true until a new value is recorded.


    Backward 
    FirstSelects the first recorded value at or after a given time.Forward
    SnapThe snap methods samples by taking the last stored value within each time interval. In case there are no stored values within a given interval the snap method considers the last value stored. An instantaneous value at each time interval will be extracted, no aggregation is provided.Forward


  6. To select the desired list of variables, firstly choose the Folder where the tags are stored.
  7. Select Tags from the tags’ lists and click the Right arrow to add them to the list for export. To remove a tag from the export list select the Tag and click the Left Arrow.
  8. Repeat steps 7 and 8 to select tags from multiple directories.
  9. Select Var. Naming to choose the tag name. Options: Tag Name Only, Subpath [sub/tag], Full path [/project/sub/tag]. Examples: Gas flow, Chemicals/Gas flow or /demo/Chemicals/Gas flow.
  10. Click Download, this downloads a DMFF or CSV files to the user’s local drive. 


User Advanced User

MethodLimit Limit
Download csvSampled 
  • max 128 columns

  • tag names : max 200 kB*
  • tag names : max 200 kB
Raw
  • max 128 columns

  • tag names : max 200 kB*
  • tag names : max 200 kB
Download DMFF Sampled 
  • tag names : max 200 kB
  • tag names : max 200 kB
Raw
  • max 128 columns

  • tag names : max 200 kB*
  • tag names : max 200 kB
  • file max 100 MB
  • file max 100 MB

*: very unlikely to happen 


Info
titleConvert unix time to excel time

To convert the timestamp in unix time (ms) to excel time, use this formula: 

=((UNIXTIME/60/1000)/60/24)+DATE(1970;1;1)+(2/24) for GMT +02:00


To export a file directly to DATAmaestro Analytics:

  1. Choose the DATAmaestro Folder.
  2. Choose a DATAmaestro Project.
  3. Click Export to DATAmaestro.
    1. Note: Pop-ups must be enabled for this website. If pop-ups are blocked, enable pop-ups and then click Export to DATAmaestro.
  4. DATAmaestro Analytics will open in a new Tab/Window. Under Variables Definition, select “Retrieve” and then click “Save” to complete the export to the DATAmaestro Analytics project .  
    1. Note: For large data sets there may be a delay between clicking export and the project opening in DATAmaestro Analytics.


User Advanced user 

MethodLimitLimit
Export DMFF



Sampled

Raw
  • max 128 columns

  • file max 100 MB
  • file max 100 MB

Tags Extract 

(Analytics) 



Sampled

Raw
  • max 128 columns
  • max 128 columns
  • file max 100 MB
  • file max 100 MB




Export Method: Snap 

The snap methods samples by taking the last stored value within each time interval. In case there are no stored values within a given interval the snap method considers the last value stored. An instantaneous value at each time interval will be extracted, no aggregation is provided.

This method is only available for backward compatibility. It is only useful if a previous extraction have been done with Snap method and the same result is needed. In other cases, Last and First methods should fill the needs and are simpler. 

The table below shows how the snap exported data is selected. 

Raw DataExported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Snap values


01/10/2019 15:37:00

36
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0022
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0026
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0027
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0026
01/10/2019 15:41:0528

01/10/2019 15:41:3526

Export Method: Average 


The average method calculates the weighted arithmetic mean which is based on the value and the duration of each value that spans the given period or time window.

The table below shows how the average exported data is calculated. 

Raw Data 


Exported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Average Values (Weighted) Average Formula


01/10/2019 15:37:0030=(25*30+36*25)/55
01/10/2019 15:37:05 25


01/10/2019 15:37:3536




01/10/2019 15:38:0033.167=(36*5+42*30+22*25)/60
01/10/2019 15:38:05 42


01/10/2019 15:38:35 22




01/10/2019 15:39:0024.167=(22*5+23*30+26*25)/60
01/10/2019 15:39:05 23


01/10/2019 15:39:35 26




01/10/2019 15:40:0025.917=(26*5+25*30+27*25)/60
01/10/2019 15:40:05 25


01/10/2019 15:40:35 27




01/10/2019 15:41:0027.083=(27*5+28*30+26*25)/60
01/10/2019 15:41:05 28


01/10/2019 15:41:35 26



Export Method: Interpolation

Linear interpolation is a type of interpolation which generates new values based on an existing set of values. It is achieved geometrically by drawing a straight line between two adjacent points. It is useful for continuous values. Moreover the interpolation method is only calculated if there is no recorded value at a time stamp.

(1) More information in Wikipedia - Linear Interpolation

The table below shows how the interpolation exported data is calculated. 


Raw DataExported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Interpolated values


01/10/2019 15:37:00

-
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0041*
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0022.833
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0025.166
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0027.833
01/10/2019 15:41:0528

01/10/2019 15:41:3526


*At 15:38:00, the interpolation method considers the previous value 36 at 15:37:35 and the next value 42 at 15:38:05. The new interpolated value is 41, calculated as follows:  36 + ((42-36)/30)*25

Export Method: Longest record

Selects the single recorded value or class that has the longest duration for each Period or time window. If the same value or symbol appears twice, the durations are not accumulated.


Raw DataExported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Longest record values


01/10/2019 15:37:0025
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0042
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0023
01/10/2019 15:39:0523

01/10/2019 15:39:0526



01/10/2019 15:40:0025
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0028
01/10/2019 15:41:0528

01/10/2019 15:41:3526

Export Method: Minimum 

Selects the lowest/minimum recorded value for each Period or time window. If there is no record in the time window, last value is returned.


Raw Data Exported Data 

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Minimum Values


01/10/2019 15:37:0025
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0022
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0022
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0025
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0026
01/10/2019 15:41:0528

01/10/2019 15:41:3526

Export Method: Maximum

Selects the highest/maximum recorded value for each Period or time window. If there is no record in the time window, last value is returned.


Raw Data Exported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

Maximum values


01/10/2019 15:37:0036
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0042
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0026
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0027
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0028
01/10/2019 15:41:0528

01/10/2019 15:41:3526

Export Method: Last 

Selects the last recorded value at or before a given time.


Raw Data Exported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw valuesDateTime (dd/MM/yyyy HH:mm:ss)Last values


01/10/2019 15:37:00-
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0036
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0022
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0026
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0027
01/10/2019 15:41:0528

01/10/2019 15:41:3526


Export Method: First 

Selects the first recorded value at or after a given time.


Raw Data
Exported Data

DateTime

(dd/MM/yyyy HH:mm:ss)

Raw values

DateTime

(dd/MM/yyyy HH:mm:ss)

First values


01/10/2019 15:37:0025
01/10/2019 15:37:0525

01/10/2019 15:37:3536



01/10/2019 15:38:0042
01/10/2019 15:38:0542

01/10/2019 15:38:3522



01/10/2019 15:39:0023
01/10/2019 15:39:0523

01/10/2019 15:39:3526



01/10/2019 15:40:0025
01/10/2019 15:40:0525

01/10/2019 15:40:3527



01/10/2019 15:41:0028
01/10/2019 15:41:0528

01/10/2019 15:41:3526


Info
titleCombine different sampling methods

Did you know you can combine different sampling methods? Change them by clicking here or above before selecting the variables from the left. 



Info
titleUpdate data extraction

Now that you have exported a data source to Analytics, you will learn how to update your data extraction using the Lake Export.

  1. Go to DM Lake, then Export and fill all the information, you must go through all the steps presented previously.
  2. Select the new tags.
  3. Choose the Folder and Project where you want to export the data.
  4. Click on Export to DATAmaestro.
  5. You are directed to the selected project in DM Analytics. Now you have two options:
    1.  To replace the existing datasource (all tasks will be automatically updated): Click on Load data in without changing the selected file in the drop down (Recommended).
    2. To add a new datasource to your project. It is possible to work with multiple datasources in a project, however each task can only use one datasource (no merging provided).

To replace the existing datasource (all tasks will be automatically updated): in DATAmaestro Analytics, click on Load data in without changing the selected file in the drop down (Recommended).

More than one data sources can complexify your project. Before creating a task, first you must select the data source. So, it is advisable to have only one data source in a project.

  1. Go to data sources icon.
  2. Click on Edit of the Cement_Tutorial_XXX DMFF file.
  3. Go to Parameters tab.
  4. Click on the folder icon.


  1. Search and select the file you have just exported.
  2. Click Close.
  3. In Parameters, click on Retrieve to retrieve you tags to the project.
  4. Click on Save.
  5. You have now updated your existing datasource.


To add a new datasource to your project. It is possible to work with multiple datasources in a project, however each task can only use one datasource (no merging provided).

  1. Once your export has finished and you are redirected to DATAmaestro Analytics, click on Retrieve to retrieve your tags in the project.
  2. Click on Save.

You have now added a second datasource to your project. Under the datasource icon, you will see two datasources.



Info
titleWarning

Note on DMFF file the will replace an existant DMFF both with the same name. This means that if the same file is used in two different Analytics projects (for example, if you create a copy of the project) and if the DMFF is updated in only one project, but both projects have the same file name, the DMFF file will be automatically updated in the other project too.  


...