Data Selection

# Contents

- Overview
- User Interface

- Database Listview
- Data Selection Listview
- Time Interval Editor

- Tips/FAQ
- Bugs

# Overview

The QSAS Data Selection window facilitates the ingestion of data from a database of files. The window presents a hierarchical view of this data, enabling the user to determine the data availability simultaneously from several sources and perform data retrieval for selected time intervals without any knowledge of file locations and names etc.

The window comprises three main components:-

- Database Listview. This presents a hierarchical view of the data available for the selected database and time interval.
- Data Selection Listview. This lists the data sources selected for data retrieval.
- Time Interval Editor. This enables the user to manipulate the time interval to be analysed/selected.

Each of these individual components is described in more detail below...

# User Interface

Choosing Data Selector from the File menu of the QSAS Main Window brings up the window shown below.

Selecting Open Database from the Database menu will produce a file dialog, from which the user is requested to select a Database Configuration Map (.dcm) file for opening. This is a flat ASCII file, which has been previously produced from the Create Database Entries menu item.

## Database Listview

Once the database has been opened, a hierarchical list view of the data sources contained therein will be produced in the Database listview, together with the data availability over the selected time range. If the user has not manually selected a time interval prior to opening, the system will default to the last day stored on the database.

The data availability for a particular data source, is displayed as a single row of cells. The number of cells is determined such that all the available data fits within the viewport. Thus, the period of time represented by one cell is dependent on the number of cells shown and the currently selected time interval. One can increase or decrease the resolution by simply editing the time interval in the Time Interval Editor and selecting a smaller or larger duration, respectively.

The cells are highlighted when some data is available in the interval covered by the cell and the button background colour when no data exists in that interval.

The finest resolution at which the data availability is stored in the Database Configuration Map (.dcm) file, is at the resolution of percentiles of the time range covered by each data file. So, for example, the data availability entry for the data source `/cluster/csds/jp/PMP/C1/Pred_B_mag__C1_JP_PMP` over the period 2002-04-01 00:00:00.000 to 2002-04-31 23:55:00.000 (from file C1_JP_PMP_20020401_V10.cdf) is stored at a resolution of the order of 7 hours, and likewise for the entry for the same data source over the period 2002-03-01 00:00:00.000 to 2002-03-31 23:55:00.000 (from file C1_JP_PMP_20020301_V17.cdf).

Right-clicking on any data source item in the Database listview will produce a sub-menu of options:-

- Copy - will copy this data source item to the clipboard. The user can then right-click in the Data Selection listview and select Paste, to add this data source to the data selection list.
- Data Browser - will popup a browse box displaying a slider range covering the number of data entries for the currently selected time interval, and a string representation of the data contents of the displayed entry number.

- File List - will popup a list box of the names of the files containing data for the selected data source, over the current time interval.

  Right-clicking on any of the file names in the list box, will produce a sub-menu containing one item - Meta Data. Selecting this will produce a browse box, displaying the meta data (Global Attributes) specific to this file.

- Meta Data - will popup a browse box displaying the meta data (Variable Attribuites) for the selected data source.

- Select DataSource - will add the data source item to the Data Selection listbox

## Data Selection Listview

In addition to the two methods described above for adding a data source to the Data Selection listview,(i.e. Copy and Paste, Select DataSource), the user also has the ability to drag a datasource from the Database Listview (by depressing the left mouse button)and drop it in the Data Selection listview (by releasing the left mouse button). Highlighting a data source in the Database listview and pressing the >>Add>> pushbutton will also have the same effect.

The selected data source items should appear in the Data Selection listview, displayed on the right-hand side of the screen.

Information (e.g. file names, the number of data entries and the data coverage as a percentage) pertaining to the highlighted item in the Data Selection listview, is displayed in the tabs immediately below this list.

The current selection can be cleared at any time by selecting the Clear<< pushbutton, and a particular data source may be removed from the list, by highlighting it in the Data Selection listview and pressing the Remove<< pushbutton.

Pushing the Get Data button will retrieve the data for the data sources named in the Data Selection listview (together with the associated timetags for the selected time interval) and place each of them in an object on the QSAS WorkingList. The Window remains open for further data selection from the same database and must be dismissed manually.

Any database currently open in the Data Selection window, may be closed by either selecting Close Database from the Database menu, or right-clicking on the data base name in the Database listview and selecting the Close Database item. In the former case, if more than one database is currently open, the user will be prompted to select the database to close.

The database files (*.dcm) described above, may be created from the Create Database Entries menu item in the Database menu. This option, when selected, will produce a File Dialog, requesting the user to select an existing directory - this should be the root directory, under which all files of the appropriate type, will have database entries generated for them.

Currently supported file formats include CDF files (which should conform to at least ISTP standard) and Key Value Format (KVF) files.

The user will also be prompted to provide a Data Root Name. This is the top-level name to be used in the naming hierarchy, so that all data sources for which data entries have been generated, will have a hierarchical name commencing with this root, followed by the path name of the directory in which the source files reside e.g. the data source `B_xyz_gse__C2_PP_FGM` will have a hierarchical name of `/cluster/csds_pp/FGM/C2/B_xyz_gse__C2_PP_FGM`, since the data files for this item are in the subdirectory `csds_pp/FGM/C2`, under the selected data root directory.

Finally, the user will be prompted to provide the name of the top-level database to create. This can be an existing database name or a new name. In the former case, the user will be prompted to overwrite the existing database. Selecting Yes at this prompt, will overwrite the database. No will enable the user to reselect a database name, and Cancel, will abort the Create Database Entries operation altogether. (The default directory that will be shown for saving the database files in, will be as set by the environment variable `CONFIG_DIR` if this exists, if not it will be `PWD`.)

The system will then recursively step through each sub-directory of the selected data root directory, and generate a database file for each directory with the appropriate data entries, wherever data files are encountered. The user will receive a visual feedback of the system's progress through the directory hierarchy and has the opportunity to abort the operation at any point.

On completion, the user will be prompted to select a database for opening. This can be the top-level database generated by the Create Database Entries option, or any of the lower-level databases in the directory-naming hierarchy.(NB The lower-level databases will have a name that is generated from the the top-level database name (as provided by the user), and the relative path name of the sub-directory containing the underlying data, (but with the "/"'s replaced by "_"'s.)

For example, the data that is specific to instrument FGM on board spacecraft two of the cluster mission is stored in the sub-directory `./csds_pp/FGM/C2` and thus the corresponding database file for this data will be called `cluster_csds_pp_FGM_C2.dcm` (where `cluster.dcm` is the name of the top-level database file.) `cluster_csds_pp_FGM.dcm` is similarly the database file for all data pertaining to the FGM instrument on board all four spacecraft.

Update Database Entries, also in the Database menu, enables incremental updates to be made to databases previously created from the Create Database Entries menu option. The user will be prompted to select the top-level database file for updating, via a file dialog.

The system will proceed to recursively step through each sub-directory associated with this database, and update the relevant database files where appropriate. On completion, the user will again be prompted to select a database for opening (selecting Cancel results in no database being opened.)

## Time Interval Editor

The time interval to be examined can be manipulated in the Time Interval Editor.

The time interval displayed will default to the period commencing at the start of yesterday's day and ending at the start of today, if no database is currently opened. Otherwise, the date displayed will be the last day of information stored on the current database.

The start and end time fields can be edited directly by entering the relevant date/time string in the appropriate field. The string must be in International Standard ISO 8601 format (e.g."2001-09-27 08:54:37.500"). The hyphens are not mandatory input nor is the time. The time will default to 00H 00M 0.000S if not provided. For example "20010927" will be a valid string, as will "20010927 8:54" etc.

The start time, end time and centre time fields will also all accept the following inputs:-

- "t" or "T" for today's date.
- "y" or "Y" for yesterday's date.
- "n" or "N" for "now" i.e. the current date/time string.

The user also has the option of specifying the time interval in a number of different ways or modes.

The default mode is Start - End, as displayed in the combo-box pulldown, on the right-hand side of the editor. For this, the start time and end time fields are open for input, and the user simply provides this information by direct keying into these fields, or selecting the appropriate date from the pulldowns available (which display a history list of previously entered/selected date/time strings for the current session.)

Other modes available are:-

- Centre - Duration - the centre time and duration fields are open for input. The user should provide the centre time of the interval as an ISO string, together with the duration of the interval in terms of d (days), h(hours), m(minutes) and s(seconds) e.g. 10d 2h 3m 1.001s.
- Start - Duration - as described above, with the exception that the user should provide the start time as an ISO string, instead of the centre time.
- End - Duration - as for Centre - Duration, but with the end time provided instead of the centre time.
- Link To Object - this option opens up the link to object field, which contains a pull-down of time interval objects from the Time Interval Objects list. The user selects which time interval object to use, and the time interval is then obtained from the specified object.

Providing all input fields in the Time Interval Editor have valid entries in them, the user can save the current time interval to the Time Interval Objects list, by pressing the Save to working list... button.

The user will be requested to provide a name for this new Time Interval Object.

This name will then appear in the Time Interval Objects list of the QSAS Main Window.

If the user selects a name which is already in use, the system will attempt to rename the new object by appending the letter 'A' to the original name. This exercise is repeated until the resultant name is unique or until the maximum number of renaming attempts has been exceeded. In the latter case, the save operation will fail.

# Tips/FAQ

- If a new sub-directory is added into a directory hierarchy, which has already been mapped into a database using the Create Database Entries menu option, the user should perform the following steps, in order to add the new data to the database.
  - Run Create Database Entries and select the new sub-directory as the data-root directory.
    Example: the directory `C5`, has been added to the directory `/mission/cluster/csds/data/csds_pp/FGM`. The user wishes to update the database `cluster.dcm`, which was previously mapped from the data-root directory `/mission/cluster/csds/data`, with the contents of the new sub-directory. In this instance, the user should select `/mission/cluster/csds/data/csds_pp/FGM/C5`, as the data-root directory from the Create Database Entries prompt.
  - Input the appropriate Data Root Name - in our example, this would be `/cluster/csds_pp/FGM/C5` if the Data Root Name for the original database was `cluster`.
  - Enter a suitable name for the new database - e.g. in our example this would be `cluster_csds_pp_FGM_C5.dcm`.
  - Once the database has been created, select Cancel at the Open Database prompt and select the menu option Merge Database Entries from the Database menu.
  - At the Select Database Configuration Maps for Merging prompt, select the original database file together with the new database just created (e.g. `cluster.dcm` and `cluster_csds_pp_FGM_C5.dcm`). NB. Pressing Ctrl plus the left mouse button will allow multiple selections to be made in this prompt.
  - Enter the name of the new database (this should be a new name) e.g. `cluster_plus.dcm`.
  - At the Enter default file priority prompt, press OK.
  The system will then merge the two databases. The resultant database (e.g. `cluster_plus.dcm`) will contain data from the original database, together with the data entries in the new sub-directory. Running Update Database Entries on the merged file will add any subsequent updates made.
- To create a database which is restricted to only some of the sub-directories, the user should perform the following steps.
  - Make separate dcm maps, using Create Database Entries, for each of the sub-directories of interest e.g. `/mission/cluster/csds/data/csds_pp/EDI` mapped to `/csds_data/cluster_csds_pp_EDI.dcm`, `/mission/cluster/csds/data/csds_pp/CIS` mapped to `/csds_data/cluster_csds_pp_CIS.dcm`, `/mission/cluster/csds/data/csds_pp/FGM` mapped to `/csds_data/cluster_csds_pp_FGM.dcm`, and `/mission/ace` mapped to `/other_data/ACE.dcm`.
  - Outside of QSAS, create an empty directory e.g. `gep_dir`.
  - Make soft links in the newly created directory to the existing dcm's that are to be included in the restricted database e.g.in `gep_dir`
    `ln -s /csds_data/cluster_csds_pp_EDI.dcm ./EDI.dcm`
    `ln -s /csds_data/cluster_csds_pp_CIS.dcm ./CIS.dcm`
    `ln -s /csds_data/cluster_csds_pp_FGM.dcm ./FGM.dcm`
    `ln -s /other_data/ACE.dcm ./ACE.dcm`
  - In the Data Selection window in QSAS, select Merge Database Entries from the Database menu and at the Select Database Configuration Maps for Merging window multiply select [Shift plus Left Mouse Button to extend or Ctrl plus Left Mouse Button to add a single element] the required dcm files from the list of soft links in the newly created directory e.g. `gep_dir` (N.B. this is the reason why the dcms to be included need to be in the same directory as soft links, rather than working from the originals. If the ACE data was not required in this example, or it was mapped in the same place as the csds/pp data i.e. `/csds_data`, then the step to create the soft links can be omitted.)
  - Press the Open (or OK) button and input a file name e.g. `GEP.dcm` at the Save as Database Configuration window, followed by Save.

# Bugs

- The ability to drag a time interval object from the Time Interval Objects list on the QSAS Main Window and drop it into the link to object field is currently not implemented, but will be available a future version.
- The Copy, Data Browser, File List, Meta Data, Paste, and Select DataSource sub-menu options are currently only accessible by selecting a data source and depressing the right mouse button. In the next release, these options will also be available from the menu bar.

Page created by Janet Barnes, csc-support-dl@imperial.ac.uk

Last up-dated: October 2008 A Rochel