Data Selection

Contents


Overview

The QSAS Data Selection window facilitates the ingestion of data from a database of files. The window presents a hierarchical view of this data, enabling the user to determine the data availability simultaneously from several sources and perform data retrieval for selected time intervals without any knowledge of file locations and names etc.

The window comprises three main components:-

Each of these individual components is described in more detail below...


User Interface

Choosing Data Selector from the File menu of the QSAS Main Window brings up the window shown below.

Data Selection

Selecting Open Database from the Database menu will produce a file dialog, from which the user is requested to select a Database Configuration Map (.dcm) file for opening. This is a flat ASCII file, which has been previously produced from the Create Database Entries menu item.

Select Database Configuration File

Database Listview

Once the database has been opened, a hierarchical list view of the data sources contained therein will be produced in the Database listview, together with the data availability over the selected time range. If the user has not manually selected a time interval prior to opening, the system will default to the last day stored on the database.

The data availability for a particular data source, is displayed as a single row of cells. The number of cells is determined such that all the available data fits within the viewport. Thus, the period of time represented by one cell is dependent on the number of cells shown and the currently selected time interval. One can increase or decrease the resolution by simply editing the time interval in the Time Interval Editor and selecting a smaller or larger duration, respectively.

The cells are highlighted when some data is available in the interval covered by the cell and the button background colour when no data exists in that interval.

The finest resolution at which the data availability is stored in the Database Configuration Map (.dcm) file, is at the resolution of percentiles of the time range covered by each data file. So, for example, the data availability entry for the data source /cluster/csds/jp/PMP/C1/Pred_B_mag__C1_JP_PMP over the period 2002-04-01 00:00:00.000 to 2002-04-31 23:55:00.000 (from file C1_JP_PMP_20020401_V10.cdf) is stored at a resolution of the order of 7 hours, and likewise for the entry for the same data source over the period 2002-03-01 00:00:00.000 to 2002-03-31 23:55:00.000 (from file C1_JP_PMP_20020301_V17.cdf).

Database listview

Right-clicking on any data source item in the Database listview will produce a sub-menu of options:-

Data Selection Listview

In addition to the two methods described above for adding a data source to the Data Selection listview,(i.e. Copy and Paste, Select DataSource), the user also has the ability to drag a datasource from the Database Listview (by depressing the left mouse button)and drop it in the Data Selection listview (by releasing the left mouse button). Highlighting a data source in the Database listview and pressing the >>Add>> pushbutton will also have the same effect.

The selected data source items should appear in the Data Selection listview, displayed on the right-hand side of the screen.

Data Selection ListView

Information (e.g. file names, the number of data entries and the data coverage as a percentage) pertaining to the highlighted item in the Data Selection listview, is displayed in the tabs immediately below this list.

The current selection can be cleared at any time by selecting the Clear<< pushbutton, and a particular data source may be removed from the list, by highlighting it in the Data Selection listview and pressing the Remove<< pushbutton.

Pushing the Get Data button will retrieve the data for the data sources named in the Data Selection listview (together with the associated timetags for the selected time interval) and place each of them in an object on the QSAS WorkingList. The Window remains open for further data selection from the same database and must be dismissed manually.

Any database currently open in the Data Selection window, may be closed by either selecting Close Database from the Database menu, or right-clicking on the data base name in the Database listview and selecting the Close Database item. In the former case, if more than one database is currently open, the user will be prompted to select the database to close.

The database files (*.dcm) described above, may be created from the Create Database Entries menu item in the Database menu. This option, when selected, will produce a File Dialog, requesting the user to select an existing directory - this should be the root directory, under which all files of the appropriate type, will have database entries generated for them.

Select Data Directory

Currently supported file formats include CDF files (which should conform to at least ISTP standard) and Key Value Format (KVF) files.

The user will also be prompted to provide a Data Root Name. This is the top-level name to be used in the naming hierarchy, so that all data sources for which data entries have been generated, will have a hierarchical name commencing with this root, followed by the path name of the directory in which the source files reside e.g. the data source B_xyz_gse__C2_PP_FGM will have a hierarchical name of /cluster/csds_pp/FGM/C2/B_xyz_gse__C2_PP_FGM, since the data files for this item are in the subdirectory csds_pp/FGM/C2, under the selected data root directory.

Finally, the user will be prompted to provide the name of the top-level database to create. This can be an existing database name or a new name. In the former case, the user will be prompted to overwrite the existing database. Selecting Yes at this prompt, will overwrite the database. No will enable the user to reselect a database name, and Cancel, will abort the Create Database Entries operation altogether. (The default directory that will be shown for saving the database files in, will be as set by the environment variable CONFIG_DIR if this exists, if not it will be PWD.)

The system will then recursively step through each sub-directory of the selected data root directory, and generate a database file for each directory with the appropriate data entries, wherever data files are encountered. The user will receive a visual feedback of the system's progress through the directory hierarchy and has the opportunity to abort the operation at any point.

On completion, the user will be prompted to select a database for opening. This can be the top-level database generated by the Create Database Entries option, or any of the lower-level databases in the directory-naming hierarchy.(NB The lower-level databases will have a name that is generated from the the top-level database name (as provided by the user), and the relative path name of the sub-directory containing the underlying data, (but with the "/"'s replaced by "_"'s.)

For example, the data that is specific to instrument FGM on board spacecraft two of the cluster mission is stored in the sub-directory ./csds_pp/FGM/C2 and thus the corresponding database file for this data will be called cluster_csds_pp_FGM_C2.dcm (where cluster.dcm is the name of the top-level database file.) cluster_csds_pp_FGM.dcm is similarly the database file for all data pertaining to the FGM instrument on board all four spacecraft.

Update Database Entries, also in the Database menu, enables incremental updates to be made to databases previously created from the Create Database Entries menu option. The user will be prompted to select the top-level database file for updating, via a file dialog.

The system will proceed to recursively step through each sub-directory associated with this database, and update the relevant database files where appropriate. On completion, the user will again be prompted to select a database for opening (selecting Cancel results in no database being opened.)

Time Interval Editor

Time Interval Editor

The time interval to be examined can be manipulated in the Time Interval Editor.

The time interval displayed will default to the period commencing at the start of yesterday's day and ending at the start of today, if no database is currently opened. Otherwise, the date displayed will be the last day of information stored on the current database.

The start and end time fields can be edited directly by entering the relevant date/time string in the appropriate field. The string must be in International Standard ISO 8601 format (e.g."2001-09-27 08:54:37.500"). The hyphens are not mandatory input nor is the time. The time will default to 00H 00M 0.000S if not provided. For example "20010927" will be a valid string, as will "20010927 8:54" etc.

The start time, end time and centre time fields will also all accept the following inputs:-

The user also has the option of specifying the time interval in a number of different ways or modes.

The default mode is Start - End, as displayed in the combo-box pulldown, on the right-hand side of the editor. For this, the start time and end time fields are open for input, and the user simply provides this information by direct keying into these fields, or selecting the appropriate date from the pulldowns available (which display a history list of previously entered/selected date/time strings for the current session.)

Other modes available are:-

Providing all input fields in the Time Interval Editor have valid entries in them, the user can save the current time interval to the Time Interval Objects list, by pressing the Save to working list... button.

The user will be requested to provide a name for this new Time Interval Object.

Save Time Interval

This name will then appear in the Time Interval Objects list of the QSAS Main Window.

If the user selects a name which is already in use, the system will attempt to rename the new object by appending the letter 'A' to the original name. This exercise is repeated until the resultant name is unique or until the maximum number of renaming attempts has been exceeded. In the latter case, the save operation will fail.


Tips/FAQ


Bugs


Page created by Janet Barnes, csc-support-dl@imperial.ac.uk

Last up-dated: October 2008 A Rochel