QSAS: The QMUL Science Analysis System

Data Handling

# Contents

- Overview
- Data Selection
- View and Direct Import of Data
- Helper Files
- Exporting Data
- Tips/FAQ
- Bugs

# Overview

QSAS allows the ingestion of data from a variety of sources. For frequently-used databases, a hierarchical view of the data enables the user to determine the availability of data simultaneously from several sources and retrieve data for selected intervals without the user knowing the locations, file names, etc. Initial capability is built around data holdings in Common Data Format (CDF) as used by Cluster and ISTP, but the underlying structure is capable of extension to other formats as the need arises.

Data files may also be opened and browsed directly. Currently supported file formats include Cluster Exchange Format (CEF) files, Common Data Format (CDF) files (which should conform to at least ISTP standard, though others may also be imported) and QSAS Flat ASCII files as specified in the documentation accompanying QSAS. This file format relies on simple ASCII headers (either attached to the file or co-located) which include the capability to specify quite general time formats. Thus many (most) foreign ASCII files can be read by QSAS after writing an appropriate header. Such direct access to files enables variables to be ingested one at a time. The direct file interface allows the data values and metadata to be browsed before import.

In all cases, the result of data ingestion is to place one or more data objects onto the QSAS Working List. The Data Selector (the hierarchical view of databases), Open Data File (Direct Import and browse of data files) and Write Data File (data export with join onto common timeline) menu items are accessed through the File pull-down menu on the QSAS Main Window.

The results of calculations performed within QSAS may also be saved by exporting them from the Working List. QSAS supports the export of Cluster Exchange Format (CEF) files, ISTP/Cluster-compliant CDF files and flat ASCII files (tabular or delimited). If multiple time-series objects are exported, QSAS will join them onto a common time line using algorithms chosen by the user.

# Data Selection

This aspect is detailed in this page.

# View and Direct Import of Data

Choosing Open Data File from the File menu of the QSAS Main Window brings up a file selection widget from which the user selects and opens a data file. This brings up the browsing window shown here. Selecting a Global Attribute will open an object browser display for that attribute. Selecting a variable will open an object browser display showing the data and a list of cross references from which individual attributes may be selected and displayed. Browsing may be achieved either by double clicking on an entry in a list or by selecting an item and pressing the Browse Data button.

Pushing the GetData button will retrieve the highlighted variable and its variable attributes (together with its timetags if it is a time-series data variable) and place it in an object on the QSAS Working List. The Window remains open for further imports from the same data file and must be dismissed manually.

Data files that have insufficient or incorrect metadata for use within QSAS may need Helper Files.

# Helper Files

When Direct Import is used to read data it will also search the directory containing the data file for any files with suffix ".qat". All such helper files found in the directory will be read after the data file and before a variable is placed on the working list. The helper files have the same syntax for attributes contained within variable blocks as cef files. Any attribute information in a variable block will be attached to a variable if it has the same name as the variable block identifier. If the attribute is already present it will be replaced - thus a helper file can be used to correcct metadata as well as add it. No other information about a variable can be read from a helper file, and in this respect it differs from a detached header file. Care must be taken with helper files as the attribute information will be available to any file in the same directory and will silently be applied if the file contains a variable with the same name as one of the variable blocks. Multiple helper files may be placed in a directory and all of them will be read. There is no guarantee of reading order, and hence no control over which helper file takes precedence if a conflict arises.

Helper files should be used sparingly and kept in a dedicated directory containing the data set to which they apply. A sample helper file "ThemisFGM.qat" can be found in the examples directory in the QSAS installation. This shows the syntax for modifying metadata for three of the variables in Themis sample FGM cdf files.

# Exporting Data

Data objects on the Working List, regardless of whether they have been imported from other data files or created by analysis within QSAS, can be exported to files by choosing Export from the File menu on the Working List. This brings up the Export User Interface as shown. Data Objects to be exported must be dragged or copied and pasted from the Working List (or any other list view showing an object held on the working list) to the export list. Objects on the export list can be grouped into folders.

Time Series data objects will be exported together by joining them onto a SINGLE set of time tags, so the source of these tags must be specified (by default the first time series data object is used as the timetag source). The behaviour of this window is identical to the join window. The user may also specify a time interval for the export (i.e., to export only a subset of the data) using a time series data object, a time interval object, or the Time Editor. Since more than one Time Series object may be exported, they must be joined onto the common time tags using either linear interpolation or boxcar averaging, with data gaps filled or removed (see the Analysis section for more details).

Join options will be applied to any selected items or folders, and different choices may be made for different selections. The time tag and time interval for the exported data apply to all items being exported as QSAS only supports exporting a single timeline into each data file.

On selecting Export a dialogue appears allowing the user to choose the file name and directory (QSAS will make an automatic choice by default) and provides a check box to indicate whether an identically named file should be overwritten or the export aborted (to allow the user to specify a new, unique name). The bottom selection slot allows the user to choose the file type to be written. If one of the ASCII types is chosen the panel on the right will affect the choices for these files. Note: at present these options always appear active even if they have no effect on the file type selected.

# Tips/FAQ

- In the Direct Import window, all quantities identified as variables in the data file are shown. In CDF files, and their QSAS ASCII equivalents, many of these do not vary with records, and include labelling and other information. These do not need to be separately ingested unless the user needs them for other purposes.
- When Time Series data are imported the time tag variable (usually denoted "epoch" in CDF files) is automatically ingested at the same time. The time tags will not appear on the working list, but are kept internally as cross references to the data objects themselves.
- Foreign ASCII files can usually be imported by first writing a simple ASCII header to describe the variables, file format, and time specification. These headers are described in other documentation.

# Bugs

- None known.

Page created by Steve Schwartz, csc-support-dl@imperial.ac.uk

Last up-dated: October 2008 A Rochel