QSAS: A Science Analysis System for Space Plasma Data
Data Handling
Contents
QSAS allows the ingestion of data from a variety of
sources. In all cases, the result of data ingestion is to place one or more data objects onto the Working List.
The Data Selector (the hierarchical view of databases), Open Data File (Direct Import and browse of data files) and Write Data File (data export with join onto common timeline) menu items are accessed through the File pull-down menu on the QSAS Main Window.
Currently supported file formats include Cluster Exchange Format (CEF) files, Common Data
Format (CDF) files (which should conform to at least ISTP standard, though
others may also be imported) and QSAS Flat ASCII files as specified in the
documentation accompanying QSAS. This file format relies on simple ASCII
headers (either attached to the file or co-located) which include the
capability to specify quite general time formats. Thus many (most) foreign
ASCII files can be read by QSAS after writing an appropriate header. Such
direct access to files enables variables to be ingested one at a time. The
direct file interface allows the data values and metadata to be browsed before
import.
Limited PDS format read capability is also provided (.TAB
and .LBL files) although helper files are required for most Rosetta RPC
files.
The results of calculations performed within QSAS may also be saved by exporting them from the Working List. QSAS supports the export of Cluster Exchange Format (CEF) files, ISTP/Cluster-compliant CDF files and flat ASCII files (tabular or delimited). If multiple time-series objects are exported, QSAS will join them onto a common time line using algorithms chosen by the user.
Data may be ingested into QSAS by opening a file directly or using the Data Selector database view.
The Open Data File option brings up a file
selector. When a file is opened a list of the variables in the file will
be presented, and double clicking a variable or selecting the Get Data
button will ingest that data variable along with all its metadata
including timetags. The Window remains open for further imports
from the same data file and must be dismissed manually. Data may be
restricted to a range of records (or time interval) using the From and To time sliders.
Selecting a Global Attribute will open an object browser display for that
attribute. Pressing the Browse Data button will open an object browser display showing the data
and a list of cross references from which individual attributes may be selected
and displayed.
The Data Selector is a more powerful interface for frequently-used databases and gives a hierarchical view of the data,
which enables
the user to determine the availability of data simultaneously from
several
sources and retrieve data for selected intervals without the needing to
know their
locations or file names.
Data files that have insufficient or incorrect metadata for use within QSAS may need Helper Files
When either Direct Import or the Data Selector is used to read data QSAS will also search the directory containing the data file for any files with suffix ".qat". All such helper files found in the directory will be read after the data file and before a variable is placed on the working list. The helper files have the same syntax for attributes contained within variable blocks as cef files. Any attribute information in a variable block will be attached to a variable if it has the same name as the variable block identifier. If the attribute is already present it will be replaced - thus a helper file can be used to correcct metadata as well as add it. No other information about a variable can be read from a helper file, and in this respect it differs from a detached header file. Care must be taken with helper files as the attribute information will be available to any file in the same directory and will silently be applied if the file contains a variable with the same name as one of the variable blocks. Multiple helper files may be placed in a directory and all of them will be read. There is no guarantee of reading order, and hence no control over which helper file takes precedence if a conflict arises.
Helper files should be used sparingly and kept in a dedicated directory containing the data set to which they apply. A sample helper file "ThemisFGM.qat" can be found in the examples directory in the QSAS installation. This shows the syntax for modifying metadata for three of the variables in Themis sample FGM cdf files.
Data objects on the Working List, regardless of whether they have been imported from other data files or created by analysis within QSAS, can be exported to files by choosing Export from the File menu on the Working List. This brings up the Export User Interface as shown. Data Objects to be exported must be dragged or copied and pasted from the Working List (or any other list view showing an object held on the working list) to the export list. Objects on the export list can be grouped into folders.
Time Series data objects will be exported together by joining them onto a SINGLE set of time tags, so the source of these tags must be specified (by default the first time series data object is used as the timetag source). The behaviour of this window is identical to the join window. The user may also specify a time interval for the export (i.e., to export only a subset of the data) using a time series data object, a time interval object, or the Time Editor. Since more than one Time Series object may be exported, they must be joined onto the common time tags using either linear interpolation or boxcar averaging, with data gaps filled or removed (see the Analysis section for more details).
Join options will be applied to
any selected items or folders, and different choices may be made for different
selections. The time tag and time interval for the exported data apply to all
items being exported as QSAS unless Do Not Join is selected. In
the case that data are not joined and of different lengths the export
window will give an option to pad to the same length. Only a CDF file
may have variables of different lengths.
On selecting Export a dialogue appears allowing the
user to choose the file name and directory (QSAS will make an automatic choice by
default) and provides a check box to indicate whether an identically named file
should be overwritten or the export aborted (to allow the user to specify a
new, unique name). The bottom selection slot allows the user to choose the file
type to be written. If one of the ASCII types is chosen the panel on the right
will affect the choices for these files. Note: at present these options
always appear active even if they have no effect on the file type selected.
In the Direct Import window, all quantities identified as variables in the data file are shown. In CDF files, and their QSAS ASCII equivalents, many of these do not vary with records, and include labelling and other information. These do not need to be separately ingested unless the user needs them for other purposes.
When Time Series data are imported the time tag variable (usually denoted "epoch" in CDF files) is automatically ingested at the same time. The time tags will not appear on the working list, but are kept internally as cross references to the data objects.
Foreign ASCII files can usually be imported by first writing a simple ASCII header to describe the variables, file format, and time specification. These headers are described in other documentation.
Page created by Steve Schwartz, csc-support-dl@imperial.ac.uk
Last up-dated: October 2016 Tony Allen