Implementation notes for CDF
----------------------------

File formats:
   Single file only supported (not multiple file)

Numeric encodings:
   Unsupported: VMS D_FLOAT, G_FLOAT

Compression formats:
   All supported.

Data types:
   Unsigned integer types supported, but transformed to larger signed types
   (CDF_UINT1 -> short, CDF_UINT2 -> int, CDF_UINT4 -> long).
   CDF_UCHAR treated like CDF_CHAR (-> char, which is 16 bit anyway).

   CDF_EPOCH, CDF_EPOCH16, CDF_TIME_TT2000 treated as double, double[2],
   long respectively, not obviously times.
   Options: transform them to ISO8601 strings on input, or make sure
   TOPCAT can transform them to epochs (in that case you could plot them,
   but they wouldn't look OK in the table view, stats window etc).
   Probably best to transform these to iso8601 strings in the STIL layer.
   Think about how that affects precision for CDF_EPOCH16 and leap seconds.
   Is precision at that level important?

   Time string formatting to ISO-8601 is performed for the time formats,
   but currently done wrong for TT2000 since it does not cope with
   leap seconds.

CDF Data Format version:
   Version 3.4 supported (v3.4 document used for implementation).
   Following notes in that document, probably 2.6, 2.7, 3.* are also 
   supported, maybe others, but I haven't checked them all.

Large files:
   No file size restriction.  Files >2Gb are allowed, but require use
   of a 64-bit system (OS+JVM).

Fill values:
   Implemented for CDF and CEF for scalars and floating point arrays.
   However, I can't make integer array elements behave like nulls
   without significant changes to the framework.  Most of these CDFs
   seem to have a lot of array-valued columns.  Are fill values in
   integer array values must used?  likely to cause trouble?

I/O:
   Read access only, output not supported at all.

Array access:
   Read raw array or shaped array - less flexibility than HyperRead.


Implementation notes for CEF
----------------------------

Version:
   Working from CEF version 2.0.  Document suggests that 1.0 is
   not likely to work, though I haven't seen 1.0 specification.
   I don't know if any other versions exist.

Data types:
   There are places where the CEF standard is not very explicit.
   For instance it doesn't say how many bits INTs/FLOAT/DOUBLEs have,
   whether a BYTE is signed, or whether the fill value is to be
   matched as a string or as a typed value.
   I've looked at some examples and made my best guess.

Syntax:
   INCLUDE = "filename" not supported.

Array data:
   CEF specifies C-type array ordering, and STIL uses FORTRAN/FITS-type
   array ordering.  Not quite sure what to do about this.  Could transform
   on read, but it would be inefficient, and if the data is addressed
   as a vector (which is natural) anyone expecting CEF-ordered arrays
   would get it wrong.

Metadata:
   Lots of per-column metadata (TENSOR_FRAME, DEPEND_i, etc etc) read
   in and available in GUI but otherwise ignored.


CEF/CDF release:
----------------

Code structure:
   Libraries for basic CDF and CEF access, with optional STIL layers
   on top to provide the TOPCAT/STILTS integration.

Code status:
   Code is public on github (https://github.com/mbtaylor/cdf)
   but not formally released.

   Javadocs mostly done.

   Some tests involving comparison of simple data files with results
   from CDF library tools, and matching CDF files that have been
   modified using CDF format conversion tools.  This is not a bad
   suite of tests, though more could be added.  Some CDF data types
   not tested, since I can't find and test data
   (e.g. EPOCH16, TIME_TT2000 variables).  More tests on time formatting
   would be good too.

Release questions:
   Any opinions on how release should be done?
   What priority is (quality of) independent CDF/CEF release?
   Java namespace (int.esa.cdf, uk.ac.starlink.cdf, uk.ac.bristol.star.cdf)?


Starjava integration:
---------------------

Auto format detection:
   CDF yes, CEF no.  CEF could do, but if the FILE_FORMAT_VERSION
   is more than 512 bytes into the file it could cause trouble.

Treeview:
   Not implemented.  Could do.

SAMP:
   You can send a CDF or CEF to TOPCAT using SAMP with the non-standard
   MType table.load.stil:

      { "samp.mtype": "table.load.stil",
        "samp.params": {
           "url": "file://localhost/mbt/data/cdf/C4_V120822.cdf",
           "format": "cdf" } }

   As well as being non-standard, this MType was not documented in the
   TOPCAT user documentation at v4.0-b, though it will be in later
   releases.

   Discussion of table.load.cdf (and maybe .cef) under way on apps-samp.


Plans:
------

CDF/CEF I/O:
   Public independent CDF library release (when?)
   CEF within STILTS?  doesn't really deserve its own library.

TOPCAT visualisation:
   Implement time series layer plot (like stacked line plot?)
   Implement better time axis labelling
   Implement time series/array plots (what are these called?)
   Implement external plot control

Requirements:
   Talk to Chris Perry at RAL when I have reasonably working line and
   vector plots to get feedback about functionality etc.

