LSST Stack Summer 2015 Release
The Summer 2015 release of the LSST stack is internally numbered "11.0" (git tag) or v11_0 (EUPS distrib).
These release notes document notable changes since 10.1, which was the Winter 2015 release.
Major Functionality and Interface Changes
Improved semantics for loading
MaskedImages from arbitrary FITS files
MaskedImage represent image data with associated mask and variance information. When serialized to FITS, these are stored as three consecutive extensions in the FITS files. It is possible to load
MaskedImages from multi-extension FITS files which were not generated by LSST, but, due to the limitations of the FITS data model, it is not possible to ensure that the creator of the file adhered to the LSST convention: while an image object may be successfully instantiated, its contents may not be logically consistent.
We now go to greater lengths to check that the information in the file is consistent with the LSST standard, warning the user – and in some cases refusing to proceed – if it does not.
Improved support for non-standard FITS headers
The LSST stack is now capable of loading FITS files which contain non-standard headers of the form
nn=5..16), as written by SCAMP, and
EQUINOX headers with a "
J" prefix, as written by SkyMapper.
(DM-2883, DM-2924, DM-3196)
It is now possible to perform instrument signal removal on an
Exposure which has no
Detector-like object object which supports returning gain and saturation level, was added to make it possible to run
saturationDetection if required.
PVi_j header cards are correctly saved to FITS files
This makes it possible to round-trip e.g. TPV headers.
Changes to compound fields and delimiters in Catalog Schemas.
In the older ("version 0") approach to table schemas, we had several compound field types (
Coord) which behaved differently from other field types - the square bracket
 operators could not be used to access them, and they could not be accessed as columns (though their scalar subfields – e.g. "x" and "y" for
Point – could be). In version 0, we used periods to separate both words and namespace elements in field names, but converted periods to underscores and back when writing to FITS. These schemas were mostly produced by the old measurement framework in
SourceMeasurementTask, which was removed in the 10.1 release..
In the new ("version 1") approach, compound objects are simply stored in catalogs as their constituent scalars, with helper classes called
FunctorKeys provided to pack and unpack them from
FunctorKeys that replace the old compound fields are all in
). Unlike the original compound fields, there's no limit to how many types of
FunctorKey we can have, or what package they can live in, making the system much more extensible. By making the constituent scalar objects what the
Schema object knows about, it will be much easier to map a
Schema to other table representations that don't know about LSST classes (e.g. SQL or Pandas). Most
FunctorKeys can be used anywhere a regular
Key can be used. Also, in version 1, we use underscores as namespace separators, and CamelCase to separate words, eliminating some ambiguity between word and namespace boundaries. The new measurement framework in
ForcedMeasurementTask uses version 1 tables exclusively.
In previous releases of the pipeline, version 0 schemas were deprecated but still supported. They have now been removed, but old catalogs saved as version 0 will still be readable - they will be converted to version 1 on read, with period delimiters converted to underscores, and all compound fields unpacked into scalar fields that can be used with a corresponding
FunctorKey. This procedure obviously does not preserve field names, but all slot definitions will be preserved, so code that only relies on slot or minimal schema accessors (
getPsfFlux(), etc.) should not need to be modified.
Allow for use of Approximate (Chebyshev) model in background estimation
In previous releases, the only method for background estimation was to use an interpolation scheme (constant, linear, or various splines). These schemes tend to lead to over-subtraction of the background near bright objects. The Approximate (Chebyshev) approach to background estimation greatly improves the background subtraction around bright objects. The relevant code to use this latter approach (including persistence and backwards compatibility issues) is now in place.
While the intention is to eventually set the Approximate background subtraction scheme as the default, there is some clean-up and restructuring that needs to be done before resetting the defaults (which may also require adjusting some defaults in the calibrate stage to be more appropriate for the approximation, as opposed to interpolation, scheme). Therefore, the default setting has not been changed (i.e. the default is still to use an interpolation scheme for background estimation). The Chebychev approximation can be selected for background estimation through configuration parameters in the obs_CAMERA packages, i.e. useApprox=True and, optionally, approxOrderX (approximation order in X for background Chebyshev), approxOrderX (approximation order in Y for background Chebyshev: currently approxOrderY must be equal to approxOrderX), weighting (if True, use inverse variance weighting in calculation).
Multi-band processing for coadds
See the description of the multi-band coadd processing work performed in S15 for details. In short, four new command-line Tasks have been added for consistent multi-band coadd processing:
Detect sources (generate Footprints for parent sources) and model background for a single band.
Merge Footprints and Peaks from all detection images into a single, consistent set of Footprints and Peaks.
Deblend and measure on per-band coadds, starting from consistent Footprints and Peaks for parent objects.
Combine separate measurements from different bands into a catalog suitable for driving forced photometry. Essentially, it must have a centroid, shape, and CModel fit for all objects, even for objects that were not detected on the canonical band. Will assume that all input catalogs already have consistent object lists.
Enable use of deblended HeavyFootprints in coadd forced photometry
Given the new multi-band processing for coadds (above), we now have a reference catalog that is consistent across all bands. This catalog allows the use of the source's HeavyFootprints to replace neighbors with noise in forced photometry, thus providing deblended forced photometry and consistent deblending across all bands. This provides much better colors for blended objects as well as measurements for drop-out objects that do not get detected in the canonical band. This functionality has been enabled for forced coadd photometry.
See the description of the multi-band coadd processing work performed in S15 for further motivation of this change.
Limited the fractional number of masked pixels per source
CModel has difficulties modelling backgrounds in vignetted regions of the focal plane, leading to a performance bottleneck. To mitigate the issue, if the fractional number of masked pixels in a particular source exceeds a given threshold, that source will be skipped.
Peak culling around large objects
An excess of "junk" peaks may be observed around large objects. Given the new multi-band processing architecture (above), these must be consistently removed across bands. We therefore provide a method to consistently "cull" this peaks at an earlier stage, immediately after merging and sorting in
Parent Footprints are the union of their children
Footprints are now trimmed so that they are strictly the union of their children: any pixels which are not assigned to a child are removed. This mitigates an issue whereby stray flux from the parent was not correctly assigned to the children. Note that this has the consequence that parent
Footprints are not necessarily contiguous.
Large Footprints may be skipped on initial processing
For practical processing purposes (specifically total processing time and memory limits due to current hardware limitations), we have the option to skip over objects with large
Footprints during large-scale processing, with the intention to return to these objects to "reprocess" them using different hardware in future. The ability to properly record which objects have been skipped and require further processing has been implemented along with optimizations to the deblender configuration for the maximum number of
Footprint, and the size and area of
Command line tasks for measurement transformation
The measurement transformation framework provides a generic mechanism for transforming the outputs of measurement plugins in raw units, such as pixel positions or flux, to calibrated, physical units, such as celestial coordinates or magnitudes. Appropriate transformations are defined on a per-measurement-plugin basis, and may make use of the calibration information and WCS stored with the data.
This system is designed such that the transformation of a given catalog is performed by a command line task. Different catalog types (such as
forced_src, etc) make use of separate command line tasks. In this release, we provide a variety of tasks to handle different source types.
- Documentation for generic transforms
- Documentation for
- Documentation for
- Documentation for
(DM-2191, DM-3473, DM-3483)
NO_DATA mask plane
Previously, we have used the
EDGE mask plane to indicate both pixels which are off-the-edge of the detector, and hence have no data available, and pixels near the edge which cannot therefore be properly searched for sources. Here, we introduce the
NO_DATA plane to refer to the former case and now use
EDGE strictly for the latter.
Add slot for flux used in photometric calibration
We define a new slot,
SourceRecords. This slot is used to record the flux used for photometric calibration, rather than hard-coding the name of a particular algorithm in the
PhotoCal task. This slot defaults to a 12 pixel circular aperture flux, the previous default in
Table field prefix for aperture flux measurements changed
Our aperture flux measurement algorithms take a list of radii, in pixels, which define the radii over which measurements should be made. Previously, the names of the table fields produced by the algorithm were defined purely based on the position of the radius in that list (thus, the first radius listed would produce a flux field named
PluginName_0_flux). This has been changed so that the fields are now named after the radius, regardless of its position in the list. Thus, a 12.5 pixel aperture will reult in a field named
PluginName_12_5_flux, regardless of its position in the list.
Faster astrometry reference catalog loading
The reference catalog loading was optimised by caching HEALpix identifiers for the catalog files. This has been observed to speed up loading times from 144 sec to 12 sec.
The cache is saved as
andCache.fits in the astrometry catalog directory. The use of the cache can be disabled through the
andConfig.py file (or the
AstrometryNetDataConfig) by setting
False. To prepare a cache,
setup astrometry_net_data and use the
generateANetCache.py script that now comes in
Bad pixels tracked when coadding images
When co-adding images, we now keep track of what fraction of the input data for a given pixel was masked. If the total masked data exceeds some user-configurable threshold, the mask is propagated to the coadd.
Polygon masking in coadded PSFs
Polygonal masks are used to define the usable area of the focal plane; they can be used to, for example, exclude vignetted areas from coaddition. We now take account of these masks to determine which PSF images to included when building co-added PSFs.
Scale counts to reflect CCD-specific zero-points when warping to create coadd inputs
Solving astrometry with distortions
The default astrometry matcher (
matchOptimisticB) can now match stars against a reference catalog when the stars are distorted (e.g., at the outskirts of a wide field imager) if there is an estimate of the distortion available.
Rejection iterations in astrometry fitting
Astrometric fitting (
FitTanSipWcsTask) now includes support for iterative fitting with rejection.
Inclusion of external package PSF Extractor as option for PSF determination
PSFEx is currently the state of the art external package for PSF determination, used in projects such as DES. LSST wrappers were created such that PSFEx could be used as a plugin in place of the built in PSF determiner. Tests with Hyper Supreme Camera data have shown that PSFEx out performs the built in PSF determiner. (DM-2961)
Improvements to CModel magnitude measurement
This release includes many miscellaneous improvements and fixes resulting from testing on HSC data, including:
- parameter tuning for computational performance improvement
- correction to uncertainty estimation to account for extrapolation beyond the fit region
- much more robust flagging of failure modes
Interface changes to forced measurement
The order of arguments to the forced measurement task was reversed, so that it takes a source catalog followed by an
Exposure. This brings it into line with the single frame measurement interface.
N-way spatial matching
A simple utility class for performing spatial matches between multiple catalogs with identical has been added as
lsst.afw.table.multiMatch.MultiMatch. This is intended as a stop-gap measure until more flexible and efficient functionality becomes available, but is already usable.
Display CCD data as laid out in the focal plane
It is now possible to use
lsst.afw.cameraGeom.utils to display CCD data laid out in the focal plane. An example of how this functionality works in practice is available as an IPython notebook.
The following fixes resolve problems visible to end users.
Build and Code Improvements
These improvements should not usually be visible to end users. They may be important for developers, however.
Release notes for Qserv and Webserv
These release notes focus on the Science Pipelines part of the LSST Stack, but there are other important components developed by Data Management teams.
An up-to-date list of known issues for 11.0 affecting external users is available.
Measurements & Characterisation
A summary of Stack characterisation measurements is available.