infochimps_schema_template:
    name:           Global Hourly Weather Data 
    handle:        global_hourly_isdlite
    collection:     Hourly Weather Data (ISD Lite) by State, 50+ years
    coll_handle:   global_hourly_isdlite
    tags:           global hourly weather atmospheric pressure temperature huge ncdc ncdc_weather climate isd surface precipitation 
    
    formats: 
      yaml: {}
      
    contributors:
      - name: National Climactic Data Center
        role: source
        handle:  ncdc.noaa.gov
        url: http://www.ncdc.noaa.gov/oa/climate/surfaceinventories.html
      - handle:  infochimps.org/users/flip
        role: converted
        name: Philip (flip) Kromer
        url: http://infochimps.org/blog
    fields:
      - name: Observation Datetime
        desc:  'Time of observation, rounded to nearest whole hour'
        tags: time.point.datetime resolution.hour
        type: datetime
        handle:  observation_datetime
        units: datetime.iso
      - name: Air Temperature
        desc:  The temperature of the air
        tags: environment.air.temperature
        type: fixed:5.1
        handle:  air_temperature
        units: temperature:celsius
      - name: Dew Point Temperature
        desc:  The temperature to which a given parcel of air must be cooled at constant pressure and water vapor content in order for saturation to occur.
        tags: environment.air.dewpoint_temperature
        type: fixed:5.1
        handle:  dew_point_temperature
        units: temperature:celsius
      - name: Sea Level Pressure
        desc:  The air pressure relative to Mean Sea Level (MSL).
        tags: environment.air.pressure
        type: fixed:5.1
        handle:  sea_level_pressure
        units: pressure:hecto.pascals
      - name: Wind Direction
        desc:  |-
          The angle, measured in a clockwise direction, between true north and the direction from which the wind is blowing.
          *NOTE:  Wind direction for calm winds is coded as 0.
        tags: environment.air.wind_direction
        type: fixed:5.1
        handle:  wind_direction
        units: angle:degrees
      - name: Wind Speed Rate
        desc:  The rate of horizontal travel of air past a fixed point.
        tags: environment.air.wind_speed
        type: fixed:5.1
        handle:  wind_speed_rate
        units: speed:meters.per_second
      - name: Sky Condition Total Coverage Code
        desc:  |-
          The code that denotes the fraction of the total celestial dome covered by clouds or other obscuring phenomena.
          DOMAIN:
           0: None, SKC or CLR
           1: One okta - 1/10 or less but not zero
           2: Two oktas - 2/10 - 3/10, or FEW
           3: Three oktas - 4/10
           4: Four oktas - 5/10, or SCT
           5: Five oktas - 6/10
           6: Six oktas - 7/10 - 8/10
           7: Seven oktas - 9/10 or more but not 10/10, or BKN
           8: Eight oktas - 10/10, or OVC
           9: Sky obscured, or cloud amount cannot be estimated
          10: Partial obscuration
          11: Thin scattered
          12: Scattered
          13: Dark scattered
          14: Thin broken
          15: Broken
          16: Dark broken
          17: Thin overcast
          18: Overcast
          19: Dark overcast
        tags: environment.weather.sky_coverage.code
        type: integer:6
        handle:  sky_condition_total_coverage_code
      - name: 'Liquid Precipitation Depth Dimension - One Hour Duration'
        desc:  |-
          The depth of liquid precipitation that is measured over a one hour accumulation period.
          h3. NOTE:  Trace precipitation is coded as -1
        tags: environment.weather.liquid_precipitation_last_hour
        type: fixed:5.1
        handle:  liquid_precipitation_depth_dimension_one_hour_duration
        units: length:millimeters
      - name: 'Liquid Precipitation Depth Dimension - Six Hour Duration'
        desc:  |-
          The depth of liquid precipitation that is measured over a six hour accumulation period.
          h3. NOTE:  Trace precipitation is coded as -1
        tags: ~
        type: fixed:5.1
        handle:  liquid_precipitation_depth_dimension_six_hour_duration
        units: length:millimeters
        

    notes:
      coverage: global
      rights: |-
        h2.WMO Resolution 40 h3.NOAA Policy
        
        The following data and products may have conditions placed on their
        international commercial use. They can be used within the U.S. or for
        non-commercial international activities without restriction. The
        non-U.S. data cannot be redistributed for commercial
        purposes. Re-distribution of these data by others must provide this same
        notification. A log of IP addresses accessing these data and products
        will be maintained and may be made available to data providers.

        For details, please consult "WMO Resolution 40 on the Exchange of
        Meterological Data":http://www.wmo.int/pages/about/Resolution40.html.

      see_also: "- Global Hourly Weather ISD Station Information"
      
      redistribution_restrictions:
        h1. Important Note on Redistribution:

        The non-U.S. data (ie, data for other countries) within this database are
        subject to WMO Resolution 40 restrictions, and cannot be redistributed
        within or outside of the U.S. for any commercial activities.  See "Rights &
        Restrictions":#rights on this page.
          
      collection_desc: |-

        h2. Global Hourly Weather: Integrated Surface Database

        The Integrated Surface Database (ISD) consists of global hourly and synoptic
        observations compiled from numerous sources, into a single common ASCII
        format and common data model. ISD was developed as a joint activity within
        Asheville's Federal Climate Complex. The National Climatic Data Center, with
        U.S. Air Force and Navy partners, began the effort in 1998 with the
        assistance of external funding from several sources. ISD integrates data
        from over 100 original data sources, including numerous data formats which
        were key-entered from paper forms during the 1950's-1970's timeframe. ISD
        Version 1 was released in 2001, with Version 2 (additional quality control
        applied) in 2003. Since 2003, there have been continued incremental
        improvements in automated quality control software.

        The database comprises over 20,000 stations worldwide, with some having data
        as far back as 1901, though the data show a substantial increase in volume
        in the 1940's and again in the early 1970's. Currently there are over 11,000
        stations "active" and updated daily in the database. Total data volume
        (uncompressed) is over 350 gigabytes. ISD includes numerous parameters such
        as wind speed and direction, wind gust, temperature, dew point, cloud data,
        sea level pressure, altimeter setting, station pressure, present weather,
        visibility, precipitation amounts for various time periods, snow depth, and
        various other elements as observed by each station.


        h3. The ISD-Lite subset

        The ISD-Lite data contain a fixed-width formatted subset of the complete
        Integrated Surface Data (ISD) for a select number of observational elements.
        The data are typically stored in a single file corresponding to the ISD
        data, i.e. one file per station per year.  For more information on the
        ISD-Lite format, consult the "ISD-Lite Technical
        Document":ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/isd-lite-technical-document.txt


        h3. Field descriptions

        The datafield descriptions come from the NCDC's "Integrated Surface Data-Lite Format Documentation":ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/isd-lite-format.txt
      usage: |-
        h2. About the Global Hourly Weather: Integrated Surface Database dataset

        (from the "ISD-Lite Technical
        Document":ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/isd-lite-technical-document.txt

        h3. Introduction

        The ISD-Lite product is designed to be an easier to work with subset of the
        larger Integrated Surface Data hourly dataset.  ISD-Lite contains eight
        common hourly time-series climatological variables represented in a
        fixed-width format.  The elements extracted are:

        # Air temperature (degrees Celsius * 10)
        # Dew point temperature (degrees Celsius * 10)
        # Sea level pressure (hectopascals)
        # Wind direction (angular degrees)
        # Wind speed (meters per second * 10) 
        # Total cloud cover (coded, see format documentation) 
        # One-hour accumulated liquid precipitation (millimeters) 
        # Six-hour accumulated liquid precipitation (millimeters)

        The ISD-Lite data are represented with a modified time stamp which
        corresponds to the nearest hour of actual observation.  Sub-hourly
        observations were removed.  Duplicate observations were resolved according
        to a ranking system.


        h3. Observation Selection

        For the purposes of ISD-Lite, a capture window was defined to extract
        observational elements closest to the top of the hour.  This capture window
        was defined as observations with observation minute greater than or equal to
        ten minutes before the top of the hour up to and including the top of the
        hour.  ISD elements with an observation minute outside of the capture window
        were ignored.


        In most cases, the choice of the hourly value was made on an
        element-by-element basis.  This means that it is possible to have an
        observation for air temperature paired with an observation for dew point
        temperature where the observation time differs by as much as ten minutes.
        The exception to this was wind speed and wind direction, which are chosen as
        a pair and are always extracted from the same observation.

        h3. Duplicate Removal


        In many cases, there can be more than one element at the same observation
        time or within the capture window.  In these situations, a single element is
        chosen from the list of duplicates according to a ranking schema.  The
        ranking schema assigns a score to each element according to these criteria:

        # Difference between actual observation time (minute) and the top of the
        # hour.  Observations that are closer to the top of the hour are favored.

        # Ranking of the data quality flag from the ISD observation.  Generally,
        # observations with a higher data quality flag (i.e. passed all quality
        # control checks) are favored over observations with a lower data quality
        # flag (i.e. suspect).

        # Ranking of the data source flag from the ISD observation.  Observations
        # that come from a merger of datasets are favored over elements that have
        # not been merged (i.e. datsav3/td3280/td3240 merged elements are favored
        # over just datsav3).

        # Ranking of the report type code from the ISD observation.  Observations
        # that are formed from merger of multiple report types are favored over
        # observations that are not (i.e. merger of Synoptic/Metar is favored over
        # just Synoptic).

        A score is assigned to each of these tests, favoring the order listed above.
        The observation with this highest score is chosen to be the hourly value.
        In case of a tie score, the element that was encountered first in the
        original ISD file is chosen.

        For the case of wind speed/wind direction, the duplicate elimination is
        based on the combined score for both wind speed and wind direction.


        h3. Data Format Information

        Care was taken to adjust the data values as little as possible from the
        original ISD format.  The units and domain for each value largely match the
        original ISD format.  There are a few exceptions:

        # Missing value.  In all cases, the missing value was reset to a common
        # value.  For all elements the missing value is coded as -9999

        # Calm winds.  In the case of calm winds, wind direction and wind speed are
        # indicated by a zero 0.  This is not to be confused with northerly wind
        # direction, which is coded as 360.

        # Trace precipitation.  In the case of 1-hour and 6-hour duration
        # precipitation accumulations, trace precipitation is coded as a -1 value.

        h3. Suitability of Data   

        The ISD-Lite product is not intended to replace the full ISD for situations
        where exact observation time and/or observation density is critical.
        Instead, it is designed to be a smaller and easier to work with format which
        may be suitable for investigating trends, larger scale patterns or rough
        climatological averages.

        It should also be noted that inter-comparison of fields across an
        observational hour may not be recommended due to the possibility of slight
        differences in actual observation time between elements.  This feature is
        most pronounced between the mandatory elements and the additional elements,
        for example air temperature versus precipitation. In many cases the
        observation time for these elements can differ by as much as ten minutes.


        No additional quality control was performed on the ISD-Lite product and the
        absence of flag data may make it difficult to determine good data from
        suspicious data.

        Data were obtained from NOAA's "NCDC FTP site":ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

    ratings:
      accurate:       { rating: 2, by: 'initial', desc: 'These were automatically processed from the NCDC data, but have not been verified against the original dataset.' }
      authoritative:  { rating: 5, by: 'initial', desc: '' }
      comprehensive:  { rating: 5, by: 'initial', desc: '' }
      interesting:    { rating: 5, by: 'initial', desc: '' }
