
# Dataset Card for DWD ICON Global Forecast

This dataset is comprised of forecasts from the German Weather Service's (DWD) ICON-EU model from March 2023 to the present with all variables included. Each forecast runs up to 4 days into the future, and the model is ran 4 times per day. This data is an archive of 
the publicly available data at https://opendata.dwd.de/weather/nwp/, converted to Zarr format with Xarray. No other processing of the data is performed.

## Dataset Details

- **Curated by:** Jacob Bieker, Open Climate Fix
- **License:** German Government Open Data License

### Dataset Sources [optional]


- **Raw files:** https://opendata.dwd.de/weather/nwp/

Note: The raw files are deleted after 24 hours, and there is no long-term archive available publicly.

## Uses

This data is intended for use in renewable energy forecasting, weather forecasting, and anything that can use high-quality weather forecasts over Europe.


## Dataset Structure

The dataset is comprised of one Zarr file per forecast initialization time, and each forecast goes out between 48-96 hours. The files are located at data/year/month/day/YYYYMMDDHH.zarr.zip.

## Dataset Creation

### Curation Rationale

The DWD ICON Global model provides high-quality, high-resolution forecasts for Global weather that is also publicly available and free of charge. The model should generally outperform
NOAA's GFS forecast model, and has a higher temporal and spatial resolution. The main downside of this model is that the files are only available for a short period publicly, so this dataset
was setup to provide a public archive of the forecasts for use by researchers in many fields, but especially renewable energy forecasting and weather forecasting.

### Source Data

The source data is the grib2 files from the DWD Open Data Server.

#### Data Collection and Processing

The data is collected every day, around 6-8 hours after forecast initialization time to ensure the forecast is finished running before the data is pulled. The grib2 files are opened
with Xarray and collated into a single Xarray Dataset, with one data variable per ICON variable. Surface variables have "_s" appended to their names to differentiate them from multi-level variables.
The Dataset is then written to Zarr using "ocf_blosc2" to encode and compress the variables. No scaling or changing of the variables values is performed. This does mean that the data is not in a regular lat/lon grid
and is instead in the icosohedral grid of the model. To obtain a regular grid, the data will need to be regridded.

#### Who are the source data producers?

German Weather Service (DWD)


### Recommendations

These files can be opened directly from HuggingFace, and streamed in with Xarray. HuggingFace is fairly slow though, so the recommended way would be to download the files you want
and open them locally. In either case, to access the data you can do the following

```python
import ocf_blosc2
import xarray as xr
data = xr.open_zarr("path/to/zarr/file")
print(data)
```

Alternatively, for using the data in forecasting, there is the `ocf_datapipes` package for loading and training renewable energy forecasting models with multi-modal inputs, including
ICON, but also satellite data, PV readings, etc.


## Dataset Card Contact

Jacob Bieker: jacob@openclimatefix.org
size_categories:1K<n<10K license:mit climate doi:10.57967/hf/0880 region:us
{"license": "mit", "size_categories": ["1K<n<10K"], "pretty_name": "DWD ICON Global Forecasts", "tags": ["climate"]}
TAGS
#size_categories-1K<n<10K #license-mit #climate #doi-10.57967/hf/0880 #region-us 


# Dataset Card for DWD ICON Global Forecast

This dataset is comprised of forecasts from the German Weather Service's (DWD) ICON-EU model from March 2023 to the present with all variables included. Each forecast runs up to 4 days into the future, and the model is ran 4 times per day. This data is an archive of 
the publicly available data at URL converted to Zarr format with Xarray. No other processing of the data is performed.

## Dataset Details

- Curated by: Jacob Bieker, Open Climate Fix
- License: German Government Open Data License

### Dataset Sources [optional]


- Raw files: URL

Note: The raw files are deleted after 24 hours, and there is no long-term archive available publicly.

## Uses

This data is intended for use in renewable energy forecasting, weather forecasting, and anything that can use high-quality weather forecasts over Europe.


## Dataset Structure

The dataset is comprised of one Zarr file per forecast initialization time, and each forecast goes out between 48-96 hours. The files are located at data/year/month/day/URL.

## Dataset Creation

### Curation Rationale

The DWD ICON Global model provides high-quality, high-resolution forecasts for Global weather that is also publicly available and free of charge. The model should generally outperform
NOAA's GFS forecast model, and has a higher temporal and spatial resolution. The main downside of this model is that the files are only available for a short period publicly, so this dataset
was setup to provide a public archive of the forecasts for use by researchers in many fields, but especially renewable energy forecasting and weather forecasting.

### Source Data

The source data is the grib2 files from the DWD Open Data Server.

#### Data Collection and Processing

The data is collected every day, around 6-8 hours after forecast initialization time to ensure the forecast is finished running before the data is pulled. The grib2 files are opened
with Xarray and collated into a single Xarray Dataset, with one data variable per ICON variable. Surface variables have "_s" appended to their names to differentiate them from multi-level variables.
The Dataset is then written to Zarr using "ocf_blosc2" to encode and compress the variables. No scaling or changing of the variables values is performed. This does mean that the data is not in a regular lat/lon grid
and is instead in the icosohedral grid of the model. To obtain a regular grid, the data will need to be regridded.

#### Who are the source data producers?

German Weather Service (DWD)


### Recommendations

These files can be opened directly from HuggingFace, and streamed in with Xarray. HuggingFace is fairly slow though, so the recommended way would be to download the files you want
and open them locally. In either case, to access the data you can do the following



Alternatively, for using the data in forecasting, there is the 'ocf_datapipes' package for loading and training renewable energy forecasting models with multi-modal inputs, including
ICON, but also satellite data, PV readings, etc.


## Dataset Card Contact

Jacob Bieker: jacob@URL
# Dataset Card for DWD ICON Global Forecast

This dataset is comprised of forecasts from the German Weather Service's (DWD) ICON-EU model from March 2023 to the present with all variables included. Each forecast runs up to 4 days into the future, and the model is ran 4 times per day. This data is an archive of 
the publicly available data at URL converted to Zarr format with Xarray. No other processing of the data is performed. ## Dataset Details

- Curated by: Jacob Bieker, Open Climate Fix
- License: German Government Open Data License ### Dataset Sources [optional]


- Raw files: URL

Note: The raw files are deleted after 24 hours, and there is no long-term archive available publicly. ## Uses

This data is intended for use in renewable energy forecasting, weather forecasting, and anything that can use high-quality weather forecasts over Europe. ## Dataset Structure

The dataset is comprised of one Zarr file per forecast initialization time, and each forecast goes out between 48-96 hours. The files are located at data/year/month/day/URL. ## Dataset Creation ### Curation Rationale

The DWD ICON Global model provides high-quality, high-resolution forecasts for Global weather that is also publicly available and free of charge. The model should generally outperform
NOAA's GFS forecast model, and has a higher temporal and spatial resolution. The main downside of this model is that the files are only available for a short period publicly, so this dataset
was setup to provide a public archive of the forecasts for use by researchers in many fields, but especially renewable energy forecasting and weather forecasting. ### Source Data

The source data is the grib2 files from the DWD Open Data Server. #### Data Collection and Processing

The data is collected every day, around 6-8 hours after forecast initialization time to ensure the forecast is finished running before the data is pulled. The grib2 files are opened
with Xarray and collated into a single Xarray Dataset, with one data variable per ICON variable. Surface variables have "_s" appended to their names to differentiate them from multi-level variables.
The Dataset is then written to Zarr using "ocf_blosc2" to encode and compress the variables. No scaling or changing of the variables values is performed. This does mean that the data is not in a regular lat/lon grid
and is instead in the icosohedral grid of the model. To obtain a regular grid, the data will need to be regridded. #### Who are the source data producers?

German Weather Service (DWD) ### Recommendations

These files can be opened directly from HuggingFace, and streamed in with Xarray. HuggingFace is fairly slow though, so the recommended way would be to download the files you want
and open them locally. In either case, to access the data you can do the following



Alternatively, for using the data in forecasting, there is the 'ocf_datapipes' package for loading and training renewable energy forecasting models with multi-modal inputs, including
ICON, but also satellite data, PV readings, etc. ## Dataset Card Contact

Jacob Bieker: jacob@URL
TAGS
#size_categories-1K<n<10K #license-mit #climate #doi-10.57967/hf/0880 #region-us 
 # Dataset Card for DWD ICON Global Forecast

This dataset is comprised of forecasts from the German Weather Service's (DWD) ICON-EU model from March 2023 to the present with all variables included. Each forecast runs up to 4 days into the future, and the model is ran 4 times per day. This data is an archive of 
the publicly available data at URL converted to Zarr format with Xarray. No other processing of the data is performed. ## Dataset Details

- Curated by: Jacob Bieker, Open Climate Fix
- License: German Government Open Data License ### Dataset Sources [optional]


- Raw files: URL

Note: The raw files are deleted after 24 hours, and there is no long-term archive available publicly. ## Uses

This data is intended for use in renewable energy forecasting, weather forecasting, and anything that can use high-quality weather forecasts over Europe. ## Dataset Structure

The dataset is comprised of one Zarr file per forecast initialization time, and each forecast goes out between 48-96 hours. The files are located at data/year/month/day/URL. ## Dataset Creation ### Curation Rationale

The DWD ICON Global model provides high-quality, high-resolution forecasts for Global weather that is also publicly available and free of charge. The model should generally outperform
NOAA's GFS forecast model, and has a higher temporal and spatial resolution. The main downside of this model is that the files are only available for a short period publicly, so this dataset
was setup to provide a public archive of the forecasts for use by researchers in many fields, but especially renewable energy forecasting and weather forecasting. ### Source Data

The source data is the grib2 files from the DWD Open Data Server. #### Data Collection and Processing

The data is collected every day, around 6-8 hours after forecast initialization time to ensure the forecast is finished running before the data is pulled. The grib2 files are opened
with Xarray and collated into a single Xarray Dataset, with one data variable per ICON variable. Surface variables have "_s" appended to their names to differentiate them from multi-level variables.
The Dataset is then written to Zarr using "ocf_blosc2" to encode and compress the variables. No scaling or changing of the variables values is performed. This does mean that the data is not in a regular lat/lon grid
and is instead in the icosohedral grid of the model. To obtain a regular grid, the data will need to be regridded. #### Who are the source data producers?

German Weather Service (DWD) ### Recommendations

These files can be opened directly from HuggingFace, and streamed in with Xarray. HuggingFace is fairly slow though, so the recommended way would be to download the files you want
and open them locally. In either case, to access the data you can do the following



Alternatively, for using the data in forecasting, there is the 'ocf_datapipes' package for loading and training renewable energy forecasting models with multi-modal inputs, including
ICON, but also satellite data, PV readings, etc. ## Dataset Card Contact

Jacob Bieker: jacob@URL