Services#

The core of wetterdienst is to provide data but as we don’t collect the data ourselves we rely on consuming data of already existing services - mostly governmental services. To simplify the implementation of weather services we created enumerations and classes that should be used in order to adapt whatever API is offered by a service to the general scheme of wetterdienst with some handful of attributes to define each API and streamline internal workflows. The following paragraphs describe how we can/may/should implement a new weather service in wetterdienst as far as our own experience goes. We’ll give examples based on the DwdObservationRequest implementation.

Step 1: Enumerations#

The basis for the implementation of a new service are enumerations. A weather service requires 5 enumerations: - parameter enumeration - unit enumeration - dataset enumeration - resolution enumeration - period enumeration

Parameter enumeration#

The parameter enumeration could look like this:

from wetterdienst import Resolution
from wetterdienst.util.parameter import DatasetTreeCore

class DwdObservationParameter(DatasetTreeCore):
    # the string "MINUTE_1" has the match the name of a resolution, here Resolution.MINUTE_1
    class MINUTE_1(DatasetTreeCore):
        # precipitation
        class PRECIPITATION(Enum):
            QUALITY = "qn"
            PRECIPITATION_HEIGHT = "rs_01"
            PRECIPITATION_HEIGHT_DROPLET = "rth_01"
            PRECIPITATION_HEIGHT_ROCKER = "rwh_01"
            PRECIPITATION_INDEX = "rs_ind_01"

        PRECIPITATION_HEIGHT = PRECIPITATION.PRECIPITATION_HEIGHT
        PRECIPITATION_HEIGHT_DROPLET = PRECIPITATION.PRECIPITATION_HEIGHT_DROPLET
        PRECIPITATION_HEIGHT_ROCKER = PRECIPITATION.PRECIPITATION_HEIGHT_ROCKER
        PRECIPITATION_INDEX = PRECIPITATION.PRECIPITATION_INDEX

        class ANOTHER_DATASET(Enum):
            QUALITY = "qn"
            # this parameter can't be accessed via MINUTE_1.PRECIPITATION_HEIGHT
            # but has to be queried with something like
            # parameter=(MINUTE_1.ANOTHER_DATASET.PRECIPITATION_HEIGHT, MINUTE_1.ANOTHER_DATASET)
            PRECIPITATION_HEIGHT = "rs_01"

Hint

Here MINUTE_1 represents the resolution of the data and it has to match one of the resolution names of the core resolution (here it matches Resolution.MINUTE_1). It has to match as we access the possible parameters e.g. via the requested resolution.

As the DWD observations are offered in datasets, DwdObservationParameter has two layers of parameters: - flat layer of all available parameters for a given resolution with favorites for parameters if two of the same name

exist in different datasets

  • deep layer of a dataset and its own parameters

Here we have a dataset PRECIPITATION in MINUTE_1 resolution, which has four parameters and one quality column. Those parameters are flattened out by adding links to them on the resolution level. This way we can now access parameters as follows:

# PRECIPITATION_HEIGHT of PRECIPITATION dataset
DwdObservationRequest(
    parameter=DwdObservationParameter.MINUTE_1.PRECIPITATION_HEIGHT
)

# same as above
DwdObservationRequest(
    parameter=DwdObservationParameter.MINUTE_1.PRECIPITATION.PRECIPITATION_HEIGHT
)

# PRECIPITATION_HEIGHT of the exact PRECIPITATION dataset, assuming that there would be another dataset with the
# same parameter
DwdObservationRequest(
    parameter=(DwdObservationParameter.MINUTE_1.PRECIPITATION.PRECIPITATION_HEIGHT, DwdObservationParameter.MINUTE_1.PRECIPITATION)
)

Hint

The values of the enumerations should represent the original name of the parameter to create renaming mappings.

Unit enumeration#

The unit enumeration has to match the parameter enumeration except that it should only have deep levels. For the above example it should look like:

from wetterdienst.util.parameter import DatasetTreeCore
from wetterdienst.metadata.unit import OriginUnit, SIUnit, UnitEnum

class DwdObservationUnit(DatasetTreeCore):
    # the string "MINUTE_1" has the match the name of a resolution, here Resolution.MINUTE_1
    class MINUTE_1(DatasetTreeCore):
        # precipitation
        class PRECIPITATION(UnitEnum):
            QUALITY = OriginUnit.DIMENSIONLESS.value, SIUnit.DIMENSIONLESS.value
            PRECIPITATION_HEIGHT = (
                OriginUnit.MILLIMETER.value,
                SIUnit.KILOGRAM_PER_SQUARE_METER.value,
            )
            PRECIPITATION_HEIGHT_DROPLET = (
                OriginUnit.MILLIMETER.value,
                SIUnit.KILOGRAM_PER_SQUARE_METER.value,
            )
            PRECIPITATION_HEIGHT_ROCKER = (
                OriginUnit.MILLIMETER.value,
                SIUnit.KILOGRAM_PER_SQUARE_METER.value,
            )
            PRECIPITATION_INDEX = (
                OriginUnit.DIMENSIONLESS.value,
                SIUnit.DIMENSIONLESS.value,
            )

Each parameter is represented by a tuple with the original unit and the SI unit. General conversations are easily possible with the pint unit system and for other more complex conversions we may have to define special mappings.

Other enumerations#

The remaining enumerations are simple enumerations. The only thing that has to be considered here is that all the names are matching the ones from the parameter enumeration, the resolution enumeration and the period enumeration:

from enum import Enum
from wetterdienst import Resolution, Period

class DwdObservationDataset(Enum):
    # 1_minute
    PRECIPITATION = "precipitation"

class DwdObservationResolution(Enum):
    # 1_minute
    MINUTE_1 = Resolution.MINUTE_1.value

class DwdObservationPeriod(Enum):
    # 1_minute
    HISTORICAL = Period.HISTORICAL.value

Step 2: Request class#

The request class represents a request and carries all the required attributes as well as the values class that is responsible for acquiring the data later on. The implementation is based on ScalarRequestCore from wetterdienst.core.

Attributes:

@property
@abstractmethod
def provider(self) -> Provider:
    """Optional enumeration for multiple resolutions"""
    pass

@property
@abstractmethod
def kind(self) -> Kind:
    """Optional enumeration for multiple resolutions"""
    pass

@property
@abstractmethod
def _resolution_base(self) -> Optional[Resolution]:
    """Optional enumeration for multiple resolutions"""
    pass

@property
@abstractmethod
def _resolution_type(self) -> ResolutionType:
    """Resolution type, multi, fixed, ..."""
    pass

@property
@abstractmethod
def _period_type(self) -> PeriodType:
    """Period type, fixed, multi, ..."""
    pass

@property
@abstractmethod
def _period_base(self) -> Optional[Period]:
    """Period base enumeration from which a period string can be parsed"""
    pass

@property
@abstractmethod
def _parameter_base(self) -> Enum:
    """parameter base enumeration from which parameters can be parsed e.g.
    DWDObservationParameter"""
    pass

@property
@abstractmethod
def _data_range(self) -> DataRange:
    """State whether data from this provider is given in fixed data chunks
    or has to be defined over start and end date"""
    pass

@property
@abstractmethod
def _has_datasets(self) -> bool:
    """Boolean if weather service has datasets (when multiple parameters are stored
    in one table/file)"""
    pass

@property
def _unique_dataset(self) -> bool:
    """If ALL parameters are stored in one dataset e.g. all daily data is stored in
    one file"""
    if self._has_datasets:
        raise NotImplementedError("define if only one big dataset is available")
    return False

@property
@abstractmethod
def _has_tidy_data(self) -> bool:
    """If data is generally provided tidy -> then data should not be tidied but
    rather tabulated if data is requested to not being tidy"""
    pass

@property
@abstractmethod
def _unit_tree(self):
    pass

@property
@abstractmethod
def _values(self):
    """Class to get the values for a request"""
    pass

ScalarRequestCore has one abstract method that has to be implemented: the _all which manages to get a listing of stations for the requested datasets/parameters. The listing includes: - station_id - from_date - to_date - height - name - state - latitude - longitude

The names can be mapped using the Columns enumeration.

Step 3: Values class#

The values class is based on ScalarValuesCore and manages the acquisition of actual data. The class is also part of the ScalarRequestCore being accessed via the _values property. It has to implement the _collect_station_parameter method that takes care of getting values of a parameter/dataset for a station id.