aixweather.transformation_functions package
AixWeather Transformation Functions
The ‘transformation_functions’ module contains a collection of static functions used for transforming weather data. These functions are designed to be imported and utilized by other modules within the AixWeather package.
Submodules
aixweather.transformation_functions.auxiliary module
includes auxiliary functions for data handling and transformation
- aixweather.transformation_functions.auxiliary.evaluate_transformations(core_format: dict, other_format: dict)[source]
Compare the units and core variables of two formats and print any required unit transformations.
- Args:
- core_format (dict): A dictionary representing the core format with keys as variable names and values
containing unit information.
- other_format (dict): A dictionary representing another format to be compared with the core format.
It contains keys and values with ‘core_name’ and ‘unit’ attributes.
- Raises:
ValueError: If a core variable in other_format doesn’t match the core variable format.
- aixweather.transformation_functions.auxiliary.fill_nan_from_format_dict(df: DataFrame, format_data: dict) DataFrame[source]
Fill NaN values in a DataFrame based on the provided format data.
- Args:
df (pd.DataFrame): The DataFrame containing the data to be processed. format_data (dict): A dictionary specifying NaN replacement values for columns,
with column names as keys and NaN replacement values as values.
- Returns:
pd.DataFrame: A DataFrame with NaN values filled as per the format data.
- aixweather.transformation_functions.auxiliary.force_data_variable_convention(df: DataFrame, format_desired: dict) DataFrame[source]
Ensure that all and only desired variable names are present and in correct order.
- Args:
df (pd.DataFrame): The DataFrame containing the data to be formatted. format_desired (dict): A dictionary specifying the desired format with variable names as keys.
- Returns:
pd.DataFrame: A DataFrame with filtered data in the desired format and order.
- aixweather.transformation_functions.auxiliary.rename_columns(df: DataFrame, format_dict: dict) DataFrame[source]
Rename DataFrame columns based on the provided format dictionary.
- Args:
df (pd.DataFrame): The DataFrame whose columns need to be renamed. format_dict (dict): A dictionary specifying the column renaming mapping,
with current column names as keys and desired names as values.
- Returns:
pd.DataFrame: A DataFrame with renamed columns.
- aixweather.transformation_functions.auxiliary.replace_dummy_with_nan(df: DataFrame, format_dict: dict) DataFrame[source]
Replace specific values in the DataFrame with NaN based on the given format dictionary. Reason: sometimes, e.g. the DWD, specifies a missing value with a dummy value like e.g. 99, which makes it hard to see where missing values are and might affect the simulation.
- Args:
df (pd.DataFrame): The DataFrame to be processed. format_dict (dict): A dictionary specifying values to be replaced with NaN,
with column names as keys and corresponding dummy values as values.
- Returns:
pd.DataFrame: A DataFrame with specified values replaced by NaN.
- aixweather.transformation_functions.auxiliary.select_entry_by_core_name(format_dict: dict, core_name_to_match: str)[source]
Select an entry from a format dictionary based on the specified core name.
- Args:
format_dict (dict): A dictionary to search for the entry. core_name_to_match (str): The core name to match in the dictionary values.
- Returns:
dict: The dictionary entry matching the specified core name, or None if not found.
aixweather.transformation_functions.pass_through_handling module
This module contains auxiliary functions for data transformation, e.g. time shifts
- aixweather.transformation_functions.pass_through_handling.create_pass_through_variables(df_shifted: DataFrame, df_no_shift: DataFrame, format: dict, transform_func, meta: MetaData)[source]
This function adds unshifted data from the df_no_shift DataFrame to the df_shifted DataFrame to prevent unnecessary interpolation of values. It takes into account the required shifting and performs transformations. The appropriate pass-through variables (unshifted variables) are added to the dataframe with a suffix specifying the shifting that would be required for them. Calculated (transformed) variables are only added if the variables they were calculated from all have the same shifting (time of measurement).
- Args:
df_shifted (pd.DataFrame): The DataFrame with shifted data. df_no_shift (pd.DataFrame): The DataFrame with unshifted data. format (dict): A dictionary specifying the format (required shifting) of the data. transform_func: The transformation function from the import2core data process. meta (MetaData): Metadata associated with the data.
- Returns:
pd.DataFrame: The modified df_shifted DataFrame with added pass-through variables.
- aixweather.transformation_functions.pass_through_handling.pass_through_measurements_with_back_and_forth_interpolating(core2output_df: DataFrame, format_outputter: dict) DataFrame[source]
Insert pass-through measurements to the output dataframe to avoid back-and-forth or double shifting interpolation.
It deletes the double interpolated variables and inserts the pass-through ones where applicable.
- Args:
core2output_df (pd.DataFrame): DataFrame containing core data in the process of core2outputfile. format_outputter (dict): Dictionary specifying the format of output data.
- Returns:
pd.DataFrame: The modified core2output_df DataFrame with pass-through variables.
aixweather.transformation_functions.time_observation_transformations module
Includes functions to execute time shift operations. It also includes a function to truncate data in given interval.
- aixweather.transformation_functions.time_observation_transformations.avg_following_hour_2_indicated_time(df)[source]
aka: foll2ind
- aixweather.transformation_functions.time_observation_transformations.avg_preceding_hour_2_indicated_time(df)[source]
aka: prec2ind
- aixweather.transformation_functions.time_observation_transformations.indicated_time_2_avg_following_hour(df)[source]
aka: ind2foll
- aixweather.transformation_functions.time_observation_transformations.indicated_time_2_avg_preceding_hour(df)[source]
aka: ind2prec
- aixweather.transformation_functions.time_observation_transformations.shift_time_by_dict(format_dict: dict, df: DataFrame) DataFrame[source]
Shift timestamps in a DataFrame based on a format dictionary.
This function shifts and interpolates values in the DataFrame df based on the specified format dictionary. The format dictionary should contain information about the desired time shifting for core data variables.
- Args:
format_dict (dict): A dictionary specifying the time shifting for core data variables. df (pd.DataFrame): The DataFrame containing timestamped data with core data variable names.
- Returns:
pd.DataFrame: The modified DataFrame with values shifted and interpolated according to the format dictionary.
- aixweather.transformation_functions.time_observation_transformations.truncate_data_from_start_to_stop(df: ~pandas.core.frame.DataFrame, start: <module 'datetime' from '/usr/local/lib/python3.11/datetime.py'>, stop: <module 'datetime' from '/usr/local/lib/python3.11/datetime.py'>) DataFrame[source]
Truncate a DataFrame to include data only between specified start and stop timestamps.
- Args:
df (pd.DataFrame): The DataFrame containing timestamped data. start (datetime): The start timestamp to include in the truncated DataFrame. stop (datetime): The stop timestamp to include in the truncated DataFrame.
- Returns:
pd.DataFrame: A new DataFrame containing data only within the specified time range.
aixweather.transformation_functions.unit_conversions module
Includes functions to convert units in weather data.
- aixweather.transformation_functions.unit_conversions.Jcm2_to_Whm2(radiation: Series)[source]
convert radiance unit from J/cm^2 to Wh/m^2
- aixweather.transformation_functions.unit_conversions.Jm2_to_Whm2(radiation: Series)[source]
convert radiance unit from J/m^2 to Wh/m^2
- aixweather.transformation_functions.unit_conversions.divide_by_1000(series: Series)[source]
divide by 1000
- aixweather.transformation_functions.unit_conversions.eigth_to_tenth(cloudgrade: Series)[source]
convert cloudgrade from eighth to tenth
- aixweather.transformation_functions.unit_conversions.hPa_to_Pa(pressure: Series)[source]
convert pressure unit from hPa to Pa
- aixweather.transformation_functions.unit_conversions.kJm2_to_Whm2(radiation: Series)[source]
convert radiance unit from kJ/m^2 to Wh/m^2
aixweather.transformation_functions.variable_transformations module
This module includes variable transformations and functions to calculate variables from given values
- aixweather.transformation_functions.variable_transformations.approximate_opaque_from_total_skycover(total_sky_cover)[source]
- aixweather.transformation_functions.variable_transformations.calculate_dew_point_temp(dry_bulb_temp: Series, rel_hum: Series) Series[source]
Calculate the dew point temperature using a simplified formula.
This function estimates the dew point temperature in degrees Celsius based on the given dry bulb temperature and relative humidity. Please note that this formula is fairly accurate for humidity above 50%. For more precise calculations, consider using a more advanced formula.
- Args:
dry_bulb_temp (pd.Series): The dry bulb temperature in degrees Celsius. rel_hum (pd.Series): The relative humidity in percentage.
- Returns:
pd.Series: The estimated dew point temperature in degrees Celsius.
- aixweather.transformation_functions.variable_transformations.calculate_direct_horizontal_radiation(glob_hor_rad: Series, diff_hor_rad: Series) Series[source]
Calculate direct horizontal radiation by subtracting diffuse horizontal radiation from global horizontal radiation.
Checked by Martin Rätz on 08.08.2023.
- Args:
glob_hor_rad (pd.Series): Global horizontal radiation in Wh/m^2. diff_hor_rad (pd.Series): Diffuse horizontal radiation in Wh/m^2.
- Returns:
pd.Series: The estimated direct horizontal radiation in Wh/m^2.
- aixweather.transformation_functions.variable_transformations.calculate_global_horizontal_radiation(dir_hor_rad: Series, diff_hor_rad: Series) Series[source]
Calculate global horizontal radiation on a horizontal plane by adding direct horizontal radiation with diffuse horizontal radiation.
Checked by Martin Rätz on 08.08.2023 using: https://pvpmc.sandia.gov/modeling-steps/1-weather-design-inputs/irradiance-insolation/global-horizontal-irradiance/
- Args:
dir_hor_rad (pd.Series): Direct horizontal radiation in Wh/m^2. diff_hor_rad (pd.Series): Diffuse horizontal radiation in Wh/m^2.
- Returns:
pd.Series: The estimated global horizontal radiation in Wh/m^2.
- aixweather.transformation_functions.variable_transformations.calculate_horizontal_infrared_radiation(dry_bulb_temp: Series, dew_point_temp: Series, opaque_sky_cover: Series) Series[source]
Calculate horizontal infrared radiation using the provided inputs.
Calculation details can be found at: https://www.energyplus.net/sites/default/files/docs/site_v8.3.0/EngineeringReference/05-Climate/index.html
Sky emissivity is considered in the calculation as per the above reference.
- Args:
dry_bulb_temp (pd.Series): Dry bulb temperature in degrees Celsius. dew_point_temp (pd.Series): Dew point temperature in degrees Celsius. opaque_sky_cover (pd.Series): Opaque sky cover in tenths.
- Returns:
pd.Series: The estimated horizontal infrared radiation in Wh/m^2.
- aixweather.transformation_functions.variable_transformations.calculate_normal_from_horizontal_direct_radiation(latitude: float, longitude: float, utc_time: DatetimeIndex, direct_horizontal_radiation: Series)[source]
Calculates the Direct Normal Irradiance (DNI) from Direct Horizontal Irradiance (DHI). Values will be set to zero when the sun is below 5° angle from the horizon.
- Args:
latitude (float): The latitude of the location in degrees. longitude (float): The longitude of the location in degrees. utc_time (pd.DatetimeIndex): The timestamps for which the DNI is to be calculated, in UTC. direct_horizontal_radiation (pd.Series): The Direct Horizontal Irradiance (DHI) in [W/m^2]
- Returns:
pd.Series: The calculated Direct Normal Irradiance (DNI) in W/m^2 for each timestamp.
- aixweather.transformation_functions.variable_transformations.robust_transformation(df: DataFrame, desired_variable: str, transformation_function, *args: str) tuple[pandas.core.frame.DataFrame, dict][source]
Apply a transformation function to calculate the desired variable in the DataFrame if the variable is missing or contains only NaN values. Skip the transformation if any required column (specified in args) is missing to perform the calculation.
- Args:
df (pd.DataFrame): DataFrame containing the data. desired_variable (str): Name of the variable to which the transformation should be applied. transformation_function: Function to apply to the desired variable. *args: Additional arguments required for the transformation function.
These can be column names (str) or other values (all other types).
- Returns:
tuple[pd.DataFrame, dict]: A tuple containing the updated DataFrame with the transformation applied to the desired variable and a dictionary indicating which variables were used for the calculation.
- aixweather.transformation_functions.variable_transformations.variable_transform_all(df: DataFrame, meta: MetaData) tuple[pandas.core.frame.DataFrame, dict][source]
Transform and compute missing variables for the given DataFrame based on specified calculations. This function performs multiple transformations on the DataFrame to calculate all completely missing variables. The applied calculations for each variable are tracked in the ‘calc_overview’ dictionary.
You can add all variable transformations here. Respect a meaningful order.
- Args:
df (pd.DataFrame): The DataFrame containing the data to be transformed. meta (MetaData): Metadata associated with the data.
- Returns:
tuple[pd.DataFrame, dict]: A tuple containing the transformed DataFrame with computed variables and a dictionary (‘calc_overview’) tracking the calculations applied to the DataFrame.