Aggregation/Visualization Utils

The HierarchicalForecast package contains utility functions to wrangle and visualize hierarchical series datasets. The aggregate function of the module allows you to create a hierarchy from categorical variables representing the structure levels, returning also the aggregation contraints matrix $\mathbf{S}$.

In addition, HierarchicalForecast ensures compatibility of its reconciliation methods with other popular machine-learning libraries via its external forecast adapters that transform output base forecasts from external libraries into a compatible data frame format.

Aggregate Function

aggregate

aggregate(df, spec, exog_vars=None, sparse_s=False, id_col='unique_id', time_col='ds', id_time_col=None, target_cols=('y',))

Utils Aggregation Function.

Aggregates bottom level series contained in the DataFrame df according to levels defined in the spec list.

Parameters:

Name	Type	Description	Default
df	Frame	Dataframe with columns [time_col, *target_cols], columns to aggregate and optionally exog_vars.	required
spec	list[list[str]]	list of levels. Each element of the list should contain a list of columns of df to aggregate.	required
exog_vars	Optional[dict[str, Union[str, list[str]]]]	dictionary of string keys & values that can either be a list of strings or a single string keys correspond to column names and the values represent the aggregation(s) that will be applied to each column. Accepted values are those from Pandas or Polars aggregation Functions, check the respective docs for guidance. Default is None.	None
sparse_s	bool	Return S_df as a sparse Pandas dataframe. Default is False.	False
id_col	str	Column that will identify each serie after aggregation. Default is “unique_id”.	‘unique_id’
time_col	str	Column that identifies each timestep, its values can be timestamps or integers. Default is “ds”.	‘ds’
id_time_col	Optional[str]	Column that will identify each timestep after temporal aggregation. If provided, aggregate will operate temporally. Default is None.	None
target_cols	Sequence[str]	list of columns that contains the targets to aggregate. Default is (“y”,).	(‘y’,)

Returns:

Type	Description
tuple[FrameT, FrameT, dict]	tuple[FrameT, FrameT, dict]: Y_df, S_df, tags Y_df: Hierarchically structured series. S_df: Summing dataframe. tags: Aggregation indices.

aggregate_temporal

aggregate_temporal(df, spec, exog_vars=None, sparse_s=False, id_col='unique_id', time_col='ds', id_time_col='temporal_id', target_cols=('y',), aggregation_type='local')

Utils Aggregation Function for Temporal aggregations.

Aggregates bottom level timesteps contained in the DataFrame df according to temporal levels defined in the spec list.

Parameters:

Name	Type	Description	Default
df	Frame	Dataframe with columns [time_col, target_cols] and columns to aggregate.	required
spec	dict[str, int]	Dictionary of temporal levels. Each key should be a string with the value representing the number of bottom-level timesteps contained in the aggregation.	required
exog_vars	Optional[dict[str, Union[str, list[str]]]]	dictionary of string keys & values that can either be a list of strings or a single string keys correspond to column names and the values represent the aggregation(s) that will be applied to each column. Accepted values are those from Pandas or Polars aggregation Functions, check the respective docs for guidance. Default is None.	None
sparse_s	bool	Return S_df as a sparse Pandas dataframe. Default is False.	False
id_col	str	Column that will identify each serie after aggregation. Default is ‘unique_id’.	‘unique_id’
time_col	str	Column that identifies each timestep, its values can be timestamps or integers. Default is ‘ds’.	‘ds’
id_time_col	str	Column that will identify each timestep after aggregation. Default is ‘temporal_id’.	‘temporal_id’
target_cols	Sequence[str]	List of columns that contain the targets to aggregate. Default is (‘y’,).	(‘y’,)
aggregation_type	str	If ‘local’ the aggregation will be performed on the timestamps of each timeseries independently. If ‘global’ the aggregation will be performed on the unique timestamps of all timeseries. Default is ‘local’.	‘local’

Returns:

Type	Description
tuple[FrameT, FrameT, dict]	tuple[FrameT, FrameT, dict]: Y_df, S_df, tags Y_df: Temporally hierarchically structured series. S_df: Temporal summing dataframe. tags: Temporal aggregation indices.

make_future_dataframe

make_future_dataframe(df, freq, h, id_col='unique_id', time_col='ds')

Create future dataframe for forecasting.

Parameters:

Name	Type	Description	Default
df	Frame	Dataframe with ids, times and values for the exogenous regressors.	required
freq	Union[str, int]	Frequency of the data. Must be a valid pandas or polars offset alias, or an integer.	required
h	int	Forecast horizon.	required
id_col	str	Column that identifies each serie. Default is ‘unique_id’.	‘unique_id’
time_col	str	Column that identifies each timestep, its values can be timestamps or integers. Default is ‘ds’.	‘ds’

Returns:

Name	Type	Description
FrameT	FrameT	DataFrame with future values.

get_cross_temporal_tags

get_cross_temporal_tags(df, tags_cs, tags_te, sep='//', id_col='unique_id', id_time_col='temporal_id', cross_temporal_id_col='cross_temporal_id')

Get cross-temporal tags.

Parameters:

Name	Type	Description	Default
df	Frame	DataFrame with temporal ids.	required
tags_cs	dict[str, ndarray]	Tags for the cross-sectional hierarchies.	required
tags_te	dict[str, ndarray]	Tags for the temporal hierarchies.	required
sep	str	Separator for the cross-temporal tags. Default is ”//”.	’//‘
id_col	str	Column that identifies each series. Default is ‘unique_id’.	‘unique_id’
id_time_col	str	Column that identifies each (aggregated) timestep. Default is ‘temporal_id’.	‘temporal_id’
cross_temporal_id_col	str	Column that will identify each cross-temporal aggregation. Default is ‘cross_temporal_id’.	‘cross_temporal_id’

Returns:

Type	Description
tuple[FrameT, dict[str, ndarray]]	tuple[FrameT, dict[str, np.ndarray]]: df, tags_ct df: DataFrame with cross-temporal ids. tags_ct: Tags for the cross-temporal hierarchies.

Hierarchical Visualization

HierarchicalPlot

HierarchicalPlot(S, tags, S_id_col='unique_id')

Hierarchical Plot

This class contains a collection of matplotlib visualization methods, suited for small to medium sized hierarchical series.

Parameters:

Name	Type	Description	Default
S	Frame	DataFrame with summing matrix of size (base, bottom), see aggregate function.	required
tags	dict[str, ndarray]	hierarchical aggregation indexes, where each key is a level and its value contains tags associated to that level.	required
S_id_col	str	column that identifies each aggregation. Default is ‘unique_id’.	‘unique_id’

HierarchicalPlot.plot_summing_matrix

plot_summing_matrix()

Summation Constraints plot

This method simply plots the hierarchical aggregation constraints matrix $\mathbf{S}$.

Returns:

Name	Type	Description
fig	Figure	figure object containing the plot of the summing matrix.

HierarchicalPlot.plot_series

plot_series(series, Y_df, models=None, level=None, id_col='unique_id', time_col='ds', target_col='y')

Single Series plot

Parameters:

Name	Type	Description	Default
series	str	string identifying the 'unique_id' any-level series to plot.	required
Y_df	Frame	hierarchically structured series ($\mathbf{y}_{[a,b]}$). It contains columns ['unique_id', 'ds', 'y'], it may have 'models'.	required
models	Optional[list[str]]	string identifying filtering model columns. Default is None.	None
level	Optional[list[int]]	confidence levels for prediction intervals available in Y_df. Default is None.	None
id_col	str	column that identifies each series. Default is ‘unique_id’.	‘unique_id’
time_col	str	column that identifies each timestep, its values can be timestamps or integers. Default is ‘ds’.	‘ds’
target_col	str	column that contains the target. Default is ‘y’.	‘y’

Returns:

Name	Type	Description
fig	Figure	figure object containing the plot of the single series.

HierarchicalPlot.plot_hierarchically_linked_series

plot_hierarchically_linked_series(bottom_series, Y_df, models=None, level=None, id_col='unique_id', time_col='ds', target_col='y')

Hierarchically Linked Series plot

Parameters:

Name	Type	Description	Default
bottom_series	str	string identifying the 'unique_id' bottom-level series to plot.	required
Y_df	Frame	hierarchically structured series ($\mathbf{y}_{[a,b]}$). It contains columns [‘unique_id’, ‘ds’, ‘y’] and models.	required
models	Optional[list[str]]	string identifying filtering model columns. Default is None.	None
level	Optional[list[int]]	confidence levels for prediction intervals available in Y_df. Default is None.	None
id_col	str	column that identifies each series. Default is ‘unique_id’.	‘unique_id’
time_col	str	column that identifies each timestep, its values can be timestamps or integers. Default is ‘ds’.	‘ds’
target_col	str	column that contains the target. Default is ‘y’.	‘y’

Returns:

Name	Type	Description
fig	Figure	figure object containing the plots of the hierarchically linked series.

HierarchicalPlot.plot_hierarchical_predictions_gap

plot_hierarchical_predictions_gap(Y_df, models=None, xlabel=None, ylabel=None, id_col='unique_id', time_col='ds', target_col='y')

Hierarchically Predictions Gap plot

Parameters:

Name	Type	Description	Default
Y_df	Frame	hierarchically structured series ($\mathbf{y}_{[a,b]}$). It contains columns [‘unique_id’, ‘ds’, ‘y’] and models.	required
models	Optional[list[str]]	string identifying filtering model columns. Default is None.	None
xlabel	Optional[str]	string for the plot’s x axis label. Default is None.	None
ylabel	Optional[str]	string for the plot’s y axis label. Default is None.	None
id_col	str	column that identifies each series. Default is ‘unique_id’.	‘unique_id’
time_col	str	column that identifies each timestep, its values can be timestamps or integers. Default is ‘ds’.	‘ds’
target_col	str	column that contains the target. Default is ‘y’.	‘y’

Returns:

Name	Type	Description
fig	Figure	figure object containing the plot of the aggregated predictions at different levels of the hierarchical structure.

Example

from statsforecast.core import StatsForecast
from statsforecast.models import AutoETS
from datasetsforecast.hierarchical import HierarchicalData

Y_df, S, tags = HierarchicalData.load('./data', 'Labour')
Y_df['ds'] = pd.to_datetime(Y_df['ds'])
S = S.reset_index(names="unique_id")

Y_test_df  = Y_df.groupby('unique_id').tail(24)
Y_train_df = Y_df.drop(Y_test_df.index)

fcst = StatsForecast(
    models=[AutoETS(season_length=12, model='AAZ')],
    freq='MS',
    n_jobs=-1
)
Y_hat_df = fcst.forecast(df=Y_train_df, h=24).reset_index()

# Plot prediction difference of different aggregation
# Levels Country, Country/Region, Country/Gender/Region ...
hplots = HierarchicalPlot(S=S, tags=tags)

hplots.plot_hierarchical_predictions_gap(
    Y_df=Y_hat_df, models='AutoETS',
    xlabel='Month', ylabel='Predictions',
)