Extend some arguments in hooks

Oct 13, 2025 by talynt.org 112 views

Introduction

As a user of Kedro, a popular open-source framework for data science and machine learning, you may have encountered the need to save a versioned set of parameters used in your pipelines. This can be a crucial step in tracking changes and ensuring reproducibility. However, finding a straightforward solution can be challenging, especially for new users. In this article, we will explore the current solution and propose possible improvements to make it easier to get versioned set of parameters saved.

Current Solution

The current solution involves using a hook to save versioned data after a pipeline completes. The hook implementation is as follows:

@hook_impl
def after_pipeline_run(
    self,
    run_params: dict[str, Any],
    catalog: DataCatalog,
) -> None:
    """Hook implementation to run after a pipeline completes to save versioned data."""
    session_id = run_params["session_id"]
    parameters = catalog.load("parameters")

    # Create dataframe with session_id and parameters
    parameters_df = pd.DataFrame(
        {"session_id": [session_id], "parameters": [str(parameters)]}
    )

    try:
        # Try to load existing versioned parameters
        versioned_df = catalog.load("parameters_aggregated")
        # Append new parameters
        combined_df = pd.concat([versioned_df, parameters_df], ignore_index=True)
    except DatasetError:
        # If doesn't exist yet, use current parameters
        combined_df = parameters_df

    # Save back to catalog
    catalog.save("parameters_aggregated", combined_df)
    self._logger.info(f"Saved versioned parameters with session {session_id}")

This solution involves loading the parameters from the catalog, creating a dataframe with the session ID and parameters, and then saving the combined dataframe back to the catalog. However, this solution requires a good understanding of Kedro's hooks and catalog, which can be overwhelming for new users.

Context

The journey to finding a solution for saving versioned parameters was not straightforward. The process involved:

Searching for parameter saving/versioning yielded no results
Realizing that hooks might be a solution, but the documentation was lacking
Struggling to understand what hooks are available and how to interact with the catalog
Realizing that the catalog contains the parameters, which was the last missing piece

This experience highlights the need for better documentation and easier access to information.

Possible Implementation

To make it easier to get versioned set of parameters saved, the following possible implementations can be considered:

Better/easier documentation: Improve the documentation on hooks and the catalog to get users to the answer of this question quicker.
Extend some arguments in hooks: Extend some arguments in hooks to have more variables, making it easier to access the parameters and session ID.
Build something even more native: Build a more native solution, but this is unclear what this would look like.

In the end, it's probably realistic to only do the first, but due to the other options, it's left as a feature request.

Conclusion

Saving versioned set of parameters is a crucial step in data science and machine learning pipelines. However, finding a straightforward solution can be challenging, especially for new users. The current solution involves using a hook to save versioned data after a pipeline completes, but it requires a good understanding of Kedro's hooks and catalog. To make it easier to get versioned set of parameters saved, better documentation and easier access to information are needed. Possible implementations include improving documentation, extending some arguments in hooks, and building a more native solution.

Recommendations

Based on the experience and possible implementations, the following recommendations can be made:

Improve the documentation on hooks and the catalog to get users to the answer of this question quicker.
Extend some arguments in hooks to have more variables, making it easier to access the parameters and session ID.
Consider building a more native solution, but this is unclear what this would look like.

By following these recommendations, Kedro can make it easier for users to save versioned set of parameters, improving the overall user experience and making it more accessible to new users.

Future Work

Future work can include:

Improving the documentation on hooks and the catalog.
Extending some arguments in hooks to have more variables.
Building a more native solution.

By addressing these areas, Kedro can continue to improve and make it easier for users to save versioned set of parameters.

Appendix

The following is the code snippet used in the article:

@hook_impl
def after_pipeline_run(
    self,
    run_params: dict[str, Any],
    catalog: DataCatalog,
) -> None:
    """Hook implementation to run after a pipeline completes to save versioned data."""
    session_id = run_params["session_id"]
    parameters = catalog.load("parameters")

    # Create dataframe with session_id and parameters
    parameters_df = pd.DataFrame(
        {"session_id": [session_id], "parameters": [str(parameters)]}
    )

    try:
        # Try to load existing versioned parameters
        versioned_df = catalog.load("parameters_aggregated")
        # Append new parameters
        combined_df = pd.concat([versioned_df, parameters_df], ignore_index=True)
    except DatasetError:
        # If doesn't exist yet, use current parameters
        combined_df = parameters_df

    # Save back to catalog
    catalog.save("parameters_aggregated", combined_df)
    self._logger.info(f"Saved versioned parameters with session {session_id}")
```<br/>
**Q&A: Make it Easier to Get Versioned Set of Parameters Saved**
===========================================================

**Q: What is the current solution for saving versioned set of parameters in Kedro?**
--------------------------------------------------------------------------------

A: The current solution involves using a hook to save versioned data after a pipeline completes. The hook implementation is as follows:

```python
@hook_impl
def after_pipeline_run(
    self,
    run_params: dict[str, Any],
    catalog: DataCatalog,
) -> None:
    """Hook implementation to run after a pipeline completes to save versioned data."""
    session_id = run_params["session_id"]
    parameters = catalog.load("parameters")

    # Create dataframe with session_id and parameters
    parameters_df = pd.DataFrame(
        {"session_id": [session_id], "parameters": [str(parameters)]}
    )

    try:
        # Try to load existing versioned parameters
        versioned_df = catalog.load("parameters_aggregated")
        # Append new parameters
        combined_df = pd.concat([versioned_df, parameters_df], ignore_index=True)
    except DatasetError:
        # If doesn't exist yet, use current parameters
        combined_df = parameters_df

    # Save back to catalog
    catalog.save("parameters_aggregated", combined_df)
    self._logger.info(f"Saved versioned parameters with session {session_id}")

Q: What are the challenges in finding a solution for saving versioned set of parameters in Kedro?

A: The challenges include:

Searching for parameter saving/versioning yields no results
Realizing that hooks might be a solution, but the documentation is lacking
Struggling to understand what hooks are available and how to interact with the catalog
Realizing that the catalog contains the parameters, which was the last missing piece

Q: What are the possible implementations to make it easier to get versioned set of parameters saved?

A: The possible implementations include:

Better/easier documentation: Improve the documentation on hooks and the catalog to get users to the answer of this question quicker.
Extend some arguments in hooks: Extend some arguments in hooks to have more variables, making it easier to access the parameters and session ID.
Build something even more native: Build a more native solution, but this is unclear what this would look like.

Q: What are the recommendations to make it easier to get versioned set of parameters saved?

A: The recommendations include:

Improve the documentation on hooks and the catalog to get users to the answer of this question quicker.
Extend some arguments in hooks to have more variables, making it easier to access the parameters and session ID.
Consider building a more native solution, but this is unclear what this would look like.

Q: What is the future work to make it easier to get versioned set of parameters saved?

A: The future work includes:

Improving the documentation on hooks and the catalog.
Extending some arguments in hooks to have more variables.
Building a more native solution.

Q: What is the code snippet used in the article?

A: The code snippet used in the article is as follows:

@hook_impl
def after_pipeline_run(
    self,
    run_params: dict[str, Any],
    catalog: DataCatalog,
) -> None:
    """Hook implementation to run after a pipeline completes to save versioned data."""
    session_id = run_params["session_id"]
    parameters = catalog.load("parameters")

    # Create dataframe with session_id and parameters
    parameters_df = pd.DataFrame(
        {"session_id": [session_id], "parameters": [str(parameters)]}
    )

    try:
        # Try to load existing versioned parameters
        versioned_df = catalog.load("parameters_aggregated")
        # Append new parameters
        combined_df = pd.concat([versioned_df, parameters_df], ignore_index=True)
    except DatasetError:
        # If doesn't exist yet, use current parameters
        combined_df = parameters_df

    # Save back to catalog
    catalog.save("parameters_aggregated", combined_df)
    self._logger.info(f"Saved versioned parameters with session {session_id}")

Q: What are the benefits of making it easier to get versioned set of parameters saved?

A: The benefits include:

Improved reproducibility of results
Easier tracking of changes
Better collaboration and communication among team members
Improved overall user experience

Q: What are the potential drawbacks of making it easier to get versioned set of parameters saved?

A: The potential drawbacks include:

Increased complexity of the code
Potential performance issues
Need for additional documentation and training

Q: What are the next steps to make it easier to get versioned set of parameters saved?

A: The next steps include:

Implementing the recommended solutions
Testing and evaluating the results
Refining and iterating on the solutions as needed
Documenting the solutions and providing training and support to users.