Benefits of Persistent Enrichment Tables

Oct 13, 2025 by talynt.org 145 views

A Note for the Community

As we continue to develop and improve Vector, we want to hear from you. If you're interested in this feature, please show your support by voting on this issue. You can do this by adding a 👍 reaction to the original issue. This will help us prioritize this request and make informed decisions about its implementation. Additionally, if you're interested in working on this issue or have submitted a pull request, please leave a comment. Your input is invaluable to us.

Use Cases

The new memory enrichment table feature in Vector has been a game-changer for many users. One such user has been utilizing this feature to enrich firewall logs with Wi-Fi controller logs. The Wi-Fi controller logs bring valuable information such as hostname, username, and MAC address. The enrichment process occurs when the source IP in the firewall logs matches an IP in the following table:

enrichment_tables:
  wifi_ip_table:
    type: memory
    ttl: 86400
    flush_interval: 1
    inputs: [ "wifi_cache_ips" ]

The "wifi_cache_ips" transform populates the table or modifies entries with up-to-date information. However, as Vector gets restarted at midnight due to log rotation, this memory table is wiped clean. This can be frustrating, especially when dealing with large datasets.

Attempted Solutions

Unfortunately, there have been no proposed solutions to this issue. It's essential that we address this problem to ensure that our users can continue to rely on Vector for their data processing needs.

Proposal

To address this issue, I propose the addition of a "survive_restarts" option in the enrichment table configuration. This option would allow users to specify whether the enrichment table should survive restarts or not. Here's an example of what this configuration might look like:

enrichment_tables:
  wifi_ip_table:
    type: memory
    ttl: 86400
    flush_interval: 1
    survive_restarts: true
    inputs: [ "wifi_cache_ips" ]

With this option, the enrichment table would be saved in a binary format when Vector receives a SIGTERM or similar signal. This would ensure that the table is preserved even after a restart. This feature would be particularly useful for users who need to maintain enrichment for IPs that are already assigned and won't appear again in Wi-Fi logs.

References

There are no references to relevant documentation or previous discussions on this topic.

Version

The version of Vector being used is 0.45.0 (x86_64-unknown-linux-gnu 063cabb 2025-02-24 14:52:02.810034614).

Benefits of Persistent Enrichment Tables

The ability to make memory enrichment tables persistent after Vector restarts would have several benefits. Firstly, it would eliminate the need to re-enrich data every time Vector restarts. This would save time and resources, especially for large datasets. Secondly, it would ensure that enrichment for IPs that are already assigned is maintained, even after a restart. This would prevent data loss and ensure that users can continue to rely on Vector for their data processing needs.

Implementation

To implement this feature, we would need to add a new option to the enrichment table configuration. This option would be used to determine whether the table should be saved in a binary format when Vector receives a SIGTERM or similar signal. We would also need to modify the Vector code to handle the saving and loading of the enrichment table. This would involve creating a new module that would be responsible for saving and loading the table.

Conclusion

In conclusion, the ability to make memory enrichment tables persistent after Vector restarts is a crucial feature that would benefit many users. By adding a "survive_restarts" option to the enrichment table configuration, we can ensure that enrichment for IPs that are already assigned is maintained, even after a restart. This would eliminate the need to re-enrich data every time Vector restarts and prevent data loss. We believe that this feature is essential and should be implemented as soon as possible.

Future Work

In the future, we plan to continue improving Vector and adding new features. Some potential future work includes:

Adding support for multiple enrichment table types
Improving the performance of enrichment tables
Adding support for more advanced data processing features

We believe that these features would further enhance the capabilities of Vector and make it an even more powerful tool for data processing.

Community Involvement

Q: What is the purpose of persistent enrichment tables in Vector?

A: The purpose of persistent enrichment tables in Vector is to allow users to save and load enrichment tables even after a restart. This ensures that enrichment for IPs that are already assigned is maintained, even after a restart, and eliminates the need to re-enrich data every time Vector restarts.

Q: How does the "survive_restarts" option work?

A: The "survive_restarts" option is a new feature that allows users to specify whether the enrichment table should survive restarts or not. When enabled, the enrichment table is saved in a binary format when Vector receives a SIGTERM or similar signal, and loaded every time Vector starts.

Q: What are the benefits of persistent enrichment tables?

A: The benefits of persistent enrichment tables include:

Elimination of the need to re-enrich data every time Vector restarts
Maintenance of enrichment for IPs that are already assigned, even after a restart
Prevention of data loss
Improved performance and efficiency

Q: How do I enable the "survive_restarts" option?

A: To enable the "survive_restarts" option, you need to add the following configuration to your Vector configuration file:

enrichment_tables:
  wifi_ip_table:
    type: memory
    ttl: 86400
    flush_interval: 1
    survive_restarts: true
    inputs: [ "wifi_cache_ips" ]

Q: What happens if I don't enable the "survive_restarts" option?

A: If you don't enable the "survive_restarts" option, the enrichment table will be wiped clean every time Vector restarts. This means that you will need to re-enrich data every time Vector restarts, which can be time-consuming and inefficient.

Q: Can I use persistent enrichment tables with other Vector features?

A: Yes, you can use persistent enrichment tables with other Vector features. However, you need to ensure that the features you are using are compatible with the "survive_restarts" option.

Q: How do I troubleshoot issues with persistent enrichment tables?

A: To troubleshoot issues with persistent enrichment tables, you can check the Vector logs for errors or warnings related to the "survive_restarts" option. You can also try disabling the "survive_restarts" option and re-enabling it to see if the issue is resolved.

Q: Can I customize the behavior of persistent enrichment tables?

A: Yes, you can customize the behavior of persistent enrichment tables by modifying the Vector configuration file. You can adjust the TTL, flush interval, and other settings to suit your specific needs.

Q: What are the system requirements for using persistent enrichment tables?

A: The system requirements for using persistent enrichment tables are the same as for using Vector in general. You need to ensure that your system has enough memory and disk space to handle the enrichment tables.

Q: Can I use persistent enrichment tables with other data processing tools?

A: Yes, you can use persistent enrichment tables with other data processing tools. However, you need to ensure that the tools you are using are compatible with the "survive_restarts" option.

Q: How do I get support for persistent enrichment tables?

A: You can get support for persistent enrichment tables by contacting the Vector community or seeking help from a Vector expert. You can also check the Vector documentation and online resources for more information.