Continuous integration and continuous deployment

Oct 13, 2025 by talynt.org 30 views

Area for Improvement

Long-haul tests on Kubernetes are usually designed to test the following criteria: detecting memory leaks and other performance issues, emulating a long-haul customer environment, including production-level cluster configuration, and version upgrade handling. However, our long-running tests today are in practice testing this list and more, including daily regression testing. Since these additional test cases are and should be tested in our other testing workflows, we should consider simplifying the long-running tests to achieve the above criteria.

The current long-running tests are designed to be comprehensive, but this has led to a few issues. Firstly, they are often prone to failure, which can generate a lot of noise, including GitHub issues. These issues take time to categorize and address, and can make the project appear unstable, when in reality the issues are test setup bugs or similar, not bugs in Radius itself. Secondly, the long-running tests are not only testing the above criteria but also daily regression testing, which can lead to unnecessary complexity.

Observed behavior

The long-running tests fail often and generate lots of noise, including GitHub issues. These issues take time to categorize and address, and makes the project appear unstable, when in reality the issues are test setup bugs or similar, not bugs in Radius itself. This can lead to a lot of unnecessary work for the development team, and can also make it difficult for new contributors to get started with the project.

For example, the long-running tests may fail due to a test setup bug, but the issue will still be reported as a bug in Radius. This can lead to a lot of confusion, and can make it difficult to determine whether the issue is a bug in Radius or a test setup bug. Additionally, the long-running tests may also fail due to changes in the environment, such as changes in the Kubernetes cluster configuration. This can lead to a lot of unnecessary work, as the development team will need to investigate and fix the issue.

Desired behavior

The long-running tests should have a much higher pass rate and they should be testing a different set of goals, as outlined above. The tests should be designed to detect memory leaks and other performance issues, emulate a long-haul customer environment, and handle version upgrades. This will ensure that the tests are comprehensive and effective, and will help to identify any issues with the project.

To achieve this, we should update the long-running tests to run on release drops, such as 0.45.0 and 0.45.1. This will ensure that the tests are run on a consistent and stable version of the project, and will help to identify any issues with the project. Additionally, we should remove the namespace cleanup steps in the long-running test workflow, as these are not necessary and can lead to unnecessary complexity.

Proposed Fix

Update the long-running tests to run on release drops: Each time a new release is detected, the workflow should do a full uninstall and clean of Radius (rad uninstall kubernetes, kubectl delete namespace radius-system, delete all application namespaces). The workflow should perform runs on this detected version of Radius until the next release.
Remove the namespace cleanup steps in the long-running test workflow: We should remove the namespace cleanup steps in the long-running test workflow, as these are not necessary and can lead to unnecessary complexity.

rad Version

N/A

Operating system

N/A

Additional context

No response

Would you like to support us?

[x] Yes, I would like to support you

AB#14977

Benefits of the Proposed Fix

The proposed fix has several benefits. Firstly, it will ensure that the long-running tests are run on a consistent and stable version of the project, which will help to identify any issues with the project. Secondly, it will remove the namespace cleanup steps in the long-running test workflow, which will reduce unnecessary complexity and make the tests more efficient. Finally, it will help to improve the pass rate of the long-running tests, which will make it easier to identify any issues with the project.

Implementation Plan

To implement the proposed fix, we will need to update the long-running tests to run on release drops. This will involve modifying the test workflow to perform a full uninstall and clean of Radius each time a new release is detected. We will also need to remove the namespace cleanup steps in the long-running test workflow.

The implementation plan will involve the following steps:

Update the long-running tests to run on release drops.
Remove the namespace cleanup steps in the long-running test workflow.
Test the updated long-running tests to ensure that they are working correctly.
Deploy the updated long-running tests to production.

Conclusion

In conclusion, the proposed fix will help to improve the pass rate of the long-running tests and reduce unnecessary complexity. It will also ensure that the long-running tests are run on a consistent and stable version of the project, which will help to identify any issues with the project. We believe that this is a good solution to the problem, and we are confident that it will help to improve the quality of the project.

Future Work

There are several areas of future work that we will need to consider. Firstly, we will need to continue to monitor the pass rate of the long-running tests and make any necessary adjustments to the test workflow. Secondly, we will need to consider adding additional tests to the long-running test workflow to ensure that the project is comprehensive and effective. Finally, we will need to consider implementing additional features to the long-running test workflow, such as automated testing and continuous integration.

References

[1] "Long-running tests on Kubernetes" by [Author]
[2] "Testing on Kubernetes" by [Author]
[3] "Continuous Integration and Continuous Deployment" by [Author]

Glossary

Long-running tests: Tests that are designed to test the project over a long period of time.
Release drops: New versions of the project that are released to production.
Namespace cleanup: The process of cleaning up the namespace of a Kubernetes cluster.
Automated testing: The process of running tests automatically as part of the continuous integration and continuous deployment process.
Continuous integration and continuous deployment: The process of automatically building, testing, and deploying the project to production.
Q&A: Long-running tests should be run on release builds ===========================================================

Q: What are long-running tests and why are they important?

A: Long-running tests are tests that are designed to test a project over a long period of time. They are important because they help to identify any issues with the project that may not be caught by shorter tests. Long-running tests can help to ensure that the project is stable and reliable, and can help to prevent issues from occurring in production.

Q: Why are long-running tests failing often?

A: Long-running tests are failing often because they are testing a lot of different scenarios and are prone to failure. Additionally, the tests may be testing the wrong things, or may be testing things in a way that is not representative of real-world usage. This can lead to a lot of unnecessary noise and can make it difficult to determine whether the issue is a bug in the project or a test setup bug.

Q: What is the proposed fix for the long-running tests?

A: The proposed fix is to update the long-running tests to run on release drops, such as 0.45.0 and 0.45.1. This will ensure that the tests are run on a consistent and stable version of the project, and will help to identify any issues with the project. Additionally, we will remove the namespace cleanup steps in the long-running test workflow, which will reduce unnecessary complexity and make the tests more efficient.

Q: How will the proposed fix improve the pass rate of the long-running tests?

A: The proposed fix will improve the pass rate of the long-running tests by ensuring that the tests are run on a consistent and stable version of the project. This will help to identify any issues with the project and will reduce the number of false positives. Additionally, removing the namespace cleanup steps will reduce unnecessary complexity and make the tests more efficient.

Q: What are the benefits of running long-running tests on release drops?

A: The benefits of running long-running tests on release drops include:

Ensuring that the tests are run on a consistent and stable version of the project
Reducing the number of false positives
Reducing unnecessary complexity and making the tests more efficient
Helping to identify any issues with the project

Q: How will the proposed fix be implemented?

A: The proposed fix will be implemented by updating the long-running tests to run on release drops, and removing the namespace cleanup steps in the long-running test workflow. This will involve modifying the test workflow to perform a full uninstall and clean of Radius each time a new release is detected.

Q: What are the next steps after implementing the proposed fix?

A: After implementing the proposed fix, we will need to continue to monitor the pass rate of the long-running tests and make any necessary adjustments to the test workflow. We will also need to consider adding additional tests to the long-running test workflow to ensure that the project is comprehensive and effective.

Q: What are the potential risks of implementing the proposed fix?

A: The potential risks of implementing the proposed fix include:

Introducing new bugs into the project
Causing the project to become unstable or unreliable
Reducing the pass rate of the long-running tests

Q: How will the potential risks be mitigated?

A: The potential risks will be mitigated by:

Thoroughly testing the proposed fix before implementing it
Monitoring the pass rate of the long-running tests and making any necessary adjustments to the test workflow
Considering adding additional tests to the long-running test workflow to ensure that the project is comprehensive and effective.

Q: What is the expected outcome of implementing the proposed fix?

A: The expected outcome of implementing the proposed fix is to improve the pass rate of the long-running tests, reduce unnecessary complexity, and ensure that the project is stable and reliable.