Retry is in the wrong level

Oct 13, 2025 by talynt.org 131 views

Introduction

Minikube is a popular tool for running Kubernetes locally on a machine. However, users have reported issues with stopping the minikube driver, particularly when using the VFKit driver. In this article, we will delve into the root cause of this issue and provide a step-by-step guide to reproduce and troubleshoot the problem.

What Happened?

Stopping minikube driver is very slow due to unreliable implementation, incorrect error handling, and pointless backup attempts. The VFKit driver is used to manage the virtual machine, and its implementation is the primary cause of the issue.

To Reproduce

To reproduce the issue, follow these steps:

Step 1: Configuration

minikube config set vm-driver vfkit
minikube config set container-runtime containerd

Step 2: Start Cluster

minikube start

Step 3: Stop Cluster

minikube stop --v=4 --alsologtostderr 2>stop.log

Timeline

The following is a step-by-step breakdown of the timeline of events when stopping the minikube driver:

1. Stopping Node Started

I0309 15:00:42.297876   13723 out.go:177] * Stopping node "minikube"  ...

2. Performing First Backup

I0309 15:00:42.307878   13723 machine.go:156] backing up vm config to /var/lib/minikube/backup: [/etc/cni /etc/kubernetes]
I0309 15:00:42.307953   13723 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/minikube/backup
I0309 15:00:42.307982   13723 sshutil.go:53] new ssh client: &{IP:192.168.106.7 Port:22 SSHKeyPath:/Users/nir/.minikube/machines/minikube/id_rsa Username:docker}
I0309 15:00:42.339888   13723 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/cni /var/lib/minikube/backup
I0309 15:00:42.386662   13723 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/kubernetes /var/lib/minikube/backup

3. Stopping the Driver

I0309 15:00:42.433319   13723 main.go:141] libmachine: Stopping "minikube"...
I0309 15:00:42.433868   13723 main.go:141] libmachine: get state: {State:VirtualMachineStateRunning}

4. Trying to Set State to `HardStop` via HTTP API

I0309 15:00:42.452986   13723 stop.go:66] stop err: Post "http://_/vm/state": EOF
W0309 15:00:42.453070   13723 stop.go:165] stop host returned error: Temporary Error: stop: Post "http://_/vm/state": EOF

5. Starting Retry

I0309 15:00:42.453102   13723 retry.go:31] will retry after 1.148164137s: Temporary Error: stop: Post "http://_/vm/state": EOF

6. Stopping the Host Again

I0309 15:00:43.601796   13723 stop.go:39] StopHost: minikube
I0309 15:00:43.608726   13723 out.go:177] * Stopping node "minikube"  ...
I0309 15:00:43.616885   13723 machine.go:156] backing up vm config to /var/lib/minikube/backup: [/etc/cni /etc/kubernetes]
I0309 15:00:43.617271   13723 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/minikube/backup
I0309 15:00:43.617329   13723 sshutil.go:53] new ssh client: &{IP:192.168.106.7 Port:22 SSHKeyPath:/Users/nir/.minikube/machines/minikube/id_rsa Username:docker}
W0309 15:01:58.617799   13723 sshutil.go:64] dial failure (will retry): dial tcp 192.168.106.7:22: connect: operation timed out
W0309 15:01:58.618132   13723 stop.go:55] failed to complete vm config backup (will continue): create dir: NewSession: new client: new client: dial tcp 192.168.106.7:22: connect: operation timed out

Issues

The following are the key issues with the VFKit driver:

Using HardStop instead of Stop: This does not perform a graceful shutdown of the host.
Using HardStop always fails: The error is not handled properly, leading to a retry loop.
Retry is in the wrong level: The backup attempt is repeated after a successful backup, causing unnecessary delays.
Backup times out: The host is not running, and the previous error is not handled properly, leading to a timeout.

Conclusion

In conclusion, the VFKit driver is the primary cause of the slow minikube stop issue. The implementation is unreliable, and the error handling is incorrect. The retry loop is in the wrong level, and the backup attempt times out due to the host not being running. To resolve this issue, the VFKit driver should be updated to handle errors properly, and the retry loop should be adjusted to avoid unnecessary delays.

Recommendations

Based on the analysis, the following recommendations are made:

Update the VFKit driver: The VFKit driver should be updated to handle errors properly and perform a graceful shutdown of the host.
Adjust the retry loop: The retry loop should be adjusted to avoid unnecessary delays and ensure that the backup attempt is successful.
Improve error handling: The error handling should be improved to handle errors properly and prevent the retry loop from running indefinitely.

Q: What is the root cause of the slow minikube stop issue?

A: The root cause of the slow minikube stop issue is the unreliable implementation of the VFKit driver, incorrect error handling, and pointless backup attempts.

Q: What are the key issues with the VFKit driver?

A: The key issues with the VFKit driver are:

Using HardStop instead of Stop: This does not perform a graceful shutdown of the host.
Using HardStop always fails: The error is not handled properly, leading to a retry loop.
Retry is in the wrong level: The backup attempt is repeated after a successful backup, causing unnecessary delays.
Backup times out: The host is not running, and the previous error is not handled properly, leading to a timeout.

Q: How can I reproduce the issue?

A: To reproduce the issue, follow these steps:

Step 1: Configuration

minikube config set vm-driver vfkit
minikube config set container-runtime containerd

Step 2: Start Cluster

minikube start

Step 3: Stop Cluster

minikube stop --v=4 --alsologtostderr 2>stop.log

Q: What is the timeline of events when stopping the minikube driver?

A: The following is a step-by-step breakdown of the timeline of events when stopping the minikube driver:

1. Stopping Node Started

I0309 15:00:42.297876   13723 out.go:177] * Stopping node "minikube"  ...

2. Performing First Backup

I0309 15:00:42.307878   13723 machine.go:156] backing up vm config to /var/lib/minikube/backup: [/etc/cni /etc/kubernetes]
I0309 15:00:42.307953   13723 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/minikube/backup
I0309 15:00:42.307982   13723 sshutil.go:53] new ssh client: &{IP:192.168.106.7 Port:22 SSHKeyPath:/Users/nir/.minikube/machines/minikube/id_rsa Username:docker}
I0309 15:00:42.339888   13723 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/cni /var/lib/minikube/backup
I0309 15:00:42.386662   13723 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/kubernetes /var/lib/minikube/backup

3. Stopping the Driver

I0309 15:00:42.433319   13723 main.go:141] libmachine: Stopping "minikube"...
I0309 15:00:42.433868   13723 main.go:141] libmachine: get state: {State:VirtualMachineStateRunning}

4. Trying to Set State to `HardStop` via HTTP API

I0309 15:00:42.452986   13723 stop.go:66] stop err: Post "http://_/vm/state": EOF
W0309 15:00:42.453070   13723 stop.go:165] stop host returned error: Temporary Error: stop: Post "http://_/vm/state": EOF

5. Starting Retry

I0309 15:00:42.453102   13723 retry.go:31] will retry after 1.148164137s: Temporary Error: stop: Post "http://_/vm/state": EOF

6. Stopping the Host Again

I0309 15:00:43.601796   13723 stop.go:39] StopHost: minikube
I0309 15:00:43.608726   13723 out.go:177] * Stopping node "minikube"  ...
I0309 15:00:43.616885   13723 machine.go:156] backing up vm config to /var/lib/minikube/backup: [/etc/cni /etc/kubernetes]
I0309 15:00:43.617271   13723 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/minikube/backup
I0309 15:00:43.617329   13723 sshutil.go:53] new ssh client: &{IP:192.168.106.7 Port:22 SSHKeyPath:/Users/nir/.minikube/machines/minikube/id_rsa Username:docker}
W0309 15:01:58.617799   13723 sshutil.go:64] dial failure (will retry): dial tcp 192.168.106.7:22: connect: operation timed out
W0309 15:01:58.618132   13723 stop.go:55] failed to complete vm config backup (will continue): create dir: NewSession: new client: new client: dial tcp 192.168.106.7:22: connect: operation timed out

Q: How can I troubleshoot the issue?

A: To troubleshoot the issue, follow these steps:

Check the logs: Check the logs for any errors or warnings that may indicate the cause of the issue.
Verify the VFKit driver: Verify that the VFKit driver is properly configured and that there are no issues with the driver itself.
Check the network connection: Check the network connection to ensure that it is stable and that there are no issues with the connection.
Try a different driver: Try using a different driver to see if the issue is specific to the VFKit driver.

Q: How can I resolve the issue?

A: To resolve the issue, follow these steps:

Update the VFKit driver: Update the VFKit driver to the latest version to ensure that you have the latest fixes and improvements.
Adjust the retry loop: Adjust the retry loop to avoid unnecessary delays and ensure that the backup attempt is successful.
Improve error handling: Improve error handling to handle errors properly and prevent the retry loop from running indefinitely.
Check the network connection: Check the network connection to ensure that it is stable and that there are no issues with the connection.