cloud-provider-kind does not respect KIND_EXPERIMENTAL_PROVIDER=podman if Docker is installed

[avosepp]

Hello All,

Trying to run a Kind cluster with Podman on the latest MacOS today and came across this.

E0504 12:22:31.375083   26245 controller.go:71] "Error listing clusters, retrying..." err="failed to list clusters: command \"docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label \"io.x-k8s.kind.cluster\"}}'\" failed with error: exit status 1"

This failure occurs when trying to run cloud-provider-kind. I have alias docker=podman and the KIND_EXPERIMENTAL_PROVIDER=podman env var in both my shell and the sudo shell. cloud-provider-kind was still attempting to use Docker and erroring with the above error.

I ran across #242 where @colinjlacy pointed out that completely uninstalling Docker from their machine resolved the issue. I tested the same on my machine and was able to verify. After Docker was removed, my machine began to respect the KIND_EXPERIMENTAL_PROVIDER=podman env var.

It seems like in the code somewhere the check of Docker being installed is overriding the env var or even the alias.

I think it's reasonable for a user to have both Docker and Podman installed on their machine. It's a little difficult to troubleshoot.

[stmcginnis]

Are you running with sudo? The question of how KIND_EXPERIMENTAL_PROVIDER was being set was never answered in #242.

[avosepp]

sudo KIND_EXPERIMENTAL_PROVIDER=podman cloud-provider-kind

This did not work for me. Would you like me to try some alternatives and report back?

EDIT: Did not work for me when Docker was installed. It worked after Docker was uninstalled.

[stmcginnis]

Let's try this:

export KIND_EXPERIMENTAL_PROVIDER=podman
sudo -E cloud-provider-kind

[avosepp]

export KIND_EXPERIMENTAL_PROVIDER=podman

❌ With Docker Installed

sudo -E cloud-provider-kind
Enter PIN for 'Certificate For PIV Authentication (Yubico PIV Authentication)':
I0504 13:27:46.955016   54670 app.go:77] FLAG: --enable-default-ingress="true"
I0504 13:27:46.955049   54670 app.go:77] FLAG: --enable-lb-port-mapping="false"
I0504 13:27:46.955053   54670 app.go:77] FLAG: --enable-log-dumping="false"
I0504 13:27:46.955057   54670 app.go:77] FLAG: --gateway-channel="standard"
I0504 13:27:46.955060   54670 app.go:77] FLAG: --help="false"
I0504 13:27:46.955063   54670 app.go:77] FLAG: --logs-dir=""
I0504 13:27:46.955065   54670 app.go:77] FLAG: --verbosity="2"
E0504 13:27:46.982155   54670 controller.go:71] "Error listing clusters, retrying..." err="failed to list clusters: command \"docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label \"io.x-k8s.kind.cluster\"}}'\" failed with error: exit status 1"

✅ Without Docker installed

sudo -E cloud-provider-kind
Enter PIN for 'Certificate For PIV Authentication (Yubico PIV Authentication)':
I0504 13:28:43.862638   55206 app.go:77] FLAG: --enable-default-ingress="true"
I0504 13:28:43.862676   55206 app.go:77] FLAG: --enable-lb-port-mapping="false"
I0504 13:28:43.862680   55206 app.go:77] FLAG: --enable-log-dumping="false"
I0504 13:28:43.862683   55206 app.go:77] FLAG: --gateway-channel="standard"
I0504 13:28:43.862686   55206 app.go:77] FLAG: --help="false"
I0504 13:28:43.862689   55206 app.go:77] FLAG: --logs-dir=""
I0504 13:28:43.862691   55206 app.go:77] FLAG: --verbosity="2"
enabling experimental podman provider
...
...

[aojea]

the problem is somehow it uses docker when it should not?

[avosepp]

Yes. It should respect KIND_EXPERIMENTAL_PROVIDER, which comes from the Kind configuration. It should respect that regardless of what daemons are running or installed.

[BenTheElder]

Yes. It should respect KIND_EXPERIMENTAL_PROVIDER, which comes from the Kind configuration. It should respect that regardless of what daemons are running or installed.

Agreed, I think there must be a bug reading the env, I would prefer to fix that versus trying to sniff if the daemon is running (which doesn't even necessarily apply to podman for example).

[stmcginnis]

I've submitted #405 related to this. That should make it a little better. But it still doesn't explain why your environment variable setting isn't being picked up.

If possible, I think it would be worth adding some temporary logging lines around the env var settings logic to validation assumptions here. Something doesn't seem right, and until we collect some more tracing to actually understand what's happening where, I'm not seeing anything obvious that explains the behavior you're seeing.

Are you comfortable cloning the repo locally, adding in a couple log lines around this variable inspection, then running from the locally modified code?

The new lines would probably be something like:

klog.Warningf("KIND_EXPERIMENTAL_PROVIDER value %q", os.Getenv("KIND_EXPERIMENTAL_PROVIDER"))

anywhere in the code where you see os.Getenv("KIND_EXPERIMENTAL_PROVIDER") being done.

[stmcginnis]

Or if you can check out and run with the changes in #406, that might address this.

[avosepp]

Let me see if I can get setup to compile and run locally. Then I can try #406.