Skip to content
This repository is currently being migrated. It's locked while the migration is in progress.

Handle partial results returned from a GetVolumes API call#229

Merged
Ricardo-Osorio merged 4 commits intomasterfrom
ctx
Feb 27, 2023
Merged

Handle partial results returned from a GetVolumes API call#229
Ricardo-Osorio merged 4 commits intomasterfrom
ctx

Conversation

@Ricardo-Osorio
Copy link
Contributor

@Ricardo-Osorio Ricardo-Osorio commented Feb 24, 2023

Handle the context deadline exceeded error from the backend (identified by the http status code PartialContent returned) and print the data we managed to get back

This data set will most likely fail to include some of the data about volume replicas so for those volumes we may see 0/2 replicas healthy even thought they may exist and be healthy

To make this clear for the user I have added a message that prints just before the table output that reads:
context deadline exceeded, these are partial results and may be missing replica states for some volumes

We expect this to be a rare case to hit

During the client.GetAllVolumes call we also fetch all namespaces
so store this information in the volume config and later use it
during the GetVolumes request.

Doing so removes one api call from being made during the GetVolumes
command.
@Ricardo-Osorio Ricardo-Osorio marked this pull request as ready for review February 27, 2023 10:51
@Ricardo-Osorio Ricardo-Osorio merged commit 302eefc into master Feb 27, 2023
@Ricardo-Osorio Ricardo-Osorio deleted the ctx branch February 27, 2023 13:40
Ricardo-Osorio added a commit that referenced this pull request Feb 27, 2023
Handle the context deadline exceeded error from the backend (identified
by the http status code PartialContent returned) and print the data we
managed to get back

This data set will most likely fail to include some of the data about
volume replicas so for those volumes we may see 0/2 replicas healthy
even thought they may exist and be healthy

To make this clear for the user I have added a message that prints just
before the table output that reads:
`context deadline exceeded, these are partial results and may be missing
replica states for some volumes`

We expect this to be a rare case to hit
@boedy
Copy link

boedy commented Mar 17, 2023

I've been experimenting with Ondat the last couple of weeks, but I've already hit this error message multiple times. This directly makes the CLI unusable to get or describe volumes.

➜  go-cli git:(master) ./bin/storageos describe node
Error: timed out performing command
➜  go-cli git:(master) ./bin/storageos get volumes -A
context deadline exceeded, these are partial results and may be missing replica states for some volumes
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x13a9f78]

goroutine 1 [running]:
code.storageos.net/storageos/c2-cli/output.newDeployment(0xc000098a00, 0x14e52a0?, 0x1, {0x1514b37, 0x1b})
        /Users/boedy/scratchpad/ondat/go-cli/output/volume.go:109 +0x98
code.storageos.net/storageos/c2-cli/output.newDeployments({0xc0000c2270, 0x1, 0x1513f01?}, 0x1514b37?, 0x1b?, {0x1514b37, 0x1b})
        /Users/boedy/scratchpad/ondat/go-cli/output/volume.go:133 +0xc8
code.storageos.net/storageos/c2-cli/output.NewVolume(0xc0000f4420, {0xc00002e274, 0x4}, 0x24?)
        /Users/boedy/scratchpad/ondat/go-cli/output/volume.go:92 +0x331
code.storageos.net/storageos/c2-cli/cmd/get.(*volumeCommand).runWithCtx.func2({0xc0000e20e0, 0xd, 0xc0000f14d0?}, 0x24?, 0xc000132008?)
        /Users/boedy/scratchpad/ondat/go-cli/cmd/get/volume.go:82 +0x114
code.storageos.net/storageos/c2-cli/cmd/get.(*volumeCommand).runWithCtx(0xc0001744d0, {0x15ce808, 0xc000134d20}, 0x13a?, {0xc00021c3e0?, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/get/volume.go:118 +0x746
code.storageos.net/storageos/c2-cli/cmd/runwrappers.AuthenticateClient.func1.1({0x15ce808, 0xc000134d20}, 0x150bb45?, {0xc00021c3e0, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/runwrappers/runwrappers.go:88 +0x12d
code.storageos.net/storageos/c2-cli/cmd/runwrappers.EnsureTargetOrSelectors.func1.1({0x15ce808?, 0xc000134d20?}, 0x1877e80?, {0xc00021c3e0?, 0x1877e80?, 0x37e11d600?})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/runwrappers/runwrappers.go:102 +0x55
code.storageos.net/storageos/c2-cli/cmd/runwrappers.EnsureNamespaceSetWhenUseIDs.func1.1({0x15ce808, 0xc000134d20}, 0x2b233e08?, {0xc00021c3e0, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/runwrappers/runwrappers.go:126 +0xfa
code.storageos.net/storageos/c2-cli/cmd/runwrappers.RunWithTimeout.func1.1({0x15ce7d0?, 0xc000136008?}, 0xc000167bb8?, {0xc00021c3e0, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/runwrappers/runwrappers.go:62 +0xa7
code.storageos.net/storageos/c2-cli/cmd/runwrappers.Chain.func1.1({0x15ce7d0, 0xc000136008}, 0x4?, {0xc00021c3e0, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/runwrappers/runwrappers.go:49 +0xa3
code.storageos.net/storageos/c2-cli/cmd/get.newVolume.func3(0xc0001c1b80?, {0xc00021c3e0, 0x0, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/cmd/get/volume.go:328 +0x1d2
github.com/spf13/cobra.(*Command).execute(0xc0001c1b80, {0xc00021c3c0, 0x1, 0x1})
        /Users/boedy/scratchpad/ondat/go-cli/vendor/github.com/spf13/cobra/command.go:826 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000147900)
        /Users/boedy/scratchpad/ondat/go-cli/vendor/github.com/spf13/cobra/command.go:914 +0x2ef
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/boedy/scratchpad/ondat/go-cli/vendor/github.com/spf13/cobra/command.go:864
main.main()
        /Users/boedy/scratchpad/ondat/go-cli/main.go:50 +0x345

What's worse is that there is currently no way to resolve the issue. The faulty volumes / replica's are not visible in k8s and I can't use the cli to remove them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants