Cirrus presubmits flake with "Instance got rescheduled!" message.

[chinmaygarde]

An instance of this failure is here. The documentation on Cirrus indicates that this should be a rare occurrence. The documentation also states that such tasks "will require rescheduling and automatically restart tasks". I don't think our presubmits are doing this since I waited about 30 mins for the automatic restart before manually re-running the job. If autorollers and external contributors are gated on such checks, we should consider mechanisms to detect and re-run these jobs on this edge case.

[chinmaygarde]

Re-running the task caused another failure with the same issue. Perhaps Cirrus is having a bad day today.

[godofredoc]

@fkorotkov do you have more server side logs with information why is this happening?

[fkorotkov]

@godofredoc do you have a task id where it happened?

Since it's running on your GKE cluster there is a limited visibility. It generally means that the particular container got killed unexpectedly. In case of the managed-by-us-infrastructure aka community tasks it means the underlying permissible VM got preempted. Do you have any preemptible VMs in your cluster?

[godofredoc]

Yes, we are using preemptible VMs in the cluster.

[godofredoc]

Closing as not actionable, the instances were preempted in the middle of the task execution which is expected with our current configurations.

If this starts happening very frequently please reopen this bug and we'll change the configuration to use not preemptible VMs

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.