ios-safari tests are failing on Mac Try Bots

[nturgut]

We added ios-safari tests with PR: flutter/engine#19141

They worked fine for prod bots:
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/703
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/702
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/701
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/700
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/699
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/698
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/697
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/696
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/695
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine/694

On try bots they initially failed due to xcrun simctl output difference.
https://ci.chromium.org/p/flutter/builders/try/Mac%20Web%20Engine/5589

Fixed in PR: flutter/web_installers#18

With that change another failure occurred:
https://chromium-swarm.appspot.com/task?id=4d01d54edb630410

Skipping the tests for now. We need to understand what is different between try and prod in more detail.

[nturgut]

My initial findings are the hardware is different between the two (can also be seen using the links)

The try bot looks like this:

cores:12
cpu:x64x86x86-64-i7-8700B
gce:0
gpu:Intel (8086)8086:3e9b
hidpi:0
id:flutter-try-mac-6
mac_model:Macmini8,1
machine_type:n1-standard-8
os:MacMac-10.14Mac-10.14.6
pool:luci.flutter.try
python:2.7.10
role:builder
server_version:5218-123b763
ssd:1
zone:usus-corpus-corp-mtv

The prod bots have a different GPU:

cores | 4
-- | --
cpu | x64 \| x86 \| x86-64-i7-4578U
gce | 0
gpu | Intel (8086) \| Intel Haswell Iris Graphics 5100 (8086:0a2e)
hidpi | 0
id | build294-m9
mac_model | Macmini7,1
machine_type | n1-standard-4
os | Mac \| Mac-10.14 \| Mac-10.14.4
pool | luci.flutter.prod
python | 2.7.10
role | builder
server_version | 5218-123b763

[nturgut]

@yjbanov @godofredoc @christopherfujino what do you think?

I think we collected enough infra from prod. It passed all 10 runs so I don't think the tests or recipe has a problem.

I'll try to run a few different things (for example just opening one test URL), and update this issue.

[nturgut]

Looks like the I was unlucky by hitting bot 7 in the first run. It failed initially: https://chromium-swarm.appspot.com/task?id=4d01d54edb630410

Yet, all the other machines trials passed the test. I spend sometime to run each one of them. Results I collected for the all different machines:

• flutter-try-mac-7 PASSES : https://chromium-swarm.appspot.com/task?id=4d01d54edb630410
• flutter-try-mac-4 PASSES: https://ci.chromium.org/p/flutter/builders/try/Mac%20Web%20Engine/5594
• flutter-try-mac-8 PASSES: https://chromium-swarm.appspot.com/task?id=4d02abb6334ce510
  https://chromium-swarm.appspot.com/task?id=4d02e4c016f2c310
• flutter-try-mac-5 PASSES: https://chromium-swarm.appspot.com/task?id=4d02af4999666010
• flutter-try-mac-6 PASSES: https://chromium-swarm.appspot.com/task?id=4d02d4d3c1d3d610

@yjbanov @godofredoc I want to continue with this PR if it works for you: flutter/engine#19298

[nturgut]

Closing this issue since all the tests are passing for a couple weeks with no issues now.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.