flutter: 3.3.8 -> 3.7.9

[algorithmiker]

This updates flutter and will also probably fix the dependency- armageddon we are seeing with #211606.

This is mostly @FlafyDev's work, who said it's okay for me to upstream the changes.

Flutter 3.7 was attempted earlier by @AlexandraSjel, whom we thank greatly.

This PR therefore
Closes #215754
Closes #215537

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[algorithmiker]

@FlafyDev can you here state publicly that you are ok with those changes being merged?
Did you use any work from AtaraxiaSjel?

[AtaraxiaSjel]

@gilice, thank you for the work! Unfortunately, I haven't had a lot of free time lately and I'm glad someone took on this problem :) Should I close my PR now?

[algorithmiker]

Let's wait until we merge this, I guess.

[FlafyDev]

@FlafyDev can you here state publicly that you are ok with those changes being merged? Did you use any work from AtaraxiaSjel?

Yes, I have no problem for these changes to be merged to Nixpkgs.
Thanks for upstreaming them!

Also no, I haven't used any of the changes from @AtaraxiaSjel's PR.

[NickCao]

The patches are failing to apply

       > applying patch /nix/store/9671j6kqrbyrpzpas7rr172kk7jwbkj5-remove-test-dependency.patch
       > patching file packages/flutter_tools/pubspec.yaml
       > Hunk #1 FAILED at 98.
       > 1 out of 1 hunk FAILED -- saving rejects to file packages/flutter_tools/pubspec.yaml.re

[algorithmiker]

Yes, I am aware. working on it, will make this WIP in the meantime

[algorithmiker]

broken by flutter/flutter#121802 lmao

edit: fixed, that was pretty fast

[algorithmiker]

This finally builds !!! :)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/2022

[MGlolenstine]

I receive an error while trying to build for Android.

Error

✦ ❯ flutter run
The new artifact "libimobiledevice" was downloaded, but Flutter failed to update its stamp file, receiving the error "FileSystemException: Cannot open file, path =
'/nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/bin/cache/libimobiledevice.stamp' (OS Error: Read-only file system, errno = 30)". Flutter can continue, but the artifact may be re-downloaded on subsequent invocations
until the problem is resolved.
The new artifact "usbmuxd" was downloaded, but Flutter failed to update its stamp file, receiving the error "FileSystemException: Cannot open file, path =
'/nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/bin/cache/usbmuxd.stamp' (OS Error: Read-only file system, errno = 30)". Flutter can continue, but the artifact may be re-downloaded on subsequent invocations until the
problem is resolved.
Running "flutter pub get" in flutter_tools...
Resolving dependencies in ../../../../nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/packages/flutter_tools... (1.6s)
  _fe_analyzer_shared 50.0.0 (58.0.0 available)
  analyzer 5.2.0 (5.10.0 available)
  archive 3.3.2 (3.3.6 available)
  args 2.3.1 (2.4.0 available)
  async 2.10.0 (2.11.0 available)
  built_value 8.4.2 (8.4.4 available)
  checked_yaml 2.0.1 (2.0.2 available)
  collection 1.17.0 (1.17.1 available)
  completion 1.0.0 (1.0.1 available)
  coverage 1.6.1 (1.6.3 available)
  dds 2.5.0 (2.7.7 available)
  dds_service_extensions 1.3.1 (1.3.3 available)
  devtools_shared 2.18.0 (2.22.2 available)
  dwds 16.0.2+1 (18.0.2 available)
  fixnum 1.0.1 (1.1.0 available)
  frontend_server_client 3.1.0 (3.2.0 available)
  html 0.15.1 (0.15.2 available)
  intl 0.17.0 (0.18.0 available)
  io 1.0.3 (1.0.4 available)
  js 0.6.5 (0.6.7 available)
  json_annotation 4.7.0 (4.8.0 available)
  logging 1.1.0 (1.1.1 available)
  matcher 0.12.13 (0.12.15 available)
  meta 1.8.0 (1.9.1 available)
  mime 1.0.2 (1.0.4 available)
  multicast_dns 0.3.2+2 (0.3.2+3 available)
  native_stack_traces 0.5.2 (0.5.5 available)
  node_preamble 2.0.1 (2.0.2 available)
  path 1.8.2 (1.8.3 available)
  petitparser 5.1.0 (5.3.0 available)
  pubspec_parse 1.2.1 (1.2.2 available)
  source_maps 0.10.11 (0.10.12 available)
  sse 4.1.1 (4.1.2 available)
  test 1.22.0 (1.24.1 available)
  test_api 0.4.16 (0.5.1 available)
  test_core 0.4.20 (0.5.1 available)
  vm_service 9.4.0 (11.3.0 available)
  web_socket_channel 2.2.0 (2.3.0 available)
  webdriver 3.0.1 (3.0.2 available)
Got dependencies in ../../../../nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/packages/flutter_tools!
Cannot open file, path = '../../../../nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/packages/flutter_tools/pubspec.lock' (OS Error: Read-only file system, errno = 30)
pub get failed
command: "/nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/bin/cache/dart-sdk/bin/dart __deprecated_pub --color --directory
../../../../nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped/packages/flutter_tools get --example"
pub env: {
  "FLUTTER_ROOT": "/nix/store/6kg35py06f1jvnmglm51nxchy1gcbc3y-flutter-3.7.9-unwrapped",
  "PUB_ENVIRONMENT": "flutter_cli:get",
  "PUB_CACHE": "/home/arch/.pub-cache",
}
exit code: 66

I have added the above repository to the flake, locked it to the latest commit

    flutterNixpkgs = {
      type = "github";
      owner = "gilice";
      repo = "nixpkgs";
      rev = "dd242357fde311d4695d265baebd232a7ad7cf1a";
    };

And I've added flutterPkgs.flutter to the buildInputs (for the mkShell).

[algorithmiker]

also @FlafyDev flutter2 fails because it's dart doesn't have precache, we need separate builders.

[FlafyDev]

also @FlafyDev flutter2 fails because it's dart doesn't have precache, we need separate builders.

Why package flutter 2?

[algorithmiker]

@FlafyDev firmware-updater depends on it.

[FlafyDev]

@FlafyDev firmware-updater depends on it.

Latest version of firmware-updater is flutter 3.
Same thing with fluffychat

[algorithmiker]

OK, going to bump that then.

[algorithmiker]

I can reproduce the issues @MGlolenstine is having, tho.

[algorithmiker]

After I fix the arm64 hashes, this should be complete.
It remains future work to deprecate flutter2 and probably mkFlutterApp with it.

In the future, the most safe way to proceed seems to be to create a dartPackages.

[algorithmiker]

We got the arm64 hashes thanks to @aktaboot.
Undrafting, ready for re-review.

I've been dogfooding this for the last few days without issues.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/2041

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/900

[NickCao]

Result of nixpkgs-review pr 224090 run on x86_64-linux 1

2 packages failed to build:

• firmware-updater
• yubioath-flutter

3 packages built:

• flutter (flutterPackages.stable)
• flutter2 (flutterPackages.v2)
• hover

Failures are due to hash mismatch.

[algorithmiker]

@NickCao can you please rerun nixpkgs-review?

[NickCao]

Result of nixpkgs-review pr 224090 run on x86_64-linux 1

1 package failed to build:

• yubioath-flutter

4 packages built:

• firmware-updater
• flutter (flutterPackages.stable)
• flutter2 (flutterPackages.v2)
• hover

There's still some issue with the hashes.

[algorithmiker]

aarch64 hash for yubioath-flutter: sha256-gReSoBt+mjtQHLEBdWwJuYClv1wG3NLRIpJZa3Mcixo=
x86: sha256-XPXyTf5NBe3DovRle+l3Fr6Y7wBOUmFZy0GpgNCtR1w=

[algorithmiker]

@NickCao can we do another go? the packages build individually fine now, but I cannot review because it OOMs

[NickCao]

@ofborg build firmware-updater yubioath-flutter hover

[NickCao]

Failed once again due to hash mismatch

error: hash mismatch in fixed-output derivation '/nix/store/483xb8c5mx76r65rjr98nvi31f64zv14-yubioath-flutter-6.1.0-deps-flutter-v3.7.9-aarch64-linux.tar.gz.drv':
         specified: sha256-gReSoBt+mjtQHLEBdWwJuYClv1wG3NLRIpJZa3Mcixo=
            got:    sha256-5TuZn8GNVKIk5J5MHyuqMScHZ3WqEOVI52Zbfd/1EKk=

They might still be non reproducible.

[hacker1024]

Would you be able to split this up into smaller commits? There is a lot being changed here (preload support, wrapper changes, added libraries, etc.).

[hacker1024]

Regarding the changes to wrapping, the FHS, and libraries, please consider my alternative approach over at #201027.

This PR makes use of autoPatchelfHook to patch SDK artifacts. My solution instead wraps the flutter tool to set appropriate path environment variables. Here is my reasoning supporting this method:

• To my understanding, there's no guarantee that all the libraries used for Linux desktop support will be present in the SDK at the time of patching. At any point, the flutter tool may decide to download or update libraries, breaking compilation and execution.
  I think it will be more appropriate to build native binaries within Nixpkgs, rather than patch the binaries from the SDK. This is being done in flutter.engine: init #212328.
• For proper Linux desktop support, some environment variables such as PATH and PKG_CONFIG_PATH must be set. It makes sense to specify all the other build dependencies in the wrapper as well.

As for the differing wrapping methods, I have a function that builds a derivation composing a single modified flutter executable with an existing SDK directory.

• This allows arbitrary modifications to be made easily to the flutter executable, such as those made with makeWrapper or buildFHSUserEnv, in a composable manner. I use this to provide a raw "wrapped" version of the SDK, and then again after putting that wrapped executable in a FHS environment for compatibility with some of Flutter build tools. This ensures that the flutter executable is always inside a complete SDK, regardless of the environment at runtime.
• Modifying the flutter script directly, as I understand this PR does, is depending on factors we do not control, and is prone to breakage. What if the SDK later switches to a different scripting language, or a binary?

My PR also makes it very clear what is required only for Linux desktop support, and allows the feature to be disabled to reduce closure size.

I have last tested my PR with Flutter 3.7.3, and it works nicely. I am now attempting to add the new preloading functionality for support with 3.7.9+.

[algorithmiker]

@hacker1024 I am fine with handing this PR over to you. I have a lot of things going on right now and not really enough time to find out why the deps of yubioath are changing.
Please implement flutter 3.7.9 in your PR, and then we can close this one and AtaraxiaSjel's

[algorithmiker]

If you confirm that you are taking over this, I'll move on to doing reproducibility work instead (see #211606 )

[hacker1024]

Sure. I'm looking forward to fixing reproducibility, eager to see what you come up with on that front. I'll implement 3.7.10 on my branch in the next day or two.

[FlafyDev]

Hey
I'd like to clear up what this PR changes.

If I haven't missed anything, this PR updates Flutter as the title says. But it also updates the way Flutter gets its artifacts.

Usually, Flutter(in nixpkgs and in general) downloads all the artifacts at runtime when you first run flutter. It will also update them whenever it deems necessary.

This PR, on the other hand, downloads and packages these artifacts with the derivation and disables Flutter's ability to modify/update them.

I think this approach is preferable to downloading the artifacts at runtime for the following reasons:

• Packaging Flutter apps will be far more convenient since Flutter can't download different artifact versions to the vendor derivation.
• Flutter will attempt to download different artifacts on different versions. So if you regularly use different versions of Flutter from nixpkgs(because you are working on multiple projects or something) you'll have to wait for Flutter to redownload all of the artifacts when switching to a different version. (I am pretty sure this is a nix-only problem as the package redirects all of the artifacts to the same place: ~/.cache.) This PR fixes that.
• With the artifacts packaged and elfpatched, Flutter no longer needs an FHS environment (for building to Linux, building to Android will still require FHS)
• In Flutter's releases and packages of other distros, Flutter already comes with all the artifacts. Currently, the Flutter package in nixpkgs deletes them so Flutter can redownload them to ~/.cache.
• The artifacts should be a part of the Flutter package.

• To my understanding, there's no guarantee that all the libraries used for Linux desktop support will be present in the SDK at the time of patching. At any point, the flutter tool may decide to download or update libraries, breaking compilation and execution.

You are referencing two different things here.

There are libraries Flutter needs to be available on the system at runtime for building projects to different platforms. For example, you'll need cmake, ninja, gtk, etc... for building Flutter projects to Linux.

Then there are artifacts, which are files Flutter always needs at runtime and are usually bundled with it when getting it (Nixpkgs is the exception, as I've explained above). This PR makes it so artifacts are bundled with Flutter at its unwrapped derivation (IMO this is necessary. See my reasons above)

The artifacts are what Flutter attempts to download/update at runtime. And the former(libraries, like gtk that Flutter needs to build apps to Linux) are the ones not guaranteed to be available at runtime.

I think it will be more appropriate to build native binaries within Nixpkgs, rather than patch the binaries from the SDK. This is being done in flutter-engine: init #212328.

I don't know how more convenient it will be than just using prebuilts, but having the option to build yourself or using prebuilts from nixpkgs rather than Google will be nice, I guess.
This isn't related to the PR, though. Correct me if I'm wrong

---

I do think the suggestion in #201027 is a good idea. That is, having different wrappers applied depending on what you'll use Flutter for.

For example, currently in nixpkgs unwrapped Flutter is wrapped with many dependencies for building Flutter apps for Android, which is not ideal.

Additionally, after this PR Flutter will no longer require FHS for building to Linux. But FHS will still be needed when building for Android, so having that managed depending on the needs would be nice.

That being said, and if I'm not wrong, this PR isn't focused on how Flutter is wrapped, so I don't think it should interfere with #201027. Perhaps there should be some changes to this PR to minimize the difference in the wrapping code, though..

---

EDIT: To clarify, there are artifacts that are used only to bulid to android or linux. But there aren't artifacts that are optional, though. If you are on Linux, Flutter will automatically download artifacts for building to android if it can't find them.

I think flutter tried to download the artifacts based on the operating system. If used on linux, Flutter can only build apps for android, web and linux, and will thus download in advance only the artifacts for building to linux/android/web, and not for IOS for example. And it will complain(at least without any patches) if it can't get them.

[hacker1024]

Thanks for that breakdown, it's very clear.

I think the easiest thing to do at this point is to incorporate the artifact and patching changes into my branch, and then update the wrappers accordingly. I've already done a lot of the modularisation required to pull out Android and the FHS components.

As for Android, I've been meaning to look into it, but figured that was out of scope for the time being. I think the current Android solution is inappropriate - to my understanding, most of the added libraries are there for the functioning of the Android SDK, but this is beyond the scope of Flutter. It makes more sense to have another derivation to patch or wrap the SDK, as is done in android-nixpkgs.