Potential deobf Concurrency Issue

[shartte]

We sometimes see ForgeGradle builds fail on our CI with the following stack trace, when running the "runData" goal:

> Task :runData
Error getting artifact: mezz.jei:jei-1.15.2:6.0.2.12_mapped_snapshot_20200715-1.15.1:null@jar from  DeobfuscatingRepo
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:230)
	at java.util.zip.ZipFile.<init>(ZipFile.java:160)
	at java.util.jar.JarFile.<init>(JarFile.java:168)
	at java.util.jar.JarFile.<init>(JarFile.java:132)
	at net.minecraftforge.gradle.common.task.JarExec.apply(JarExec.java:61)
	at net.minecraftforge.gradle.userdev.util.Deobfuscator.deobfBinary(Deobfuscator.java:130)
	at net.minecraftforge.gradle.userdev.util.DeobfuscatingRepo.findRaw(DeobfuscatingRepo.java:125)
	at net.minecraftforge.gradle.userdev.util.DeobfuscatingRepo.findFile(DeobfuscatingRepo.java:81)
	at net.minecraftforge.gradle.common.util.BaseRepo.getArtifact(BaseRepo.java:81)
	at net.minecraftforge.gradle.common.util.BaseRepo.getArtifact(BaseRepo.java:41)
	at net.minecraftforge.gradle.common.util.BaseRepo$Builder$1.lambda$getArtifact$0(BaseRepo.java:127)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
	at net.minecraftforge.gradle.common.util.BaseRepo$Builder$1.getArtifact(BaseRepo.java:127)
	at net.minecraftforge.gradle.common.util.BaseRepo$Builder$1.getArtifact(BaseRepo.java:124)
[...]

We have two dependencies using fg.deobf:

    // Runtime, Mods
    runtimeOnly fg.deobf("mezz.jei:jei-${minecraft_version}:${jei_version}")
    runtimeOnly fg.deobf("mcjty.theoneprobe:TheOneProbe-${minecraft_release}:${minecraft_release}-${top_version}")

When debugging this on my machine, I noticed the following:

• Gradle will resolve these dependencies concurrently (Build operations threadpool)
• If the installertools fatjar is not present locally, multiple threads may start downloading it concurrently
• There seems to be no synchronization to prevent this

My current hypothesis is: Two threads start downloading the same jar file into the same location (maven_downloader cache), Jar is corrupted -> boom.

Potential solutions I see: nuclear option is synchronizing the methods in MavenArtifactDownloader. Alternatively, the existing field CACHE could be converted into a LoadingCache. I think Guava will take of the underlying concurrency, but I'd have to check that first.

p.s.: Alternatively we could exclude these mods from the runData task (which we would prefer); but we have not found a way to do that.

[shartte]

We are currently testing the workaround of setting --max-workers 1 in our CI. Although that increases build times, we hope it'll make them stable until this can be adressed.

[Jonathing]

I get the same issue when running runServer on CI. We run a temporary server to load a world and then crash it after it's done loading to see if it worked. Sometimes it will fail processing dependencies or downloading data. If I can find a workaround for this.

Using GitHub Actions CI btw.

[williewillus]

Adding --max-workers 1 fixed this issue in GH Actions for us

[Jonathing]

Adding --max-workers 1 fixed this issue in GH Actions for us

Will add that now. If I somehow get an error sometime in the future I'll come back here lol

[robotgryphon]

Seeing this issue as well during automated testing:
https://github.com/CompactMods/CompactMachines/runs/2355825618?check_suite_focus=true