node-test-binary-windows CI jobs are failing with Git errors

[kfarnung]

e.g. https://ci.nodejs.org/job/node-test-binary-windows/16980/

Started by upstream project "node-test-binary-windows" build number 16980
originally caused by:
 Started by upstream project "node-test-commit-windows-fanned" build number 17667
 originally caused by:
  Started by upstream project "node-test-commit" build number 18165
  originally caused by:
   Started by upstream project "node-test-pull-request" build number 14614
   originally caused by:
    Started by user Anna Henningsen
[EnvInject] - Loading node environment variables.
Building remotely on test-azure_msft-win2016-x64-5 (win2016-1p-vs2015 win2016 win2016-vs2015 win2016-1p) in workspace c:\workspace\node-test-binary-windows
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.jenkins_tmp.url binary_tmp@147.75.70.237:binary_tmp.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
ERROR: Error fetching remote repo 'jenkins_tmp'
hudson.plugins.git.GitException: Failed to fetch from binary_tmp@147.75.70.237:binary_tmp.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1727)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout: 
stderr: fatal: Unable to create 'c:/workspace/node-test-binary-windows/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:463)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:786)
	at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)
	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:927)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:901)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:850)
	at hudson.remoting.UserRequest.perform(UserRequest.java:210)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:364)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Unknown Source)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 13.64.192.187/13.64.192.187:1177
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
		at hudson.remoting.Channel.call(Channel.java:952)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:278)
		at com.sun.proxy.$Proxy74.clean(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:450)
		at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:30)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:858)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1727)
		at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'jenkins_tmp'
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE

[kfarnung]

@joaocgreis Any ideas?

[MSLaguana]

Looks like the complaint is that there is a pre-existing .git/index.lock file, possibly from a previous run which was interrupted. I suspect that wiping out the folder / that file might cause things to work again.

[joaocgreis]

This happens when a job is aborted while git is running. In this case it was caused by https://ci.nodejs.org/job/node-test-binary-windows/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=2/16962/console .

I started git-clean-windows, the next runs should be ok.

Thanks @kfarnung !

[BridgeAR]

@joaocgreis I personally only see this as a intermediate step. We are likely running into this issue over and over again and I wonder if there is no possibility to automate a fix for this.

I think it would be great to do the following:

1. Run the build. If it works -> perfect. If it does not, go to 2.
2. Check the error if there was a former .git/index.lock file.
3. Run git-clean-windows in that case.
4. Start the worker again.

I can imagine that there are more efficient ways of doing this but it was just one way that came to my mind right away.

[joaocgreis]

@BridgeAR this is only one of the issues we have with the Git plugin in Jenkins (ref #1122). In this case, the job was aborted while the plugin was running, and from then on only the Git plugin runs and throws an exception, giving us no chance of running anything before or after.

We could run something as a post-job of node-test-commit (I actually started git-clean-conditional with this in mind). However, this job would have to be enabled for every worker individually and that would create a lot of backlog in Jenkins whenever a worker goes offline. Jenkins works really bad with backlog, I believe this would create more problems than it solves.

cc @nodejs/build in case anyone has a better idea or is leaning towards giving git-clean-conditional a try.

[BridgeAR]

This is just happening really often on the Windows machines. E.g. https://ci.nodejs.org/job/node-test-binary-windows/17150/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=2/console
https://ci.nodejs.org/job/node-test-binary-windows/17150/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=3/console