Skip to content

Cache clean up might fail on Windows #20584

@damccorm

Description

@damccorm

Details can be found in this PR:
#12779


apache_beam\runners\interactive\recording_manager_test.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam\runners\interactive\interactive_environment.py:118:
in new_env
    _interactive_beam_env.cleanup()
apache_beam\runners\interactive\interactive_environment.py:272:
in cleanup
    cache_manager.cleanup()
apache_beam\runners\interactive\caching\streaming_cache.py:391:
in cleanup
    shutil.rmtree(self._cache_dir)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500:
in rmtree
    return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390:
in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395:
in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

...

>                   os.unlink(fullname)
E
                  PermissionError: [WinError 32] The process cannot access the file because it is being
used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'

This does not happen on Linux-like systems:
https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png

A potential solution is to enable ignore_errors and log about manually cleaning up onerror.

Imported from Jira BEAM-11339. Original Jira may contain additional context.
Reported by: ningk.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions