Listen to cache event for managing metadata by artonge · Pull Request #34586 · nextcloud/server

artonge · 2022-10-13T15:48:42Z

NodeDeletedEvent is called when a file is put to the trash, and we want to remove metadata when the file is really deleted, so we now listen to CacheEntryRemovedEvent.

Also, a bit of refactoring to adapt code to this change.

artonge · 2022-10-17T17:19:56Z

I currently have this kind of errors when creating a new file: is locked, existing lock on file: exclusive need to check whether it is due to this PR or global.

lib/private/Metadata/FileEventListener.php

github-actions · 2022-10-18T16:39:59Z

Possible performance regression detected

Show Output

4 queries added

= /remote.php/dav/files/test
= /remote.php/dav/files/test/test.txt
= /remote.php/dav/files/test/many_files
≠ /remote.php/dav/files/test/new_file.txt with 5 queries added
  + SELECT "storage", "path", "mimetype" FROM "oc_filecache" WHERE "fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
≠ /remote.php/dav/files/test/new_file.txt with 1 queries removed
  - DELETE FROM "oc_file_metadata" WHERE "id" = :dcValue1

lib/private/Metadata/FileEventListener.php

github-actions · 2022-10-18T17:01:22Z

Possible performance regression detected

Show Output

4 queries added

= /remote.php/dav/files/test
= /remote.php/dav/files/test/test.txt
= /remote.php/dav/files/test/many_files
≠ /remote.php/dav/files/test/new_file.txt with 5 queries added
  + SELECT "storage", "path", "mimetype" FROM "oc_filecache" WHERE "fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
≠ /remote.php/dav/files/test/new_file.txt with 1 queries removed
  - DELETE FROM "oc_file_metadata" WHERE "id" = :dcValue1

CarlSchwan · 2022-10-19T09:12:22Z

The performance bot is finding suspicious things

github-actions · 2022-10-19T09:17:41Z

Possible performance regression detected

Show Output

4 queries added

= /remote.php/dav/files/test
= /remote.php/dav/files/test/test.txt
= /remote.php/dav/files/test/many_files
≠ /remote.php/dav/files/test/new_file.txt with 5 queries added
  + SELECT "storage", "path", "mimetype" FROM "oc_filecache" WHERE "fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("storage_id" = ?) AND ("user_id" = ?)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" WHERE "filecache"."fileid" = :dcValue1
≠ /remote.php/dav/files/test/new_file.txt with 1 queries removed
  - DELETE FROM "oc_file_metadata" WHERE "id" = :dcValue1

artonge · 2022-10-24T13:27:50Z

lib/private/Metadata/FileEventListener.php

 	private function isCorrectPath(string $path): bool {
 		// TODO make this more dynamic, we have the same issue in other places
-		return !str_starts_with($path, 'appdata_') && !str_starts_with($path, 'files_versions/') && !str_starts_with($path, 'files_trashbin/');
+		return !str_starts_with($path, 'appdata_') && !str_starts_with($path, 'files_versions/');


Review hint: Removed file_trashbin as we want to react on CacheEntryRemovedEvent even if the node is in the trash bin.

come-nc

What is the difference between OCP\Files\Events\NodeRemovedFromCache and OCP\Files\Cache\CacheEntryRemovedEvent ?

artonge · 2022-11-03T11:47:41Z

What is the difference between OCP\Files\Events\NodeRemovedFromCache and OCP\Files\Cache\CacheEntryRemovedEvent ?

NodeRemovedFromCache is emitted from the Scanner.php
CacheEntryRemovedEvent is emitted from the Cache.php

Apart from that, I don't really know ^^

Signed-off-by: Louis Chemineau <louis@chmn.me>

artonge · 2023-11-21T10:38:27Z

Superseded by #41634

artonge added bug 2. developing Work in progress labels Oct 13, 2022

artonge added this to the Nextcloud 26 milestone Oct 13, 2022

artonge self-assigned this Oct 13, 2022

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch from 860e4c2 to e579dfd Compare October 13, 2022 16:15

artonge marked this pull request as ready for review October 17, 2022 17:17

artonge requested a review from CarlSchwan as a code owner October 17, 2022 17:17

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch from e579dfd to 7f962a3 Compare October 17, 2022 17:17

artonge requested review from a team, PVince81 and skjnldsv and removed request for a team October 17, 2022 17:17

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch 2 times, most recently from e2a54a6 to f5638ca Compare October 17, 2022 17:21

github-advanced-security bot found potential problems Oct 17, 2022

View reviewed changes

lib/private/Metadata/FileEventListener.php Fixed Show fixed Hide fixed

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch 2 times, most recently from c3fd7c2 to 11b56d0 Compare October 18, 2022 15:58

github-advanced-security bot found potential problems Oct 18, 2022

View reviewed changes

lib/private/Metadata/FileEventListener.php Fixed Show fixed Hide fixed

lib/private/Metadata/FileEventListener.php Fixed Show fixed Hide fixed

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch from 11b56d0 to de007ae Compare October 19, 2022 09:09

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch 2 times, most recently from 8af96af to 5a4502c Compare October 24, 2022 13:16

artonge commented Oct 24, 2022

View reviewed changes

This was referenced Oct 25, 2022

Dispatch event for all removed entries #34773

Merged

Add CLI and listeners to reverse geocode pictures' coordinates nextcloud/photos#1310

Merged

PVince81 requested a review from icewind1991 October 25, 2022 15:30

PVince81 requested a review from come-nc October 25, 2022 15:30

come-nc reviewed Nov 3, 2022

View reviewed changes

blizzz mentioned this pull request Feb 1, 2023

26.0.0 beta 2 #36484

Merged

skjnldsv mentioned this pull request Feb 23, 2023

26.0.0 beta 5 #36834

Merged

blizzz mentioned this pull request Mar 7, 2023

26.0.0 RC2 #37078

Merged

blizzz modified the milestones: Nextcloud 26, Nextcloud 27 Mar 9, 2023

This was referenced May 3, 2023

27.0.0 beta 1 #38038

Merged

27.0.0 beta 2 #38137

Closed

27.0.0 beta 2 #38139

Merged

feat(changelog): group PR list by author in markdown nextcloud/github_helper#79

Merged

blizzz mentioned this pull request May 17, 2023

27.0.0 RC1 #38341

Merged

blizzz modified the milestones: Nextcloud 27, Nextcloud 28 May 23, 2023

skjnldsv mentioned this pull request Nov 1, 2023

chore: 28.0.0 beta 1 #41228

Merged

This was referenced Nov 6, 2023

28.0.0 beta 2 #41314

Merged

28.0.0 beta 3 #41379

Merged

This was referenced Nov 14, 2023

28.0.0 beta 4 #41477

Merged

28.0.0 RC1 #41621

Merged

Listen to cache event when deleting metadata

38f6463

Signed-off-by: Louis Chemineau <louis@chmn.me>

artonge force-pushed the artonge/fix/listen_to_cache_events_for_metadata branch from 5a4502c to 38f6463 Compare November 21, 2023 10:37

artonge requested review from ArtificialOwl and come-nc November 21, 2023 10:37

artonge closed this Nov 21, 2023

artonge mentioned this pull request Nov 21, 2023

Listen to cache event when deleting metadata #41634

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Listen to cache event for managing metadata#34586

Listen to cache event for managing metadata#34586
artonge wants to merge 1 commit intomasterfrom
artonge/fix/listen_to_cache_events_for_metadata

artonge commented Oct 13, 2022 •

edited

Loading

Uh oh!

artonge commented Oct 17, 2022

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

CarlSchwan commented Oct 19, 2022

Uh oh!

github-actions bot commented Oct 19, 2022

Uh oh!

artonge Oct 24, 2022 •

edited

Loading

Uh oh!

come-nc left a comment

Uh oh!

artonge commented Nov 3, 2022

Uh oh!

artonge commented Nov 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

artonge commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

artonge commented Oct 17, 2022

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

CarlSchwan commented Oct 19, 2022

Uh oh!

github-actions bot commented Oct 19, 2022

Uh oh!

artonge Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

come-nc left a comment

Choose a reason for hiding this comment

Uh oh!

artonge commented Nov 3, 2022

Uh oh!

artonge commented Nov 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

artonge commented Oct 13, 2022 •

edited

Loading

artonge Oct 24, 2022 •

edited

Loading