Reviews: Ideas to speed up the Review

[dwertheimer]

/Project List takes a long time to gather all the project files. It's painful any time I want to do it and actually discourages me from doing the reviews.

The speed problem is that it's looking in 100% of my notes (I understand why).

Some ideas (@jgclark you will have better ones):

1. It's fantastic that you can ignore certain folders and you can ignore the top most folder name in the output, but I'm wondering if maybe there should/could be a "Project Root" folder setting so that you could limit the scan to just that root folder (if set). I've organized it so my projects are all in a /Projects folder. That may or may not be the best way/place to put them, but it's working for now. Looking just in this folder would help me.
2. What makes the scan slow is that it has to look in the content of each document looking for hashtags. But DataStore itself is not slow (it's cached). So maybe another setting could be a "Project Note Titles Must Contain" setting, and if set, it only looks in notes that have a word (e.g. "Project") in the title. That way, you could filter DataStore down to just those files before scanning contents.

Thoughts?

[dwertheimer]

@jgclark Thinking more about idea #1, if I as a user say that my projects are always in a "Projects/*" folder, then it would not just need to be one Projects folder at the root but could be:
ClientA/Projects/FooProject
ClientB/Future/Something/Projects/BarProject
etc.
Would still be way faster

[jgclark]

I agree its slow, and slower than expected given the caching of hashtags. I certainly found an issue with .hashtags being stale, so perhaps I switched to do a full search? Next week I'll try to do some instrumenting to see where the most time goes.

And what about this option instead?
3. New setting with 'Folder(s) to find Project notes', to suit the wider case (that includes myself) where I have several top-level folders with Projects in. This would take any sub-folders too. The existing 'Folders to Ignore' setting would still apply and could defeat any sub-folders if necessary.

[dwertheimer]

@jgclark Just loaded the latest version and even with the "include" folder, it's taking 3s to scan for my 9 project notes.

I had a look at the code and see why. Even though the filter is correctly filtering down the notes, that filtered set is not being passed to the hashtag checker, and the hashtag checker is reading the entire DataStore for each hashtag.

[dwertheimer]

Also @jgclark , can I ask that the new folder include filter does not just filter for root level folders but for any folder in the path that matches the include?

See my note here:
#442 (comment)

[dwertheimer]

Also @jgclark, I think if you do the includes before the excludes, you will get some marginal speed benefit. Something like:

    const fullFolderList = DataStore.folders
    let reducedList: Array<string> = fullFolderList
    
... slash termination code here ...

    // filter reducedList to only folders that start with an item in the inclusionsTermintedWithSlash list
    reducedList = inclusionsTerminatedWithSlash.length ? reducedList.filter((folder) =>  inclusionsTerminatedWithSlash.some((ff) => folder.startsWith(ff))) : reducedList

    // filter reducedList to only folders that don't start with an item in the exclusionsTerminatedWithSlash list
    reducedList = inclusionsTerminatedWithSlash.length ? reducedList.filter((folder) =>  !exclusionsTerminatedWithSlash.some((ee) => folder.startsWith(ee))) : reducedList

    // filter reducedList to only folders that don't start with the character '@' (special folders)
    reducedList = excludeSpecialFolders ? reducedList.filter((folder) => !folder.startsWith('@')) : reducedList

    return reducedList

This is basically a slimmed down version of folders.js::getFilteredFolderList(). But as I said above, I would prefer to see the inclusions not be a startsWith but rather an includes, e.g.:

reducedList.filter((folder) =>  inclusionsTerminatedWithSlash.some((ff) => folder.includes(ff))) : reducedList

[dwertheimer]

@jgclark I edited the code a little above, so pls look at the version on Github (not email)

[jgclark]

Many thanks for all this. I will try and make a new release tomorrow. J

[jgclark]

Done, thanks.
Though the findNotesMatchingHashtag() function hadn't actually been "reading the entire DataStore for each hashtag" AFAICT. So the saving might not be as much as you expect.
I've changed the Include setting to be 'includes' not 'startsWith' for you.

[dwertheimer]

Closing this issue with PR: #444