Beanie improvements (id, mongoclient, file calls) #504
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
longshuicy
reviewed
May 16, 2023
| extractor_name, extractor_out.id, process | ||
| ) | ||
| db["feeds"].insert_one(processed_feed) | ||
| processed_feed.insert() |
Member
There was a problem hiding this comment.
wonder if we need await here?
Contributor
Author
There was a problem hiding this comment.
good catch, added this and a few other missing awaits also
max-zilla
added a commit
that referenced
this pull request
Jun 13, 2023
* Setup Beanie and use it to create new dataset. * Use Beanie to query MongoDB view. FIXME endpoints return MongoDB `_id` but javascripts expects `id`. * Temporary workaround to map from `_id` returned by API to `id` expected by javascript. * Simple cache example (needs fix). * Testing migrations. Doesn't work yet. Mongo URI throws error regarding no database name provided. * Use beanie to retrieve specific dataset. * Fixed all AuthorizationDB calls to use Beanie. * Updated condegen. * 491 beanie for metadata, dataset_metadata (#493) * import container (#462) * 446 gui to manage api keys (#465) * add new routes placeholder * add list and delete endpoint; inline docs * codegen and pytest * ugly ui? * onstart clear user keys * delete works but pytest doesn't work yet * pytest all works * codegen * redux * reducer and types * write the table * correctly wire in the apikey and error handling * wire in the delete modal of apikeys * wire in the creation api key * everything working in the gui * formatting --------- Co-authored-by: toddn <[email protected]> * Sharing tab cleanup (#430) * fix bug where users not being fetched * simplifying role query pt 1 * refactor tables and API calls * cleanup * start moving GroupAndRole to separate table * add expanding sub table * remove AddGroup buttons for now * formatting * codegen * add response model * codegen * remove container * fix bug with Roles return object and autocomplete on groups * codegen * Fix reload bug (#451) * 461 fix library version in pipfile (#468) * fix pipfile version * regenerate the piplock file * 443 frontend need to display more verbose backend error (#458) * Updated frontend error message to be more verbose * Backend returns verbose logs, frontend displays it as generic messages depending on the HTTP status code * Updated the error message logic to include original backend logs wherever possible * Fixed report button issue * Rootcause: The onClick event for the report button has a semantic error, which is causing the issue reason string to be replaced with an object * Also added encode for the reason * Fixing issue with error message * Revert "Updated frontend error message to be more verbose" This reverts commit 7d230d3. * Fixing issue with error message * 469 create profile page (#471) * profile page exists * new route for user profile profile page includes layout * codegen new route for get current user profile * we now get the profile, but nothing shows yet * page has fake data * we see the profile now * formatting * Implemented role delete in sharing tab (#472) * Tested by deleting a user, group. Verified entry is removed upon refresh * Show Creator on Group Page (#428) * the owner is visible, and we cannot change the owner in the table * showing group creator on top * creator link is wrong, need to fix * bold text for word 'creator' * using more conditionals to reduce duplicate code * missing modal added creator name matches other names --------- Co-authored-by: Chen Wang <[email protected]> * Fixed bug where error detail might be empty (#478) * Fixed bug where error detail might be empty * Added null check to the detail string to prevent errors when detail is not set * Second check to validate the body field * authorization deps * update models * refactoring the metadata route * include in main * mongoout pattern * black * see if this new query pattern works * metadata datasets * finish metadata dataset and metadata file * remove _beanie database * codegen * new pattern of matching id * fix typo * get/post works * more fixes * fix deleting endpoint * fix patch endpoint * fix put metadata * try max's style to minimize merge conflict --------- Co-authored-by: toddn <[email protected]> Co-authored-by: Max Burnette <[email protected]> Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * Beanie - Listeners, Metadata (#495) * import container (#462) * 446 gui to manage api keys (#465) * add new routes placeholder * add list and delete endpoint; inline docs * codegen and pytest * ugly ui? * onstart clear user keys * delete works but pytest doesn't work yet * pytest all works * codegen * redux * reducer and types * write the table * correctly wire in the apikey and error handling * wire in the delete modal of apikeys * wire in the creation api key * everything working in the gui * formatting --------- Co-authored-by: toddn <[email protected]> * Sharing tab cleanup (#430) * fix bug where users not being fetched * simplifying role query pt 1 * refactor tables and API calls * cleanup * start moving GroupAndRole to separate table * add expanding sub table * remove AddGroup buttons for now * formatting * codegen * add response model * codegen * remove container * fix bug with Roles return object and autocomplete on groups * codegen * Fix reload bug (#451) * 461 fix library version in pipfile (#468) * fix pipfile version * regenerate the piplock file * 443 frontend need to display more verbose backend error (#458) * Updated frontend error message to be more verbose * Backend returns verbose logs, frontend displays it as generic messages depending on the HTTP status code * Updated the error message logic to include original backend logs wherever possible * Fixed report button issue * Rootcause: The onClick event for the report button has a semantic error, which is causing the issue reason string to be replaced with an object * Also added encode for the reason * Fixing issue with error message * Revert "Updated frontend error message to be more verbose" This reverts commit 7d230d3. * Fixing issue with error message * 469 create profile page (#471) * profile page exists * new route for user profile profile page includes layout * codegen new route for get current user profile * we now get the profile, but nothing shows yet * page has fake data * we see the profile now * formatting * Implemented role delete in sharing tab (#472) * Tested by deleting a user, group. Verified entry is removed upon refresh * Show Creator on Group Page (#428) * the owner is visible, and we cannot change the owner in the table * showing group creator on top * creator link is wrong, need to fix * bold text for word 'creator' * using more conditionals to reduce duplicate code * missing modal added creator name matches other names --------- Co-authored-by: Chen Wang <[email protected]> * Fixed bug where error detail might be empty (#478) * Fixed bug where error detail might be empty * Added null check to the detail string to prevent errors when detail is not set * Second check to validate the body field * filter the option without group owner * filter group owner out * Made the updaterole function async, and added a call to fetch roles to refresh the list of roles in state * Updated remove role logic to automatically refresh the roles list * Executed eslint * stub for feeds * add basic Feed support * various updates * introduce listener views * update config model * More updates to metadata models * Update datasets.py * update metadata usage * formatting * replace db["datasets"] * Update metadata.py * ObjectID & response cleanups * syntax fix * author -> creator * formatting * consistent syntax * fix init of listener views --------- Co-authored-by: Chen Wang <[email protected]> Co-authored-by: toddn <[email protected]> Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * move provenance, formatting * 492 beanie tokens users groups (#494) * users fixed to use beanie * methods throw exceptions groups * token uses BaseModel and Document UserDB used in authentication routes * user_out uses dict * checking against Max's branch to avoid conflicts * matching example from Chen on new db get and find * black formatting * rename methods * returning None, not sure why * routes tested for users still working on groups * fixing groups need to use different replace * fix key save * more key deletion * fixing replace * consistent syntax, regex fixes * simplify syntax --------- Co-authored-by: Max Burnette <[email protected]> * 491 beanie for file and folder (#496) * import container (#462) * 446 gui to manage api keys (#465) * add new routes placeholder * add list and delete endpoint; inline docs * codegen and pytest * ugly ui? * onstart clear user keys * delete works but pytest doesn't work yet * pytest all works * codegen * redux * reducer and types * write the table * correctly wire in the apikey and error handling * wire in the delete modal of apikeys * wire in the creation api key * everything working in the gui * formatting --------- Co-authored-by: toddn <[email protected]> * Sharing tab cleanup (#430) * fix bug where users not being fetched * simplifying role query pt 1 * refactor tables and API calls * cleanup * start moving GroupAndRole to separate table * add expanding sub table * remove AddGroup buttons for now * formatting * codegen * add response model * codegen * remove container * fix bug with Roles return object and autocomplete on groups * codegen * Fix reload bug (#451) * 461 fix library version in pipfile (#468) * fix pipfile version * regenerate the piplock file * 443 frontend need to display more verbose backend error (#458) * Updated frontend error message to be more verbose * Backend returns verbose logs, frontend displays it as generic messages depending on the HTTP status code * Updated the error message logic to include original backend logs wherever possible * Fixed report button issue * Rootcause: The onClick event for the report button has a semantic error, which is causing the issue reason string to be replaced with an object * Also added encode for the reason * Fixing issue with error message * Revert "Updated frontend error message to be more verbose" This reverts commit 7d230d3. * Fixing issue with error message * 469 create profile page (#471) * profile page exists * new route for user profile profile page includes layout * codegen new route for get current user profile * we now get the profile, but nothing shows yet * page has fake data * we see the profile now * formatting * Implemented role delete in sharing tab (#472) * Tested by deleting a user, group. Verified entry is removed upon refresh * Show Creator on Group Page (#428) * the owner is visible, and we cannot change the owner in the table * showing group creator on top * creator link is wrong, need to fix * bold text for word 'creator' * using more conditionals to reduce duplicate code * missing modal added creator name matches other names --------- Co-authored-by: Chen Wang <[email protected]> * Fixed bug where error detail might be empty (#478) * Fixed bug where error detail might be empty * Added null check to the detail string to prevent errors when detail is not set * Second check to validate the body field * filter the option without group owner * filter group owner out * Made the updaterole function async, and added a call to fetch roles to refresh the list of roles in state * Updated remove role logic to automatically refresh the roles list * Executed eslint * remove _beanie * wire in folder db * download folder * refactored everything in file/folder * reformat * missed a spot * codegen * forget to add FileVersionDB * add await * fix more merge conflict * duplicate * fix inc * black fromat * codegen * fix inconsistency * black * fix frontend author/creator * fix detail * fix some of the async bug * download works * init & use TokenDB everywhere * fix files/submit endpoint * remove manual index creation * fix extractor submission --------- Co-authored-by: toddn <[email protected]> Co-authored-by: Max Burnette <[email protected]> Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * codegen * fixing mongomodel (#501) * fixing mongomodel * deleting unused mongomodel * Simple example of return `id` instead of `_id` using beanie. (#502) * UserDB fix and Frontend revert id (#503) * revert id * remove unused import * codegen * fix user token * remove all the mentioning of db * fix more inconsistencies * fix more to_mongo * add error db model * remove all the to mongo * force list of dataset out object to have .dict so id can show correctly * black formatting --------- Co-authored-by: Max Burnette <[email protected]> * Beanie improvements (id, mongoclient, file calls) (#504) * remove PydanticObjectId * begin converting IDs * major updates to db calls, id support * replace Outs with dict returns * add missing awaits * Beanie fix tests (#505) * fix api key and dataset tests * test fixes for extractors * fix job view definitions * minor fixes * fix files, folders, listener tests * fix remaining tests * indexing methods use beanie * fix tests * Removed duplicate code due to bad merge. * Fixed get_file_versions route. * Using asyc for loop for querying file versions. Note that we drop the to_list() in this case. Based on Beanie docs. * Use PydanticObjectId in the route definition instead of string. * Ran codegen. * Cast group to GroupOut when GET /groups. * for loop formatting * Update listener services (#509) * make heartbeat listener async * update heartbeat listener * update message_listener to async * fix typo * update Dockerfiles * update Dockerfiles * Use dedicated test DB (#514) * make heartbeat listener async * update heartbeat listener * update message_listener to async * fix typo * update Dockerfiles * update Dockerfiles * WIP * clean up dockerfiles * remove unused config * revert config to localhost * update RabbitMQ connection info for services * revert localhost IP change for docker * Update package-lock.json * Fix to file update, download and delete (#533) * add ListenerKey to main * Metadata & search bugfixes (#534) * various metadata fixes * try/except on ES update calls * Fix search, remove console logs * use insert over save * properly await listener insert * codegen * syntax change to query (#545) * inheritance fix * codegen * remove print statement * bugfix on messages service * improve message listener logic * remove comments * replace user calls * fix file view on datasets to include bytes, type --------- Co-authored-by: Chen Wang <[email protected]> Co-authored-by: toddn <[email protected]> Co-authored-by: Max Burnette <[email protected]> Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> Co-authored-by: Dipannita <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Many aspects are untested, but wanted to get these fixes in.