Context matches v1 type. #328

tcnichol · 2023-02-15T23:27:13Z

In clowder v1 context was a list that consisted in URL and dictionary. This pull request makes v2 match v1. Front end and back end have both been changed. Extractor registration should work correctly.

If testing, make sure to remove old metadata/listeners from DB, or else conflicts will happen.

…ake v2 compatible with some extractors from extractors-core which dynamically build the context.

fix test

longshuicy · 2023-02-16T14:58:49Z

Can you fix the linting?

tcnichol · 2023-02-16T15:38:51Z

To test with extractors, use this branch of pyclowder:

clowder-framework/pyclowder#60

Wordcount and other sample extractors there can be used.

tcnichol · 2023-02-16T15:42:14Z

For the errors in pytest. These are happening when metadata with no context_url is being posted to the backend for files and datasets. The field is an optional[str] but the validator looks like it throws an error if it is missing. Not sure what to do to fix it. Also that the changes made in this PR shouldn't affect that.

changed context from optional[list] to list with default value empty, but still is not fixing the issues.

tcnichol · 2023-02-21T18:53:39Z

@ddey2 I pushed a small change that might fix the errors you are seeing.

ddey2

Looks good, tested.

max-zilla

everything seemed to work, remove the log and i will merge after tests pass thanks

max-zilla · 2023-02-28T14:27:15Z

frontend/src/components/metadata/CreateMetadataDefinition.tsx

        })

+		console.log([JSON.stringify(contextJson)]);
+


please remove

* context is now changed to list union of AnyUrl or dict. This should make v2 compatible with some extractors from extractors-core which dynamically build the context. * formatting fix test * formatting * still getting pytest errors changed context from optional[list] to list with default value empty, but still is not fixing the issues. * should fix the tests * formatting * getting rid of class 'context element' not needed * getting rid of class 'context element' not needed * removing console log --------- Co-authored-by: Chen Wang <[email protected]>

* Added group_id to authorization * Added list of group_ids instead of one group_id per entry * Addressing comments * fixing lint failure * Added new component for file actions (#352) * Added new component for file actions * Tested file download and file delete from File page * Verified after deleting file, page navigates back to main dataset page * Fixed version-not-updating error * Updated onSave sequence to first synchronously update the file and then subsequently call listVersions API to display on the frontend * Fixed lint issues by running eslint * UI for displaying logs on extractors (#317) * Static UI for displaying logs on extractors * Created 2 React custom components: * ExtractorStatus -- UI table to display summary of job/progress so far * ExtractorLogs -- terminal-like UI to display logs in real-time as they are being fetched * [WIP] Enable extractor job id retrieval from backend * Updated redux store value when job id is returned * Added actions for job update, job summary API endpoints * [WIP DO NOT MERGE] Testing frontend functionality with dynamic log rendering * Updated API endpoints, return payload, reducers and actions (#330) * Updated API endpoints, return payload, reducers and actions for the extractor jobs * Tested API calls to fetchJobSummary, fetchJobUpdates from frontend, able to receive extractor job data * Tested submit job process flow, able to retrieve extractor job id from the backend * Removed redundant code --------- Co-authored-by: Chen Wang <[email protected]> * Integrated UI with backend * Tested with sample extractor logs, able to view both extractor status and summary from the UI * Removed hardcoding of job id * Fixed indentation * Adding download button on UI to download correct verison file (#334) * Adding download button on UI to download correct verison file * fix filename issue * Update FileVersion.ts --------- Co-authored-by: Chen Wang <[email protected]> * fix member typing (#336) * fix member typing * adjust the test so it can pass * fixing error of blank page on submit file to extractor (#338) files.py - renaming resubmit method codegen fixing error in listeners.py * Modified extractor job summary duration calculation * Moving fetchJobSummary, fetchJobSummary API calls to ExtractorStatus component to fix interval issue --------- Co-authored-by: Chen Wang <[email protected]> Co-authored-by: Dipannita <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * Properly interpret extractor statuses (#356) * various fixes to message listener * black formatting * Added new component for file actions (#352) * Added new component for file actions * Tested file download and file delete from File page * Verified after deleting file, page navigates back to main dataset page * Fixed version-not-updating error * Updated onSave sequence to first synchronously update the file and then subsequently call listVersions API to display on the frontend * Fixed lint issues by running eslint * UI for displaying logs on extractors (#317) * Static UI for displaying logs on extractors * Created 2 React custom components: * ExtractorStatus -- UI table to display summary of job/progress so far * ExtractorLogs -- terminal-like UI to display logs in real-time as they are being fetched * [WIP] Enable extractor job id retrieval from backend * Updated redux store value when job id is returned * Added actions for job update, job summary API endpoints * [WIP DO NOT MERGE] Testing frontend functionality with dynamic log rendering * Updated API endpoints, return payload, reducers and actions (#330) * Updated API endpoints, return payload, reducers and actions for the extractor jobs * Tested API calls to fetchJobSummary, fetchJobUpdates from frontend, able to receive extractor job data * Tested submit job process flow, able to retrieve extractor job id from the backend * Removed redundant code --------- Co-authored-by: Chen Wang <[email protected]> * Integrated UI with backend * Tested with sample extractor logs, able to view both extractor status and summary from the UI * Removed hardcoding of job id * Fixed indentation * Adding download button on UI to download correct verison file (#334) * Adding download button on UI to download correct verison file * fix filename issue * Update FileVersion.ts --------- Co-authored-by: Chen Wang <[email protected]> * fix member typing (#336) * fix member typing * adjust the test so it can pass * fixing error of blank page on submit file to extractor (#338) files.py - renaming resubmit method codegen fixing error in listeners.py * Modified extractor job summary duration calculation * Moving fetchJobSummary, fetchJobSummary API calls to ExtractorStatus component to fix interval issue --------- Co-authored-by: Chen Wang <[email protected]> Co-authored-by: Dipannita <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * Millisecond (#360) * it's seconds * add more icons * default to open tab --------- Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Chen Wang <[email protected]> Co-authored-by: Dipannita <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> * Context matches v1 type. (#328) * context is now changed to list union of AnyUrl or dict. This should make v2 compatible with some extractors from extractors-core which dynamically build the context. * formatting fix test * formatting * still getting pytest errors changed context from optional[list] to list with default value empty, but still is not fixing the issues. * should fix the tests * formatting * getting rid of class 'context element' not needed * getting rid of class 'context element' not needed * removing console log --------- Co-authored-by: Chen Wang <[email protected]> * Mongo views (#353) * add a readme * add listener job update view * temp * add init * add to production docker-compose as well * 343 list resources that user has access (#355) * add logic to filter user * add filters to file * list files working too * add same filter for folder * include executions * exempt the current resource owner * add test database * rewrite createView part * rewrite init script but it's still not working yet * fix typo * fix linting * Replaced EmbeddedSearch with custom search box component to fix session refresh issue (#365) * Tested dataset search by clicking button and pressing enter, able to view search results * match user_ids which is a list now (#368) * update to lookup query * update authorization_deps queries * simplify group logic in auth check * formatting * Formatting --------- Co-authored-by: Aruna Parameswaran <[email protected]> Co-authored-by: Chen Wang <[email protected]> Co-authored-by: Todd Nicholson <[email protected]> Co-authored-by: Max Burnette <[email protected]>

context is now changed to list union of AnyUrl or dict. This should m…

20920bd

…ake v2 compatible with some extractors from extractors-core which dynamically build the context.

tcnichol requested review from longshuicy and max-zilla as code owners February 15, 2023 23:27

tcnichol linked an issue Feb 15, 2023 that may be closed by this pull request

metadata context is list of dict or url #306

Closed

tcnichol mentioned this pull request Feb 15, 2023

59 change context type for v2 clowder-framework/pyclowder#60

Merged

formatting

6b54a77

fix test

formatting

25d315b

longshuicy and others added 6 commits February 16, 2023 16:26

Merge branch 'main' into 306-metadata-context-is-list-of-dict-or-url

3845c49

Merge branch 'main' into 306-metadata-context-is-list-of-dict-or-url

0409e3b

Merge branch 'main' into 306-metadata-context-is-list-of-dict-or-url

456f4c5

still getting pytest errors

3dd6504

changed context from optional[list] to list with default value empty, but still is not fixing the issues.

should fix the tests

729a501

formatting

e94e2d2

ddey2 self-requested a review February 20, 2023 21:36

tcnichol added 2 commits February 21, 2023 12:51

getting rid of class 'context element' not needed

31a1db1

getting rid of class 'context element' not needed

9e09651

ddey2 approved these changes Feb 21, 2023

View reviewed changes

max-zilla requested changes Feb 28, 2023

View reviewed changes

removing console log

7e55872

tcnichol requested a review from max-zilla February 28, 2023 19:41

max-zilla requested a review from ddey2 March 1, 2023 19:15

max-zilla approved these changes Mar 1, 2023

View reviewed changes

max-zilla merged commit f9fd810 into main Mar 1, 2023

max-zilla deleted the 306-metadata-context-is-list-of-dict-or-url branch March 1, 2023 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Context matches v1 type. #328

Context matches v1 type. #328

Uh oh!

tcnichol commented Feb 15, 2023

Uh oh!

longshuicy commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 21, 2023

Uh oh!

ddey2 left a comment

Uh oh!

max-zilla left a comment

Uh oh!

max-zilla Feb 28, 2023

Uh oh!

tcnichol Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Context matches v1 type. #328

Context matches v1 type. #328

Uh oh!

Conversation

tcnichol commented Feb 15, 2023

Uh oh!

longshuicy commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 16, 2023

Uh oh!

tcnichol commented Feb 21, 2023

Uh oh!

ddey2 left a comment

Choose a reason for hiding this comment

Uh oh!

max-zilla left a comment

Choose a reason for hiding this comment

Uh oh!

max-zilla Feb 28, 2023

Choose a reason for hiding this comment

Uh oh!

tcnichol Feb 28, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants