127 add parameters to extractor submit add submit dataset #128
Conversation
working on adding submit dataset to extractor
tcnichol
requested review from
lmarini,
longshuicy and
max-zilla
as code owners
lmarini
left a comment
lmarini
left a comment
There was a problem hiding this comment.
Can you add comments, doc strings, and pytest? Thank you.
|
Will do. |
also adding a sample extractor_info.json file for registration purposes
|
Testing is implemented and build errors there fixed. |
…-extractor-submit-add-submit-dataset # Conflicts: # backend/app/routers/datasets.py
beginning of adding listener/extractor to front end
|
Parent branch merged into main. Please update to resolve conflicts. Thank you. |
…-extractor-submit-add-submit-dataset # Conflicts: # backend/app/routers/datasets.py
…-extractor-submit-add-submit-dataset # Conflicts: # backend/app/models/extractors.py # backend/app/models/feeds.py # backend/app/models/listeners.py # backend/app/models/search.py # backend/app/rabbitmq/listeners.py # backend/app/routers/feeds.py # backend/app/routers/files.py # backend/app/routers/listeners.py
does this fix registration?
|
I'm getting errors with the pytest because the extractor_info has an author field (a string) but Listeners have an Author that is UserOut. Could we perhaps use a different term (maybe 'Creator') for the Listener, so as not to conflict with the field in the existing extractor_info.json files? |
…fields. Making Optional in case not supplied (registration through rabbitmq heartbeat monitor instead of directly by user)
|
New change - to make Listeners work when being registered through the heartbeat listener I changed 'author' to 'creator' but also made creator optional. This seemed like the best solution. It also seems to work if Listeners are built in top of Extractors, like someone wants to fine tune when an existing extractor runs. This one should be ready for review again. Let me know if anything else needs to be changed. |
max-zilla
deleted the
127-add-parameters-to-extractor-submit-add-submit-dataset
branch
This pull request does the following:
It is now possible to submit datasets to extractors.
Parameters can be added for either file or dataset extractors.
This relies on a pull request from pyclowder :
clowder-framework/pyclowder#51
That branch has changes to make it work with the new API. If extractors are run using a different pyclowder, they will not work.