Initial implementation of metadata indexing #173
Conversation
ddey2
commented
Nov 1, 2022
ddey2
commented
- metadata indexing
- also updated syntax of elasticsearch dependency injection
…or elasticsearch dependency injection
| } | ||
|
|
||
| metadata_mappings = {} | ||
| # "properties": { |
There was a problem hiding this comment.
I removed the mappings because I noticed when the mapping are there for static fields, it somehow creating duplicate fields under a field "doc". So it looked like below:
{
"metadata": {
"aliases": {},
"mappings": {
"properties": {
"contents": {
"type": "object"
},
"context": {
"type": "text"
},
"context_url": {
"type": "text"
},
"created": {
"type": "date"
},
"creator": {
"type": "keyword"
},
"doc": {
"properties": {
"contents": {
"properties": {
"alternateName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
}
}
},
"context_url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"created": {
"type": "date"
},
"creator": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"reource_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"resource_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"resource_id": {
"type": "text"
},
"resource_type": {
"type": "text"
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "5",
"provided_name": "metadata",
"creation_date": "1667489155177",
"number_of_replicas": "5",
"uuid": "g5MQr--BQB6q2B8boKZWkA",
"version": {
"created": "8030399"
}
}
}
}
}
It made the search complicated. I removed the mappings and the index now looks like this:
{
"metadata": {
"aliases": {},
"mappings": {
"properties": {
"doc": {
"properties": {
"contents": {
"properties": {
"alternateName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
}
}
},
"context_url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"created": {
"type": "date"
},
"creator": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"reource_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"resource_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "5",
"provided_name": "metadata",
"creation_date": "1667507228722",
"number_of_replicas": "5",
"uuid": "WO4zd7trTGyf4P_2wGpPOw",
"version": {
"created": "8030399"
}
}
}
}
}
I can now refer to the fields as doc.creator or doc.content.latitude etc. Sorry for long post, I hope it makes sense.
|
If it's possible we might to think about what will the search interface of metadata looks like. Right now the query syntax of search works; but because the record for metadata is so different than "dataset" or "file", how can we display the result?
|
I agree that it looks different. We can probably discuss this and address this in a separate PR. |
|
Could we index enough information about the file or datasets in the metadata index? If a user searches for metadata, they probbaly just want to see the file or dataset it belongs to. |
|
Right now, we have dataset_id and file_id in the metadata index. we can ad more info or just retrieve the info from mongo using the id |
lmarini
left a comment
lmarini
left a comment
There was a problem hiding this comment.
When I first add metadata to a dataset, the elasticsearch document has the contents under doc.contents. But if I update that entry, the new value is stored under 'contents. For example, this is what it looks like for a field called alternateName.
Looking into this. |
|
@lmarini I addressed the above issue. Great find, thanks! We don't need 'doc' while inserting record but need that while updating it. |
