feat: add a feature to choose where to extract metadata by SaulLu · Pull Request #116 · bigscience-workshop/metadata

SaulLu · 2021-12-21T19:31:14Z

Here is the beginning of the PR discussed in the meeting. This RP includes:

the possibility to choose in which column the metadata are extracted
create the column if it does not exist
a new UrlProcessor to extract the url in the same format as the other metadata
tests that extract in cascade several types of metadata (in the case 1) where they are extracted in different columns and 2) in the same column): can you all see if the extraction on this toy dataset suits you? 🙏

Ready for review:

Timestamp @cccntu
Website Desc @shanyas10
Entities @manandey (please note that I've addded some mocks to test your entities extraction in our github workflows)
Datasource & Generation Length @chkla (I've pinged you on some part of the code because I've included some changes I've suggested yesterday)

cc @timoschick (if you can have a more deep review for the HtmlProcessor and UrlProcessor part it would be of a great help!)

…n_init

SaulLu · 2021-12-21T19:36:18Z

tests/test_preprocessing_utils.py

+                    {
+                        "char_end_idx": None,
+                        "char_start_idx": None,
+                        "relative_end_pos": None,
+                        "relative_start_pos": None,
+                        "html_attrs": None,
+                        "ent_desc": None,


Note here that with the datasets library all dictionaries in the lists of a unique column must have the same keys!

SaulLu · 2021-12-22T11:21:47Z

bsmetadata/preprocessing_utils.py

+                if not example_length:
+                    continue
+                example_metadata.append(example_length)
            elif self.mode == "sentence":
                example_length = self._extract_length_from_sentences(example_text)
+                example_metadata.extend(example_length)
            else:
                print("Please select a valid length type [text or sentence].")


@chkla , fyi, I've applied the changes proposed here, otherwise I've got an error with the index 0 in m[0]

SaulLu · 2021-12-22T11:22:28Z

bsmetadata/preprocessing_utils.py

+        # Iterate through the metadata associated with all examples in this batch.
+        for example_url, example_metadata in zip(examples[self.col_url], example_metadata_list):
            example_datasource = self._extract_datasource_from_url(example_url)
-            print(example_datasource)


@chkla , nit, I've removed it (cf this comment)

SaulLu · 2021-12-22T17:24:57Z

tests/test_preprocessing_utils.py

+        col_to_store_all_metadata = "metadata"
+        columns_names_to_concat = [
+            col_to_store_metadata_html,
+            col_to_store_metadata_url,
+            col_to_store_metadata_timestamp,
+            col_to_store_metadata_website_desc,
+            col_to_store_metadata_entities,
+            col_to_store_metadata_generation_length_text,
+            col_to_store_metadata_generation_length_sentence,
+            col_to_store_metadata_datasource,
+        ]
+        concat_columns_fn = concat_columns(
+            columns_names_to_concat=columns_names_to_concat,
+            new_colum_name=col_to_store_all_metadata,
+        )
+        ds = ds.map(concat_columns_fn, batched=True, batch_size=3, remove_columns=columns_names_to_concat)
+
+        for metadata_type in [
+            self.target_metadata_html,
+            self.target_metadata_url,
+            self.target_metadata_timestamp,
+            self.target_metadata_website_desc,
+            self.target_metadata_entities,
+            self.target_metadata_generation_length_text,
+            self.target_metadata_generation_length_sentence,
+            self.target_metadata_datasource,
+        ]:
+            for id, metadata_example in enumerate(metadata_type):
+                for metadata in metadata_example:
+                    for potential_missing_key in [
+                        "char_end_idx",
+                        "char_start_idx",
+                        "relative_end_pos",
+                        "relative_start_pos",
+                        "html_attrs",
+                        "ent_desc",
+                    ]:
+                        if potential_missing_key in metadata:
+                            continue
+                        metadata[potential_missing_key] = None
+                    self.assertIn(metadata, ds[id][col_to_store_all_metadata])


@cccntu , to accompany my answer on slack, here is the part I just added to the tests to show how we can reconcile all the metadata in the same column at the end

manandey

LGTM!

timoschick

My review comes a bit late, but I have now taken a look at the UrlPreprocessor and HtmlPreprocessor and everything looks good to me :)

* master: (141 commits) build: bump nltk to 3.6.7 for security and performance (bigscience-workshop#130) build: bump nltk to 3.6.7 for security and performance (#5) Add fp16, multi-GPU training script (toy dataset) (bigscience-workshop#123) create dataset with html, timestamp, url, datasource, generation length and website description metadata and tittles, footers and headers from HTML (bigscience-workshop#119) remove `#SBATCH --gres=gpu:0 ` from `03_create_dataset.slurm` (bigscience-workshop#121) Add joint training slurm script (bigscience-workshop#111) Add features types for the metadata to extract and test multiprocessing (bigscience-workshop#118) feat: add a feature to choose where to extract metadata (bigscience-workshop#116) Use dateutil to parse date (bigscience-workshop#117) feat: change how the entity extraction process use ids (bigscience-workshop#115) add `path_or_url_flair_ner_model` in order to execute the entity extraction on a partition without internet (bigscience-workshop#106) delete old submodule delete ds_store style check style & quality imports handle IndexError for `wikipedia_desc_utils` (bigscience-workshop#102) handle the comment specific type not recognized by pyarrow (bigscience-workshop#83) quality check Change torch version + make it optional (bigscience-workshop#82) ... # Conflicts: # bsmetadata/metadata_utils.py

SaulLu added 6 commits December 21, 2021 19:17

add flexibility arround the column in which to store the result

1c83314

create a preprocessing pipeline test

ff2840b

test put in same column

fd81ccf

Merge remote-tracking branch 'upstream/master' into put_column_name_i…

70f352e

…n_init

add entity extraction

f26de22

format

89d2734

SaulLu commented Dec 21, 2021

View reviewed changes

SaulLu added 9 commits December 21, 2021 20:37

sort imports

e489412

fix test

3676bd2

remove unnecessary print

3f3dbbf

change for generation length

b090b56

change example to have sentence length

f5cdc45

simply generationlenthpreprocessor

c9cb003

add datasource and generation length to tests

bada05e

format

de89154

change position UrlPreprocessor

f615119

SaulLu commented Dec 22, 2021

View reviewed changes

SaulLu requested review from changjonathanc, chkla, manandey, shanyas10 and timoschick December 22, 2021 11:27

add test to concat in one columns

47d96f3

SaulLu commented Dec 22, 2021

View reviewed changes

SaulLu added 2 commits December 22, 2021 18:33

format

e5d1010

isort

e3e82cd

manandey approved these changes Dec 22, 2021

View reviewed changes

chkla approved these changes Dec 22, 2021

View reviewed changes

SaulLu merged commit 8de2394 into bigscience-workshop:master Dec 23, 2021

timoschick reviewed Dec 23, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add a feature to choose where to extract metadata#116

feat: add a feature to choose where to extract metadata#116
SaulLu merged 18 commits intobigscience-workshop:masterfrom
SaulLu:put_column_name_in_init

SaulLu commented Dec 21, 2021 •

edited

Loading

Uh oh!

SaulLu Dec 21, 2021

Uh oh!

SaulLu Dec 22, 2021

Uh oh!

chkla Dec 22, 2021

Uh oh!

SaulLu Dec 22, 2021

Uh oh!

chkla Dec 22, 2021

Uh oh!

SaulLu Dec 22, 2021

Uh oh!

manandey left a comment

Uh oh!

timoschick left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

SaulLu commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SaulLu Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

SaulLu Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

chkla Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

SaulLu Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

chkla Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

SaulLu Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

manandey left a comment

Choose a reason for hiding this comment

Uh oh!

timoschick left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SaulLu commented Dec 21, 2021 •

edited

Loading