UI for displaying logs on extractors

[arunapa]

• Created 2 React custom components:
• ExtractorStatus -- UI table to display summary of job/progress so far
• ExtractorLogs -- terminal-like UI to display logs in real-time as they are being fetched

[tcnichol]

I'm getting the blank page error when I try to upload files here. Does this need main merged into it? That bug was fixed and merged in.

[arunapa]

[longshuicy]

You might want to clear this interval when leaving this widget; Right now it's continuing calling the endpoint every two seconds even though I close the modal.

The way of clearing the interval is to store the interval in state and give it a defaut:

Whenever you are setting the interval, make sure you update the state

const interval = setInterval(() => {
			if (isFormSubmitted == true && job_id.length > 0) {
				jobSummaryFetch(job_id);
				jobUpdatesFetch(job_id);
			}
		}, 2000);
setIntervalId(interval);

Then you can clear it this way

const _clear_old_timeinterval = () => {
		if (intervalId) {
			clearInterval(intervalId);
			setIntervalId(0);
		}
	};

[tcnichol]

I'm not sure why but I am running the wordcount extractor, but getting nothing for logs.

I merged main into the branch to fix the blank page error, so I hope that didn't mess something up.

[tcnichol]

I pulled the latest and tested with wordcount but I'm still not seeing any logs. Is this dependent on any other pull requests, or do I need to add something to the db manually to test? Thanks.

[arunapa]

I pulled the latest and tested with wordcount but I'm still not seeing any logs. Is this dependent on any other pull requests, or do I need to add something to the db manually to test? Thanks.

@tcnichol You should be able to see the UI even without any changes to the DB.
It will look something like this:

I'm also trying to get an extractor running to test from my end, but in the meantime let me also share the steps to display static extractor logs.

Steps:

1. If it is not already present, create a new mongo collection listener_jobs then listener_job_updates
2. Add some sample logs to the collection, like the following
   listener_jobs:

{
    "_id" : ObjectId("63d146bc79a3d39c71d0e0b1"),
    "listener_id" : "ncsa.wordcount",
    "resource_ref" : {
        "collection" : "file",
        "resource_id" : ObjectId("63d140925cda10061fcd6768"),
        "version" : 1.0
    },
    "creator" : {
        "id" : ObjectId("63a32586cdf079dcd6077a1a"),
        "email" : "[email protected]",
        "first_name" : "Chen",
        "last_name" : "Wang"
    },
    "parameters" : {

    },
    "created" : ISODate("2023-01-25T15:11:56.615+0000"),
    "started" : null,
    "updated" : ISODate("2023-01-25T14:11:56.000+0000"),
    "finished" : null,
    "duration" : 500.0,
    "latest_message" : "StatusMessage.done: Done processing.",
    "status" : "StatusMessage.done: Done processing."
}

listener_job_updates:

{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c31c"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.start: Started processing."
}
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c31e"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.processing: Downloading file."
}
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c320"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.processing: Loading contents of file..."
}
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c322"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.processing: Found 21 lines and 162 words..."
}
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c324"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.processing: Uploading file metadata."
}
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c326"),
    "job_id" : "63d146bc79a3d39c71d0e0b1",
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.done: Done processing."
}

4. Modify the value of job_id in ExtractorStatus.tsx

--- a/frontend/src/components/listeners/ExtractorStatus.tsx
+++ b/frontend/src/components/listeners/ExtractorStatus.tsx
@@ -86,8 +86,8 @@ export default function ExtractorStatus(props: { job_id: any; }) {
        useEffect(() => {
                const interval = setInterval(() => {
                        if (job_id.length > 0) {
-                               jobSummaryFetch(job_id);
-                               jobUpdatesFetch(job_id);
+                               jobSummaryFetch('63d146bc79a3d39c71d0e0b1');
+                               jobUpdatesFetch('63d146bc79a3d39c71d0e0b1');
                        }
                }, 2000);

[longshuicy]

I followed the steps you listed in the comment and it works well. Approving!

[longshuicy]

Ah i see. You need to move the timer logic into the status component instead of the parent component. Neat!