[Common] Isolated lumi calculation in a dedicated task

[ekryshen]

Main changes:

• lumi calculation moved from bc-selection-task to a dedicated lumi task
• introduced pileup correction using rates from CTP scalers
• introduced 2D lumi tables for various RCT selection
• Changed default ccdb path for RCT to RCT/Flags/RunFlags
• setting bit 31 in rct flags to 1 if rct map is not found in ccdb

[github-actions]

O2 linter results: ❌ 0 errors, ⚠️ 0 warnings, 🔕 42 disabled

[victor-gonzalez]

@ekryshen I believe this is going to fail when running on converted Run1/2 data. What do you think?

[ddobrigk]

This is a good point - @ekryshen, can you please check? Thanks!

[ekryshen]

@victor-gonzalez @ddobrigk Indeed, thanks for noticing. I'll fix it on Monday.

[ekryshen]

@ddobrigk Should be fixed now (added processRun3)

[victor-gonzalez]

@ekryshen Thanks!
But shouldn't it have a default when running on converted data?
My guess is that the workflow will hang in such a case but perhaps that is an old behavior. Sorry for the noise if that is the case

[ekryshen]

@victor-gonzalez Indeed, the workflow hangs without dummy process function for Run2. I've just committed another fix.

[victor-gonzalez]

@victor-gonzalez Indeed, the workflow hangs without dummy process function for Run2. I've just committed another fix.

Thanks @ekryshen!!!!
Fine from my side!!!

[ddobrigk]

Thanks @ekryshen, I will approve and merge now! Since you added an extra struct, this PR may mean an increase in memory. I am not sure how relevant this is, but let's just go ahead so that we get the added features immediately and we can then evaluate the memory changes in a tag including this PR.

[lubynets]

Dear @ekryshen,
After you added LumiTask, by default arguments lumi-task processRun2 and processRun3 are switched to false (this is how they appear in the newly produced dpl-config.json file). In this case the job is stuck without any error message.
Is it worth adding an error Log and termination of the task if both arguments are false?

[ekryshen]

Dear @ekryshen, After you added LumiTask, by default arguments lumi-task processRun2 and processRun3 are switched to false (this is how they appear in the newly produced dpl-config.json file). In this case the job is stuck without any error message. Is it worth adding an error Log and termination of the task if both arguments are false?

@lubynets Thanks for the suggestion, I will implement it in the next iteration. I wonder if this issue (no process functions) can be catched by the central framework. @aalkin, what do you think?

[aalkin]

@ekryshen The workflow gets stuck due to the lack of inputs, rather than lack of enabled process functions - the latter is a normal situation. The lack of inputs is also "normal" at the stage before the configuration is read, so we need something more involved.

[lubynets]

Dear @aalkin (@ekryshen cc),
Thanks for your reply. Let me disagree with what you write - the workflow gets stuck not because of lack of input, but because of both processes false - I checked it explicitly.
This commit lubynets@1afd2cb may help users to localize the problematic place, because it immediately throws fatal error and does not get stuck.
Optionally one can replace false here

O2Physics/Common/TableProducer/eventSelection.cxx

Line 1454 in 675e6ba

PROCESS_SWITCH(LumiTask, processRun3, "Process Run3 lumi task", false);

with true, since Run3 analysis is now more popular. Or is it not a good idea from the point of view of Run 2 analyzers?

[aalkin]

@lubynets The task inputs are determined by enabled processes, when both are disabled in this case, the input list is cleared. If you run with the DebugGUI you'll notice the distinct lack of channels leading to the task as it has no inputs. Due to this, task's processing callback is not triggered and it is stuck. There are examples of tasks that have no process functions at all and are functioning, because they have Spawns or Builds declarations in them, that also add to input list.