Workspace preparation for FinalFit interface

Standard Procedure

The standard way to get HiggsDNA Ntuples and transform them in FinalFit friendly output is to use the prepare_output_file.py script, provided and maintained in the script repository. The script will perform multiple steps: * Merge all the .parquet files and categorise the events, obtaining one file for each category of each sample. * Convert the merged.parquet into ROOT trees. * Convert the ROOT trees into FinalFit compatible ``RooWorkspace``s.

All the steps can be performed in one go with a command more or less like this:

python3 prepare_output_file.py --input [path to output dir] --merge --root --ws --syst --cats --args "--do_syst"

or the single steps can be performed by running the auxiliary files (merge_parquet.py, convert_parquet_to_root.py, Tree2WS) separately. A complete set of options for the main script is listed below.

Merging step

During this step the main script calls merge_parquet.py multiple times. The starting point is the output of HiggsDNA, i.e. out_dir/sample_n/. These directory must contain only .parquet files that have to be merged. The script will create a new directory called merged under out_dir, if this directory already exists it will throw an error and exit. When converting the data (in my case they were split per era, Data_B_2017, Data_C_2017 etc.) the script will put them in a new directory Data_2017 and then merge again the output in a .parquet called allData_2017.parquet. During this step the events are also split into categories according to the boundaries defined in the cat_dict in the main file. An example of such dictionary is presented here:

if opt.cats:
cat_dict = {
    "best_resolution": {
        "cat_filter": [
            ("sigma_m_over_m_decorr", "<", 0.005),
            ("lead_mvaID", ">", 0.43),
            ("sublead_mvaID", ">", 0.43),
        ]
    },
    "medium_resolution": {
        "cat_filter": [
            ("sigma_m_over_m_decorr", ">", 0.005),
            ("sigma_m_over_m_decorr", "<", 0.008),
            ("lead_mvaID", ">", 0.43),
            ("sublead_mvaID", ">", 0.43),
        ]
    },
    "worst_resolution": {
        "cat_filter": [
            ("sigma_m_over_m_decorr", ">", 0.008),
            ("lead_mvaID", ">", 0.43),
            ("sublead_mvaID", ">", 0.43),
        ]
    },
}

if you don’t provide the dictionary to the script all the events will be put in a single file labelled as UNTAGGED.

During the merging step MC samples can also be normalised to the efficiency x acceptance value as required later on by FinalFits, this step can be skipped using the tag --skip-normalisation.

Root step

During this step the script calls multiple times the script convert_parquet_to_root.py. The arguments to pass to the script, for instance if you want the systematic variation included in the output ROOT tree are specified when calling prepare_output_file.py using --args "--do_syst". As before the script creates a new called root under out_dir, if this directory already exists it will throw an error and exit. In the script there is a dictionary called outfiles that contains the name of the output root file that will be created according to the process tipe, if the wf is run using the main script this correspond to the proces containd in process_dict.

Workspace step

During this step the main script uses multiple time the Flashgg_FinalFit, it moves to the directory defined in the --final_fit option (improvable) and uses the Tree2WS script there on the content of the root directory previously created. The output is stored in out_dir/root/smaple_name/ws/.

Commands

The workflow is meant to be run in one go using the prepare_output_file.py script, it can be also split in different steps or run with the single auxiliary files but it can result a bit cumbersome.

To run everything starting from the output of HiggsDNA with categories and systematic variatrion one can use:

python3 prepare_output_file.py --input [path to output dir] --merge --root --ws --syst --cats --args "--do_syst"

and everithing should run smoothly, it does for me at least (I’ve not tried the scripts in a while so thing may have to be adjusted in this document). Some options can be removed. If you want to use --syst and --root you should also add --args "--do_syst".

The complete list of options for the main file is here:

  • --merge, “Do merging of the .parquet files”

  • --root, “Do root conversion step”

  • --ws, “Do root to workspace conversion step”

  • --ws_config, “configuration file for Tree2WS, as it is now it must be stored in Tree2WS directory in FinalFit”,

  • --final_fit, “FlashggFinalFit path” # the default is just for me, it should be changed but I don’t see a way to make this generally valid

  • --syst, “Do systematics variation treatment”

  • --cats, =”Split into categories”,

  • --args, “additional options for root converter: –do_syst, –notag”,

  • --skip-normalisation, “Independent of file type, skip normalisation step”,

  • --verbose, “verbose lefer for the logger: INFO (default), DEBUG”,

The merging step can also be run separately using:

python3 merge_parquet.py --source [path to the directory containing .paruets] --target [target directory path] --cats [cat_dict]

the script works also without the --cats option, it creates a dummy selection of Pt > -1 and call the category UNTAGGED.

Same for the root step:

python3 convert_parquet_to_root.py [/path/to/merged.parquet] [path to output file containing also the filename] mc (or data depending what you're doing) --process [process name (should match one of the outfiles dict entries)] --do_syst --cats [cat_dict] --vars [variation.json]

--do_syst is not mandatory, but if it’s there also the dictionary containing the variations must be specified with the --var option. As before the script works also without the --cats option.