{"slug": "event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts", "title": "Event ID based splitting of 2024 MC samples into 2024, 2025 and 2026 parts", "summary": "Marcel Rieger developed a function to assign CMS Monte Carlo events to years 2024, 2025, and 2026 based on recorded luminosities. The assignment is pseudo-random, deterministic, and unbiased, preserving event id distributions for downstream analyses.", "body_md": "|\n\"\"\" |\n|\nFunctions to assign CMS MC events to years 2024, 2025, and 2026 based on the recorded luminosities |\n|\nin each year. The assignment is pseudo-random, deterministic and unbiased, i.e., it does not skew |\n|\nthe distribution of event ids for any of the years. This means that analysis downstream that need |\n|\nsplit events (e.g. for ML datasets) can still rely on the event id and modulo-based splittings. |\n|\n|\n|\nSee :py:func:`assign_year` below for more details. |\n|\n|\n|\nFor implementations in C and Python (with tests), see: |\n|\nhttps://gist.github.com/riga/f9476f3b1477f1609683bea68ae64897 |\n|\n|\n|\nAuthor: Marcel Rieger |\n|\n\"\"\" |\n|\n|\n|\nfrom __future__ import annotations |\n|\n|\n|\nfrom typing import TypeVar, Any |\n|\n|\n|\nimport numpy as np |\n|\nfrom numpy.typing import NDArray |\n|\n|\n|\n|\n|\nIntTypes = TypeVar(\"IntTypes\", int, NDArray) |\n|\n|\n|\n|\n|\n# |\n|\n# constants |\n|\n# |\n|\n|\n|\n# recorded CMS luminosities |\n|\n# (from https://twiki.cern.ch/twiki/bin/view/CMSPublic/LumiPublicResults?rev=213) |\n|\nlumis = { |\n|\n2024: 112.70, |\n|\n2025: 114.85, |\n|\n2026: 30.36, |\n|\n} |\n|\n|\n|\n# build fractions relative to their sum, multiply by 1000 (for accuracy), then convert to ranges |\n|\nlumi_sum = sum(lumis.values()) |\n|\nlumi_fractions = {year: int(round(1000 * lumi / lumi_sum)) for year, lumi in lumis.items()} |\n|\nif sum(lumi_fractions.values()) != 1000: |\n|\nraise RuntimeError(f\"fractions do not sum up to 1000: {lumi_fractions}\") |\n|\nlumi_ranges = {} |\n|\noffset = 0 |\n|\nfor year, fraction in lumi_fractions.items(): |\n|\nlumi_ranges[year] = (offset, offset + fraction) |\n|\noffset = lumi_ranges[year][1] |\n|\n|\n|\n|\n|\n# |\n|\n# year assignment function and helpers |\n|\n# |\n|\n|\n|\n|\n|\ndef assign_year( |\n|\nevent_id: IntTypes, |\n|\nlumi_ranges: dict[int, tuple[int, int]] = lumi_ranges, |\n|\n) -> IntTypes: |\n|\n\"\"\" |\n|\nTakes a single *event_id* or an array of ids and assigns them to one of the years defined in *lumi_ranges*. The |\n|\nassignment is pseudo-random and deterministic, i.e. the same event id will always be assigned to the same year |\n|\nwithout any bias towards specific event id. See :py:func:`split_mix_hash` for details on the hashing algorithm used. |\n|\n|\n|\n*lumi_ranges* is a dictionary mapping years to a tuple of (min, max) values that define the range of hashed values |\n|\nthat will be assigned to that year. The ranges must be non-overlapping and cover the entire range of possible hashed |\n|\nvalues, 0 (incl.) to 1000 (excl.). Thereby, the frequency of assigned years will be proportional to the |\n|\nluminosities. |\n|\n|\n|\nExamples: |\n|\n|\n|\n.. code-block:: python |\n|\n|\n|\n# single event id |\n|\nassign_year(1234567890) # -> 2025 |\n|\nassign_year(1234567891) # -> 2024 |\n|\n... |\n|\n|\n|\n# array of event ids |\n|\nassign_year(np.array([1234567890, 1234567891])) # -> np.array([2025, 2024]) |\n|\n\"\"\" |\n|\n# handle single integers |\n|\nsingle_input = isinstance(event_id, int) |\n|\n|\n|\n# apply hashing |\n|\nh = split_mix_hash(np.array([event_id], dtype=np.uint64) if single_input else event_id) |\n|\n|\n|\n# take last three digits and cast down |\n|\nh = (h % 1000).astype(np.uint16) |\n|\n|\n|\n# perform assignment based on lumi ranges |\n|\nyears = np.zeros(len(h), dtype=np.uint16) |\n|\nfor year, (min_val, max_val) in lumi_ranges.items(): |\n|\nyears[(h >= min_val) & (h < max_val)] = year |\n|\n|\n|\nreturn years[0] if single_input else years |\n|\n|\n|\n|\n|\ndef split_mix_hash(a: IntTypes) -> IntTypes: |\n|\n\"\"\" |\n|\nDeterministic integer mixing using the splitmix64 algorithm with the goal that a small change in the input causes a |\n|\nlarge change in the output hash. See https://rosettacode.org/wiki/Pseudo-random_numbers/Splitmix64. |\n|\n\"\"\" |\n|\n# handle single integers |\n|\nsingle_input = isinstance(a, int) |\n|\n|\n|\n# start from uint64 |\n|\nh = np.array([a], dtype=np.uint64) if single_input else a.copy().astype(np.uint64) |\n|\n|\n|\n# start mixing |\n|\nh += np.uint64(0x9e3779b97f4a7c15) |\n|\nh = (h ^ (h >> 30)) * np.uint64(0xbf58476d1ce4e5b9) |\n|\nh = (h ^ (h >> 27)) * np.uint64(0x94d049bb133111eb) |\n|\nh = (h ^ (h >> 31)) |\n|\n|\n|\nreturn h[0] if single_input else h |\n|\n|\n|\n|\n|\ndef uproot_select_year(year: int) -> dict[str, Any]: |\n|\n\"\"\" |\n|\nReturns a dictionary with the necessary arguments to pass to :py:meth:`uproot.TTree.arrays` (and alike) to read only |\n|\nevents that are assigned to the given *year*. The returned dictionary contains aliases and cut expressions. |\n|\n|\n|\nExample: |\n|\n|\n|\n.. code-block:: python |\n|\n|\n|\n# only read events from nano aod file that are assigned to 2025 |\n|\nwith uproot.open(\"nano.root\") as f: |\n|\ntree = f[\"Events\"] |\n|\nevents_2025 = tree.arrays(**uproot_select_year(2025)) |\n|\n\"\"\" |\n|\n_patch_uproot_uint64() |\n|\n|\n|\n# nested aliases that represent the splitmix64 hash algorithm from :py:func:`split_mix_hash` |\n|\naliases = { |\n|\n\"split_mix_hash_1\": \"event + uint64(11400714819323198485)\", |\n|\n\"split_mix_hash_2\": \"(split_mix_hash_1 ^ (split_mix_hash_1 >> 30)) * uint64(13787848793156543929)\", |\n|\n\"split_mix_hash_3\": \"(split_mix_hash_2 ^ (split_mix_hash_2 >> 27)) * uint64(10723151780598845931)\", |\n|\n\"split_mix_hash\": \"split_mix_hash_3 ^ (split_mix_hash_3 >> 31)\", |\n|\n\"event_split_id\": \"split_mix_hash % 1000\", |\n|\n} |\n|\n|\n|\n# cut expression on lumi range for that year |\n|\ncut = f\"(event_split_id >= {lumi_ranges[year][0]}) & (event_split_id < {lumi_ranges[year][1]})\" |\n|\n|\n|\nreturn {\"aliases\": aliases, \"cut\": cut} |\n|\n|\n|\n|\n|\ndef _patch_uproot_uint64() -> None: |\n|\ntry: |\n|\nimport uproot |\n|\nexcept ImportError: |\n|\nreturn |\n|\n|\n|\n# add \"uint64\" conversion |\n|\npy_lang_functions = uproot.language.python.PythonLanguage.default_functions |\n|\nif \"uint64\" not in py_lang_functions: |\n|\npy_lang_functions[\"uint64\"] = np.uint64 |\n|\n|\n|\n|\n|\n# |\n|\n# testing |\n|\n# |\n|\n|\n|\ndef main() -> None: |\n|\nimport os |\n|\nimport argparse |\n|\nimport uproot |\n|\nimport uniplot |\n|\n|\n|\nparser = argparse.ArgumentParser(description=__doc__.strip().splitlines()[0].strip()) |\n|\nparser.add_argument(\"input\", help=\"input NanoAOD file\") |\n|\nargs = parser.parse_args() |\n|\n|\n|\n# show lumis and fractions |\n|\nprint(\"luminosities and fractions:\") |\n|\nfor year, lumi in lumis.items(): |\n|\nprint(f\" - {year}: {lumi:.2f}/fb, 1k-fraction: {lumi_fractions[year]}, range: {lumi_ranges[year]}\") |\n|\nprint() |\n|\n|\n|\n# test specific event ids |\n|\ntest_ids = [1, 9, 10, 16849655] |\n|\nfor event_id in test_ids: |\n|\nprint(f\"assign {event_id} -> {assign_year(event_id)}\") |\n|\nprint() |\n|\n|\n|\n# load event ids from nano file |\n|\ninput_file = os.path.expanduser(os.path.expandvars(args.input)) |\n|\nwith uproot.open(input_file) as f: |\n|\nids = f[\"Events\"][\"event\"].array(library=\"np\") |\n|\n|\n|\n# assign years |\n|\nyears = assign_year(ids) |\n|\n|\n|\n# for each year, plot the distribution of last and last-two digits of event ids |\n|\n# to check for bias (via uniformity) |\n|\nfor digits in range(1, 4): |\n|\nbins = 10**digits |\n|\ndata = [ |\n|\n(ids[years == year] % bins) |\n|\nfor year in lumis |\n|\n] |\n|\nuniplot.histogram( |\n|\ndata, |\n|\nlegend_labels=list(map(str, lumis)), |\n|\ncolor=True, |\n|\nwidth=100, |\n|\nheight=25, |\n|\nbins=min((bins := 10**digits), 100), |\n|\nbins_min=-0.5, |\n|\nbins_max=bins - 0.5, |\n|\ntitle=f\"Distribution of last {digits} digit(s) of event ids after splitting\", |\n|\n) |\n|\nprint() |\n|\n|\n|\n|\n|\nif __name__ == \"__main__\": |\n|\nmain() |", "url": "https://wpnews.pro/news/event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts", "canonical_source": "https://gist.github.com/riga/f9476f3b1477f1609683bea68ae64897", "published_at": "2026-07-18 10:31:53+00:00", "updated_at": "2026-07-20 17:01:15.474507+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["Marcel Rieger", "CMS"], "alternates": {"html": "https://wpnews.pro/news/event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts", "markdown": "https://wpnews.pro/news/event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts.md", "text": "https://wpnews.pro/news/event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts.txt", "jsonld": "https://wpnews.pro/news/event-id-based-splitting-of-2024-mc-samples-into-2024-2025-and-2026-parts.jsonld"}}