Search for the nonresonant and resonant production of a Higgs boson in association with an additional scalar boson in the γγττ final state in proton-proton collisions at s=13 TeV

The results of a search for the production of two scalar bosons in final states with two photons and two tau leptons are presented. The search considers both nonresonant production of a Higgs boson pair, HH, and resonant production via a new boson X which decays either to HH or to H and a new scalar Y. The analysis uses up to 138 fb−1 of proton-proton collision data, recorded between 2016 and 2018 by the CMS experiment at the LHC at a center-of-mass energy of 13 TeV. No evidence for signal is found in the data. For the nonresonant production, the observed (expected) upper limit at 95% confidence level (CL) on the HH production cross section is set at 930 (740) fb, corresponding to 33 (26) times the standard model prediction. At 95% CL, HH production is observed (expected) to be excluded for values of κλ outside the range between −12 (−9.4) and 17 (15). Observed (expected) upper limits at 95% CL for the X → HH cross section are found to be within 160 to 2200 (200 to 1800) fb, depending on the mass of X. In the X → Y(ττ)H(γγ) search, the observed (expected) upper limits on the product of the production cross section and decay branching fractions vary between 0.059–1.2 fb (0.087–0.68 fb). For the X → Y(γγ)H(ττ) search the observed (expected) upper limits on the product of the production cross section and Y → γγ branching fraction vary between 0.69–15 fb (0.73–8.3 fb) in the low Y mass search, tightening constraints on the next-to-minimal supersymmetric standard model, and between 0.64–10 fb (0.70–7.6 fb) in the high Y mass search.