Search for the Double Dalitz Decays <i>η/η' →e</i> ⁺ <i>e</i> ^-μ <i>⁺μ ^-</i> and <i>η'→μ⁺μ^-μ⁺μ^-</i>

Using a data sample of (10087 +/- 44)x10(6) J/psi events collected with the BESIII detector, we search for the decays eta/eta' -> e (+) e (-)mu (+)mu (-) and eta'->mu(+)mu(-)mu(+)mu(-) via the radiative decaysJ/psi ->gamma eta/gamma eta '. No excess of events over expected background is observed for any of the decays of interest. At 90% confidence level, we report the first upper limits on the branching fractions of eta' -> e (+) e (-)mu (+)mu (-) and eta'->mu(+)mu(-)mu(+)mu(-) to be 1.75 x10(-6) and 5.28 x10(-7,) respectively. In addition, we set an upper limit on the branching fraction of eta -> e (+) e (-)mu (+)mu (-) to be 6.88 x10(-6,) which improves the previous result by about two orders of magnitude.