Competition between three-sublattice order and superfluidity in the quantum three-state Potts model of ultracold bosons and fermions on a square optical lattice

We study a quantum version of the three-state Potts model that includes as special cases the effective models of bosons and fermions on the square lattice in the Mott-insulating limit. It can be viewed as a model of quantum permutations with amplitudes J(parallel to) and J(perpendicular to) for identical and different colors, respectively. For J(parallel to) = J(perpendicular to) > 0 it is equivalent to the SU(3) Heisenberg model, which describes the Mott-insulating phase of 3-color fermions, while the parameter range J(perpendicular to) < min(0, - J(parallel to)) can be realized in the Mott insulating phase of 3-color bosonic atoms. Using linear flavor wave theory, infinite projected entangled-pair states (iPEPS), and continuous-time quantum Monte Carlo simulations, we construct the full T = 0 phase diagram, and we explore the T not equal 0 properties for J(perpendicular to) < 0. For dominant antiferromagnetic J(parallel to) interactions, a three-sublattice long-range ordered stripe state is selected out of the ground-statemanifold of the antiferromagnetic Potts model by quantum fluctuations. Upon increasing vertical bar J(perpendicular to)vertical bar, this state is replaced by a uniform superfluid for J(perpendicular to) < 0, and by an exotic three-sublattice superfluid followed by a two-sublattice superfluid for J(perpendicular to) > 0. The transition out of the uniform superfluid (that can be realized with bosons) is shown to be a peculiar type of Kosterlitz-Thouless transition with three types of elementary vortices.