Learning Cross-lingual Sentence Representations for Multilingual Document Classification with Token-level Reconstruction

Fixed-dimensional cross-lingual sentence representations [1, 2, 3, 4, 5, 6, 7] are widely used to initialize another light neural network for cross-lingual sentence understanding downstream tasks. When learning such representations, efficiency and robustness form the core. In this work, we explore different training tasks for learning fixeddimensional cross-lingual sentence representations in a 2- layer dual-transformer framework aiming to improve the efficiency and retain the robustness. We observe that current cross-lingual training tasks leave a lot to be desired for this shallow architecture. To ameliorate this, we propose a novel cross-lingual language model, which combines the Single-word Masked Language Model (SMLM) with Token-level Reconstruction (TR) task. Our comparisons with competing models on multilingual document classification verify the effectiveness of the newly proposed training tasks for a shallow model.