"If Allah brings you back to a party of them"—He is speaking to the Prophet—to His saying: "then stay with those who remain behind." Al-Kalbi said: It means: the wicked.
Muhammad said: The singular of "al-khalifin" (those who remain behind) is "khalif"; it is said of one who is not noble/virtuous: so-and-so is "khalif" of his family, and "khalifat" of his family.