"And we had no authority over you" to compel you into polytheism, "but you were a transgressing people," meaning: misguided.