"And we had no authority over you" to compel you into polytheism, "but you were a transgressing people," meaning: misguided.
﴿وَمَا كَانَ لَنَا عَلَيْكُم مِّن سُلْطَانٍ﴾ نقهركم بِهِ عَلَى الشّرك ﴿بَلْ كُنتُمْ قَوْمًا طَاغِينَ﴾ أَي: ضَالِّينَ