"Do not leave upon the earth any of the disbelievers as inhabitants" means: anyone. This was when Allah granted him permission to invoke against them.
﴿لَا تذرعلى الأَرْض من الْكَافرين ديارًا﴾ أَيْ: أَحَدًا وَهَذَا حَيْثُ أَذِنَ الله لَهُ بِالدُّعَاءِ عَلَيْهِم