"So where are you going?" meaning: are you turning away from it. He says this to the polytheists.