"So they turned away from him, departing" to their festival. That is because they had invited him to accompany them to their festival—according to the tafsir (exegesis) of al-Kalbi—so he wrapped his head and said, "I saw in the stars tonight that I shall be afflicted with the plague tomorrow!" They used to look into the stars, so he said this to them out of his aversion to going with them. When he intended to do what he did to their idols, he used this as a ruse against them.
﴿فتولوا عَنهُ مُدبرين﴾ إِلَى عيدهم؛ وَذَلِكَ أَنهم استتبعوه لعيدهم - فِي تَفْسِير الْكَلْبِيّ - فعصب رَأسه، وَقَالَ: إِنِّي رأيتُ اللَّيْلَة فِي النُّجُوم أَنِّي سأطعن غَدا! وَكَانُوا ينظرُونَ فِي النُّجُوم، فَقَالَ لَهُم هَذَا كَرَاهِيَة مِنْهُ للذهاب مَعَهم، وَلما أَرَادَ أَن يفعل بآلهتهم كادهم بذلك