Al-Sha'bi (may Allah have mercy on him) said that he was asked about a slave who had children by a free woman, while the slave's father was free: to whom does the wala of his children belong? He said: To those who freed the grandfather.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الولاء / 21528