A family member of a well known Rav [so well known that everybody knows of him] related [לשבח] that he paid no attention to his children. All he did was learn. When he was unable to learn because it was too dark, he would go on a walk with a child but the rule was that the child was not allowed to speak b/c that would distract him from his thoughts in learning. When a child was hospitalized his wife didn't tell him as to not distract him from his learning.
Many would harshly criticize such seemingly neglectful behavior. It is certainly not the way of even the biggest Masmidim. But I am not going there. It is not my place to criticize this person, although I wouldn't recommend it for others. I will let Hashem judge and try my best not to complain about the "snow on my neighbor's roof when my own doorstep is dirty" [as the old saying goes]. Baruch Hashem, I have plenty of things to fix and I hope Hashem gives me the time to fix them before expiring my visa here on the planet.
My point is that his children did not go off the derech. They actually became themselves huge Talmidei Chachomim. While it is true that some of his children can't read a Gemara - their husband's are great Talmidei Chachomim... He produced hundreds of grandchildren and great grandchildren יראים ושלמים.
It is great to get love from one's parents. It is amazing to have had an idyllic childhood. But who we ultimately become is up to us.
I am reminded of Clarence Thomas who went from being a poor black child in the segregated south, abandoned by his father and raised by his poor uneducated grandparents, who worked hard, went to Yale Law school [which he greatly regretted, by the way.....] and has been a Supreme Court Justice for the last 35 or so years. Before he received the job there were people who worked very very hard to bring him down [including Joe Biden who lied to him in order to obstruct his promotion but that is another story]. But he overcame.
History is not destiny.