However, new research may give mums and dads some reassurance, as it has been found that time spent in crèche or daycare has a "strongly positive effect" on kids.
Research conducted by scientists at Oxford University and the London School of Economics found that going to a childminder - including grandparents or other family members - helped children with their everyday skills like talking and social interactions.
According to the report, which is based on a survey of more than 800 mothers by the German Socio-Economic Panel, those attending nursery received a 10 percent positive impact on everyday skills and those who are minded by grandparents received a 10 percent boost with better social skills.
"It should give parents some reassurance that nurseries are not going to harm their children, and are likely to be beneficial," Laurence Roope, of Oxford University, who co-wrote the paper, said, according to the MailOnline.
"It seems that what is important is engaging in interactive activities. It could be there is a trade-off. Going out to work brings in more money for the family, which leads to more financial security and the ability to partake in more activities."