Simon Cowell: I Love That I'm Having A Baby

By Gillian Telling and Patrick Gomez

09/13/2013 at 06:45 PM EDT

Simon Cowell has enjoyed a career of being Mr. Mean on TV – and a personal life of yachts and parties, not marriage and babies.

So no one could have predicted last month's news that the confirmed bachelor was expecting a child with Lauren Silverman. Not even Cowell himself.

"I didn't think it was going to happen," he admits. "But you have to deal with it as a man: Let it happen naturally."

Now that the surprise has sunk in, the 53-year-old smiling, mellow father-to-be, told PEOPLE at The X Factor's premiere party Sept. 5, "I am the happiest now than I have been for a long time."

Of course, his path to parenthood hasn't exactly been smooth. Silverman, 36, was still married to Cowell's friend Andrew Silverman, 37, when news of her pregnancy broke. (The pair since finalized their divorce on Aug. 14.)

Cowell says that the situation was difficult, but he is at peace with how things have unfolded.

"That's all been settled now," he says. "I love her and I love the fact that I'm having this baby."