{blackbabes} Simon Cowell: 'We Have to Find a Role' for Mariah Carey on The X Factor

iTunes, App Store, iBookstore, and Mac App Store

Whatever Mariah Carey wants, Mariah Carey gets.

The X Factor judge Simon Cowell, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says "wanted to be involved in the show from day one."

"You can't say no to Mariah," Cowell, 51, tells PEOPLE. "We have to find a role for her. I'm a huge fan of hers – not just as an artist, but as a person, and I love her to death."

But Cowell also says that having Carey on the show poses a slight problem.

"We just can't shut Paula [Abdul] and Nicole [Scherzinger] up," he joked. "Add Mariah into the mix as well and you won't even have anyone singing."