Simon Cowell could be set to leave The X Factor.

The 54-year-old media mogul is a judge on the singing contest - which is his brainchild - however he could be planning to leave the show since it is pulling in fewer viewers than in previous years - with the live show on Tuesday night, attracting only 4.8 million viewers. The drop in numbers could force bosses to lower the budget on the series, meaning Simon will walk away and back onto the judging panel of the UK version of the talent show.

"People have talked as if it is all over for The X Factor, but that's not the case at all. It is true that ratings are down but it still does well with younger viewers if it was to come back next year it would have to have a lower budget. Simon leaving the show would allow a cheaper panel to be introduced - and give him the time to be reunited with the British show," an insider said.

The X Factor is currently in its third season with the winner awarded a $1 million recording contract with Simon's label Syco, however, in the previous two series' it was worth $5 million. Simon is currently on the judging panel alongside Demi Lovato, Kelly Rowland and Paulina Rubio, and he is mentoring the groups category.