Dancing with the Stars: Was Sean Avery Forced Off the Show?

TV Watch Buzz

By Emily Strohm

04/01/2014 at 04:05 PM EDT

Sean Avery and Karina Smirnoff

Craig Sjodin/ABC

Was Sean Avery's Dancing with the Stars elimination a question of votes – or vengeance?

After last week's shocking double elimination, which sent the former hockey pro and partner Karina Smirnoff packing, Avery says it was a disagreement with producer Conrad Green – not his dance technique – that caused his early exit.

"I will tell everyone on #GMA tomorrow why i was voted off ... my partner @Karina_Smirnoff is AMAZING," he Tweeted March 24, after the duo was eliminated.

The following morning Avery, 33, appeared on Good Morning America with Smirnoff, 36, and hinted that a dispute with a producer may have been the real reason for his departure from the show in only its second week of competition.

"I think we knew as of probably Monday last week that our fate was sealed," he said. "I had a conversation with Conrad, one of the producers, and I was upset at the package that they had brought in the week previous, and it was a very tacky attempt to show me as the bad boy of DWTS. And I think in reality TV or some sort of competition, if you have a disagreement with the producers then generally that seals your fate quite early."

The video package that ABC aired featured Avery's ex-girlfriend, actress Elisha Cuthbert, and a quip of him publicly referring to her as his "sloppy seconds," a comment that landed him in hot water with the NHL in 2008 and for which he later apologized.

While he was visibly and audibly annoyed on the night of the premiere, a source tells PEOPLE that he later called producer Conrad Green irate over the matter, which may have led to his early departure.

Despite being in the bottom two with Diana Nyad and Henry Byalikov, Avery and Smirnoff normally would have been safe. But with a surprise double elimination that night, which was kept a secret from some of the cast and crew, the pair was sent home.

"Eliminations are based solely on public voting combined with scores from the judges," ABC tells PEOPLE in a statement. "The double elimination in week two was planned in advance and producers have no hand in determining who is voted off each week."