Joe Jonas Says of Drug Rumors: 'It's Ridiculous'

People Exclusive

By Sharon Cotliar

10/30/2013 at 03:00 PM EDT

After the Jonas Brothers abruptly canceled their tour three weeks ago – and went silent as they contemplated the band's future – reports surfaced suggesting that Joe Jonas had a drug problem.

Now the 24-year-old singer wishes to set the record straight, telling PEOPLE: "It's ridiculous!"

"We're not saying anything for two or three weeks and I'm a drug addict!"

What upset him even more is that his girlfriend, Blanda Eggenschwiler, was also dragged into the rumors.

"It's obviously not true. I've never touched heroin in my life. Either has my girlfriend," he said. "It's hilarious to me, but also frustrating as well. It pissed me off when it's about my girl and even about myself."

At one point, he said, people even approached him personally about it.

"I was walking down the street and people are asking me if they can give me a hug and how rehab is," says Joe. "I'm like, you tell me because I've never been."

So why didn't he speak out right away to correct the impression?

After Nick Jonas, 21, first confessed to older brothers Joe and Kevin that he had concerns about the band – they then unanimously decided to call it quits – Joe says: "We all had to deal with it in our own way. I needed time to clear my head."

Already, that is a thing of the past. Looking ahead, Joe says, "I'm genuinely excited."

For more on the Jonas Brothers' decision to call it quits, including Joe's reaction to false reports he had drug problem, pick up this week's issue of PEOPLE, on newsstands Friday