Harry Potter is inarguably one of the most successful media franchises of all time, with the entire brand of books, movies, theme parks, and merchandise worth an estimated $25 billion. But though it may be hard to remember, there was a time when some people weren’t all that eager to become a part of J.K. Rowling’s magical wizarding world. That’s right — believe it or not, several popular actors have admitted to refusing to take on roles in the Harry Potter films.

Whether due to schedule constraints or a simple lack of interest, these high-profile names turned down offers to star in the Harry Potter flicks — which they’re probably kicking themselves over now, considering that the movie franchise is the second highest-grossing film series of all time. So who came close to nabbing which parts? Below, check out seven actors who declined these now-iconic Harry Potter roles:

1. Ian McKellen

Two different actors have portrayed Dumbledore (first Richard Harris, then Michael Gambon after Harris’s death), but McKellen was also once offered the role of the Hogwarts headmaster. What stopped him from taking the role? Harris had publicly criticized McKellen in the past and the actor apparently didn’t feel right taking the part after that.

“When they called me up and said would I be interested in being in the Harry Potter films, they didn’t say in what part,” McKellen told Variety of the offer. “I worked out what they were thinking, and I couldn’t … I couldn’t take over the part from an actor who I’d known didn’t approve of me.”

2. Naomi Watts

Naomi Watts received an offer to play Narcissa, Lucius Malfoy’s wife and Draco’s mother, in the Harry Potter movies. But she never got the chance to appear in the blockbuster franchise, as her agency apparently turned down the role without even talking to her. Yikes … bet they kicked themselves later.

The role of Narcissa later went to English actress Helen McCrory.

3. Hugh Grant

Hugh Grant may have the winning grin to play Gilderoy Lockhart, Defense Against the Dark Arts Professor and five-time winner of Witch Weekly‘s Most Charming Smile Award, but unfortunately he didn’t have the time. Though Grant reportedly initially accepted the offer, he later had to reverse his decision and decline the role, due to scheduling conflicts.

That left the part open for Kenneth Branagh, who took on the role of Lockhart with gusto.

4. Tim Roth

Pulp Fiction star Tim Roth turned down the part of potions master Severus Snape in 2001, despite that his children were eager for him to take it. “I wasn’t ready to be on a lunch box,” he later told Metro UK of his decision.

The career-defining role eventually went to Alan Rickman, who sadly passed away in 2016.

5. Tilda Swinton

Tilda Swinton was approached for the role of Professor of Divination, Sybil Trelawney, who was ultimately played by Emma Thompson. Scheduling conflicts prevented her from taking on the part, but it seems Swinton isn’t much of a Potter fan anyway.

“I’m not a believer in Harry Potter,” she later told Out magazine. “Because I believe it fetishizes boarding school for children.” OK then.

6. Rosamund Pike

Rosamund Pike, who earned an Oscar nomination for her role in 2014’s Gone Girl, was apparently producers’ first choice to play nosy gossip journalist, Rita Skeeter. She declined the part for unspecific reasons and the job later went to Miranda Richardson, who played the sensationalist character excellently.

7. Kate Winslet

Kate Winslet was rumored to be eyed for the part of Helena Ravenclaw aka the Grey Lady, who played an important role in helping Harry destroy Voldemort’s horcruxes in Harry Potter and the Deathly Hallows: Part 2. Though it’s a significant role, it’s also a small one — which is likely why Winslet’s agent reportedly turned down the part for her.

While it would have been interesting to see Winslet’s version of the character, it all worked out — particularly for Kelly Macdonald, who eventually nabbed the role.