When you’re a member of the royal family, there’s always one person whose opinions matter more than your own. And that’s Queen Elizabeth.

A surprising new report from The Sunday Times found that Prince Harry and Meghan Markle didn’t originally want to live in Frogmore Cottage, which is where they currently reside with their son, Archie Harrison. The couple made headline months ago for organizing a pricey renovation and potentially telling neighbors not to say hello if they encountered them on the street (the verdict is still out on whether that truly happened).

But if they had their way, the Duke and Duchess of Sussex would have lived somewhere else instead. Rumor has it they had their sights set on the sprawling halls of Windsor Castle. Then Queen Elizabeth talked them out of it.

Nottingham Cottage didn’t suit their growing family

Before Frogmore, Prince Harry was enjoying life at Nottingham Cottage, a modest living space at Kensington Palace. Kensington is an enclave for a host of other royals, including Prince William and Kate Middleton.

Though Nottingham Cottage is one of the smallest residences associated with Kensington Palace, it was still plenty spacious for Prince Harry during his bachelor days and later Meghan Markle once the pair moved in together. It’s believed to have two bedrooms and a small garden.

Some royal fans expected that Prince Harry and Meghan Markle would want to move into a larger apartment at Kensington so they could be close to the other half of the “Fab Four” group. But instead, they moved to a more remote location that’s about an hour away from the rest of the royal family.

Prince Harry and Meghan Markle supposedly considered Windsor Castle

Royal fans surely remember Windsor Castle featuring prominently in Prince Harry and Meghan Markle’s shared history. It’s the place where the pair got married and later where they had their photoshoot with their new son and had him baptized in a private ceremony.

Now it appears the couple originally wanted to make Windsor their home base.  According to royal correspondent Roya Nikkhah, Harry and Meghan requested Queen Elizabeth for “living quarters (at Windsor Castle) could be made available after their marriage.”

But the queen had other ideas. She “politely but firmly suggested” that they choose Frogmore Cottage instead, “which is said to be her favorite home.”

This revelation changes everything

It’s quite surprising news. While Prince Harry and Meghan Markle received so much criticism for moving away from the rest of the family to do their own thing, finding out that they might have wanted to live in the largest inhabited palace in the world, which had around 1,000 rooms, makes the couple seem a lot less secretive.

There’s also the small matter that the queen herself lives at Windsor Castle. If this report is true, then it means Harry and Meghan did originally want to live near the family, not run from them.

The Sussex family seems to be enjoying their lives at Frogmore. But now we know it almost didn’t turn out that way!