There are so many unconfirmed affair rumors in the British royal family it’s hard to keep track. But one infidelity that everyone agrees happened — including the adulterers themselves — is the clandestine romance between Prince Charles and Camilla Parker Bowles.

It’s one of the most infamous affairs of all time. Prince Charles cheated on Princess Diana with his former paramour Camilla, who was also married at the time. The drama that ensued eventually led to the disastrous divorce of Charles and Diana one year before her tragic death.

For a long time, people blamed Camilla and Prince Charles for Princess Diana’s emotional turmoil. But even if he had never cheated, their marriage was clearly doomed from the start. Read on to find out why.

Prince Charles never wanted to marry Princess Diana

Though it wasn’t an arranged marriage, it might as well have been. Princess Diana was selected for Prince Charles as a suitable, virginal bride. But these two barely knew each other before getting married and they certainly weren’t in love with each other.

Meanwhile, Prince Charles had met Camilla Shand before being introduced to Lady Diana Spencer. However, Camilla lacked the proper lineage, so Charles was encouraged to ignore his feelings for her. His sense of duty forced him to follow the rules instead of following his heart.

He was honest about his feelings from the start

An engagement should be the happiest time of any couple’s life. But while making their engagement announcement in 1981, the interviewer asked if Prince Charles and Lady Diana were in love. “Of course!” she said immediately. But Charles was more hesitant, and said frankly, “Whatever ‘in love’ means.”

If Prince Charles wasn’t even feeling romantic toward his future bride then, how could anyone expect him to in later years?

Diana and Charles’ personalities were not compatible

The first issue this ill-fated couple ran into was their age difference. The Prince of Wales was 32 when they became engaged while Lady Diana was just 20. This might not seem like a huge age gap compared to others. But in terms of maturity and life experience, these two were worlds apart.

It didn’t help that young Diana was not prepared for the onslaught of media scrutiny following her nuptials to Prince Charles. When she went to her husband for support and expected him to defend her from their cruelest comments, he reacted in an aloof manner that made her even more despondent.

Royal biographer Andrew Morton explained the phenomenon in his book, Diana: Her True Story. “Diana pleaded, cajoled and quarreled violently with Charles as she tried to win his assistance,” he wrote. “The Prince seemed incapable of understanding or wishing to comprehend the turmoil in Diana’s life.”

In short, she was highly emotional while he was distant and cool. The combination led to a relationship that was doomed to fail, affair or no affair.

Prince Charles didn’t begin his affair right away

Even though Prince Charles was still thinking about Camilla in the early days of his marriage, he did wait a while before beginning a full-fledged affair.

An authorized biographer claims that Prince Charles began seeing Camilla in 1986, a full five years after getting married. There’s no excuse for infidelity, it’s true. But this proves that Prince Charles did put aside his deepest desires for a time and tried to make things work with his wife.

The Prince of Wales felt guilty for how things turned out

Ultimately, Prince Charles believed he let his family down with his failed marriage. But his father Prince Philip saw things differently. Royal biographer Angela Levin explained in Harry: Conversations with the Prince, “Based at Highgrove, Charles was consumed with guilt that he had made Diana so unhappy and in doing so had also let down the Queen, the Queen Mother, Prince Philip, his sons, and the British people.”

She continued: “He found a rare moment of solace in a letter he received from his normally critical father wrote that he felt Charles had displayed he had the ‘fortitude of a saint’ by putting up with Diana.”

The affair was a terrible time. But neither Prince Charles nor Princess Diana was completely blameless in how things turned out. With or without Camilla, these two were never going to work out.