Prince Harry reached his limit with the British tabloids smearing his wife Meghan Markle for a deeply personal reason, Us Weekly is reporting, and one that may lead the couple to move abroad eventually if things don’t improve.

The outlet was told by a source that the UK tabloids’ stories have hurt Meghan, and Harry fears his relationship with her could be jeopardized if they just remain silent about the treatment.

There was no one story that was the prince’s tipping point—and that led him to file separate lawsuits against The Daily Mail and The Sun’s publishers. “It’s an accumulation, and Harry refuses to sit and do nothing as the British press scrutinizes Meghan and invades her privacy,” Us’ source said. “Meghan comes across as a strong woman, but she’s been deeply affected by the negative stories about her.”

The false stories have had a real effect on their lives—and could really damage the couple’s marriage, or so Harry fears. “[Harry is] worried that the recent escalation of press intrusion has an impact on his relationship with Meghan, as well as their relationship with the royal family and the public,” the source said.

A source claimed to Us that if things don’t change, the two haven’t ruled out the possibility of moving to another Commonwealth country permanently with their son Archie. “Meghan and Harry have considered moving to Canada,” that source said. (Meghan lived in Toronto before, when she shot Suits.) “Meghan and Harry are considering it for the future.” Whether that’s likely to seriously happen is doubtful for now, as the two remain committed to their royal duties in the UK.

As friends told earlier this summer, the Duchess of Sussex takes her Royal duties “very seriously” and is “conscious of tradition.” That’s why she attended Trooping the Colour just weeks after giving birth to her son, out of respect for the Queen.