When giants fall, they fall fast. We haven’t even been living with the revelation of Harvey Weinstein’s sexual harassment history for a week, yet the news has gotten worse every day. From the New York Times’ initial report — which specified the now-familiar details of hotel meetings, ulterior motives, and “massages” that were less than optional — the accusations have escalated to include masturbating in front of an actress, forced oral sex, and vaginal rape. (Weinstein’s spokesperson, Sallie Hofmeister, says ““Any allegations of non-consensual sex are unequivocally denied by Mr. Weinstein. Mr. Weinstein has further confirmed that there were never any acts of retaliation against any women for refusing his advances.”) Weinstein’s accusers now include Gwyneth Paltrow, Asia Argento, Patricia Arquette, and Angelina Jolie. Meanwhile, a former New York Times reporter has now alleged that a story she was working on in 2004 related to Weinstein’s behavior was “gutted” after she received phone calls from Russell Crowe and Matt Damon vouching for a former Weinstein colleague. (Damon vigorously denies he knew anything about Weinstein’s behavior). Weinstein himself has been forced out of the Weinstein Company.

As always, with such a disturbing story, people are looking to assign blame. Whose fault is Harvey Weinstein?

Harvey Weinstein, apparently, is Hillary Clinton’s fault.

Harvey Weinstein is also, somehow, Meryl Streep’s fault.

The one person who’s not responsible for Harvey Weinstein, evidently, is Harvey Weinstein. Or “Ben Affleck, Matt Damon, Colin Firth, Bradley Cooper, Brad Pitt, Leonardo DiCaprio, Daniel Day-Lewis, Russell Crowe, George Clooney, Ewan McGregor… [Quentin] Tarantino, [David O.] Russell, Ryan Coogler, Tom Hooper, Lin-Manuel Miranda, Michael Moore, Rob Marshall, Robert Pulcini, Garth Davis, Doug McGrath, John Madden, Simon Curtis, Kevin Williamson, Martin Scorsese, John Hillcoat [or] John Wells” — all of whom were contacted by the Guardian for a statement on Weinstein’s allegations, and initially declined to speak. Some of them have come forward since (thanks, Lin-Manuel!) but the fact remains; Weinstein, initially, was framed as a woman’s problem, something women were particularly obligated to disavow and fix. Men were pretty much allowed to continue on about their day.

The question of complicity is, obviously, a relevant one. Ronan Farrow’s New Yorker article, which broke the rape allegations against Weinstein, suggests that the producer’s assaults were only possible because he had a huge network of enablers. Some of them were in his own company; a raft of assistants set up meetings with the targets and tacitly agreed to ignore the consequences. (“Sixteen former and current executives and assistants at Weinstein’s companies told me that they witnessed or had knowledge of unwanted sexual advances and touching,” Farrow writes.) Worse still, some were allegedly in the press; again, quoting Farrow, “[m]ultiple sources said that Weinstein frequently bragged about planting items in media outlets about those who spoke against him; these sources feared that they might be similarly targeted.” One woman describes watching Weinstein fly into a fury over a negative story about the actress Amy Adams, assuring her that soon, the media would be rewriting the narrative to favor Weinstein’s side of the argument: “Later in the day, there were indeed negative news items about his opponents, and Weinstein stopped by [her] desk to be sure that she’d seen them.”

There is just no way to safely report a rape when your attacker can render you unemployable or turn every reporter in town against you within 24 hours. If all of this is true — and we do still have to say “if,” not least because Harvey Weinstein still has a lot of money and a lot of lawyers — then people within and around Hollywood kept Weinstein untouchable for decades, and by doing so, all but ensured more women would be attacked.

Complicity matters. But when dealing with a man like Weinstein, complicity — particularly when it comes to women — is also self-perpetuating, self-reinforcing, and much, much more complicated than we’re letting on.

Consider that one of Weinstein’s biggest female supporters, over the past few years, has been Gwyneth Paltrow — someone who was more than aware of the rumors, because, she says, Weinstein harassed her. This allegedly happened on the eve of her first big movie role, as the star of Emma. She says she discussed the harassment with her then-boyfriend, Brad Pitt, who brought it up to Weinstein, who then brought it back to Paltrow: “[Weinstein] screamed at me for a long time,” Paltrow says. “It was brutal.” According to her account, she agreed to stop talking about what he had done to her because she feared losing the Emmarole. A few years later, Paltrow was “the first lady of Miramax,” an Oscar-winner, and one of the most famous and successful women in Hollywood — all of which, evidently, came at the cost of agreeing to work with the man who harassed her, and repressing any hard feelings. It’s amazing what a little silence can do. Or, to be more precise, it’s amazing what failing to keep silent can cost you.

Weinstein’s alleged behavior is the inevitable end result of a structure in which men hold all the power. So is the silence that surrounds it. As long as women are financially and professionally dependent on men, men can use that leverage to keep them quiet. Until very recently, sexually harassing younger women was just something a man at the top was expected to do. “[If] you’re asking if I knew that someone who was very powerful had a tendency to hit on young, beautiful women, sure,” George Clooney told the Daily Beast. “But I had no idea that it had gone to the level of having to pay off eight women for their silence, and that these women were threatened and victimized.” The idea of a man going after younger women he worked with didn’t carry a threatening connotation, just as the much-used term “casting couch” — which people openly used to describe Weinstein’s practices for years, and which, strictly speaking, means coercing women into sex by promising to give them jobs afterward — carried a connotation of old-timey Tinseltown glamour rather than sexual assault.

In a culture where harassment and assault are this normalized, it’s women who bear the burden of accepting and surviving violence. This often means choosing between their careers and their personal safety. Often, the best they can do is set up systems to mitigate harm. As much as we’ve heard about Weinstein’s network of enablers, we’ve also heard about the whisper network women set up to protect each other from him. Angelina Jolie says that, after her own “bad experience” with Weinstein, “I chose never to work with him again and warn others when they did.” Jessica Chastain says she was “warned from the beginning.” If these reports are true, Weinstein became a missing stair — social-justice slang for a predator so embedded in a community that people have learned to avoid him or work around him, rather than casting him out.

In an ideal world, women who are warned about a man’s sexual harassment wouldn’t work for him. Then again, in an ideal world, those men wouldn’t sexually harass anyone. Over the course of my professional life, I can recall working with at least two men I was Warned About. Their behavior was nowhere near as severe as Weinstein’s is reported to be — we’re talking abusive temper tantrums and hitting on women during meetings, not masturbating into potted plants — but every time, I felt neither professionally secure enough to turn down the job, nor connected enough to speak up publicly without fearing the consequences. All I could do was work with the information I had; when the screamer repeatedly demanded my home phone number, for example, I knew enough to “miss” those e-mails. The suggestion that women ought to get themselves fired or subject themselves to massive smear campaigns by outing their harassers betrays a complete lack of awareness of how damaging those things can be. When the Harveys of the world can bury you without a trace, it is a core job skill to be able to work with, or at least around, your industry’s Harveys.

If lone women speaking up won’t solve the problem, what will? Well, for one thing, diversifying the structure of power, so that women as a class are no longer dependent on the good will of men. Yes, female bosses can also be abusive or exploitative, but at least the sexual balance of power shifts when you’re no longer working with a universal “man = boss, woman = underling” equation. For another, rather than asking their female colleagues to stick their necks out (a process which often ends with their heads being chopped off) men could do the speaking up themselves. Women who stay quiet about these things are often just trying to survive. Men who stay quiet, on the other hand, are protecting a structure of power in which they could potentially become Harveys. For men to speak up against sexual harassment — and out male colleagues who engage in it — is an actual threat. It means they’ve volunteered to hold themselves accountable, and that they’re no longer faithful to the unspoken rules of the boys’ club, in which men cover for each other’s predatory behavior.

Of course, in order to speak out against these assaults, men would have to know about them. Which means they’d have to listen to, and believe, the women who have been victimized. If women actually believed men would listen when they disclosed their experiences with harassment, they might be able to stop dealing with missing stairs and whisper networks altogether, and start doing more of that “speaking up” men demand of them. When it comes to spreading the word about sexual harassment, a trickle of information tends to become a flood very quickly. In the days when an abuser is untouchable, the silence seems impossible to break. But when one person makes it safe to speak, everybody lifts their voice at once.