The book American Psycho has several scenes that people list among the most disturbing. The starved rat, for instance, is commonly listed as one of the most twisted things described in literature.
Now, I don't know how Mr. Ellis felt about it while he was writing, but it was certainly risky to write such graphic scenes. Even fans of Horror have their limits and when you write scenes that push boundaries, it's very possible that you exceed those limits. What happens then? People close your book and don't come back.
It's a calculated risk though. As the saying goes, "People will forget what you said, but they will never forget how you made them feel." Even some people who close the book on your scene will eventually make their way back. They may still have a need to know what happens next, or they may remember your name and try a different work.
Even if they never come back to your work, they'll remember the one they stopped at, and likely your name. If the subject of horror, or disturbing things ever comes up, you can bet your name will come up, which is one of the things you want as an author. Word of mouth is just about the best advertising you can hope for.
Still, it's a fine line, and like almost everything else in writing, it comes down to how well it's executed. If you build up a story, with characters your readers care about, escalating tension and guiding the story in a way where the horrific scene makes some kind of sense. Then the aftermath has to make sense as well.
Recently, there was a video game that was released. Outlast 2. I must confess, I haven't played it myself. I watched people do Let's Plays on youtube though. (Possible spoilers below the image!)
Outlast 2 hits on a lot of disturbing themes. They're trying to push the boundaries all over the place. There's hints of religious extremism, infanticide, demonic rituals, murder, torture, and rape. I'll be honest, just watching someone else play the game at one point left me fucking triggered. As pissed as that scene made me, I kept watching, but then, the ending. The ending was completely unsatisfying. It didn't even feel like an ending, it was completely open-ended.
And I now have no intention of buying and playing Outlast 2, even though I loved the first game.
People have their own limits though, as I said. So what is too far for some might not be for others, who might absolutely love that you went there.
So, when you're working on a piece and a scene seems to call for something hardcore, it's up to you whether to take the risk
It's usually not a bad thing to challenge yourself and your own boundaries though.
~ Shaun