The direct answer is "because they can". Demand for journals is pretty inelastic - increasing the price doesn't cause much of a fall in demand. Can you imagine your university not subscribing to a journal like The Lancet or Science?
Behind that answer however is a storm of interlinked factors. Here're some of them:
- Publishers are under pressure from their shareholders to show revenue and income growth. This is no different from university administrators pressuring their faculty to show research output growth, university ranking growth, and so on.
- The total number of academic papers being published worldwide is increasing quickly. To shareholders this sounds like an excellent opportunity for growth! But not all these papers can be published in top journals, so a lot of them end up in not-so-good journals.
- Most of the papers in these not-so-good journals are actually very seldom used. For example take the Elsevier journal Wave Motion. It's got an impact factor of 1.575, which is pretty medium. I picked a random paper, "An elastodynamic computational time-reversal method for shape reconstruction of traction-free scatterers", Vol 72, page 23-40, July 2017. Then I went to Web of Science, found the paper, and checked the usage statistics. Guess how many times it's been used in the past 180 days - three times (!).
- Since the usage is so low, libraries are incentivized not to subscribe (and remember, this is a solidly middle-of-the-road journal - imagine what it's like for the bottom-tier ones). But if libraries don't subscribe, the journal makes a loss. What is the publisher to do?
- Let's take the obvious thing to do, and one commonly argued for, which is to close the journal. If publishers did this en masse, there'd suddenly be no more journals except the few top-tier ones. Remember that the papers published in these not-so-good journals aren't necessarily bad - they're usually just rather boring, low-novelty papers: the backbone of science. Without the journals, these papers would never be published.
- Another thing about these not-so-good journals is that they're crucial to researchers from developing countries. It's true that these researchers might be slightly less capable than those in developed countries, but it's also probable that they simply have fewer resources. If one is not able to access the Large Hadron Collider, one is also not likely to make breakthroughs in particle physics. Hence one ends up with boring, low-novelty papers that won't make it to top journals.
- On top of all that, let's add the fact that some academics think that they write papers, they edit journals, they serve as reviewers, they deserve to be paid for it. I'm not saying that they're wrong, but you can see the tension this belief causes.
And then there's open access. Most forms of OA ties the revenue to the number of papers published (there are other models, but those journals are loss-making and only stay afloat because of an external source of funding). This means there's an immediate conflict of interest for the publisher, which is why there's predatory OA publishing, and in turn is why there's a backlash against OA. Also consider what it's like from the publisher's and library's points of view.
- Publisher. If you convert a journal to OA, you're giving up subscription revenue. A bird in hand is worth two in the bush. Further, many academics prefer to publish in subscription journals even if they have access to OA funding, since if they publish OA they have to jump through some administrative hoops. There're also lots of academics in developing countries that simply cannot pay for OA. There's no guarantee that your 100-papers/year journal will still be 100-papers/year after you convert to OA. Do you dare? Remember your job is on the line (see comment above about shareholders).
- Library. OA costs money. You need to free up funds somewhere to fund OA. In theory you should be able to use subscription money to pay for OA, but in practice, you're either subscribed to a journal or not. You can't just pay $3000 less for a subscription and use that for OA for a single article. The only way is to unsubscribe, which effectively removes access to every paper in the journal.
Finally, let's not forget that publishing is a notoriously poorly-paid job, and the publisher also has to keep its employees happy.
tl; dr: it's a difficult world out there. If you have a solution, you'd be in contention to win the Nobel Prize in Publishing.
