Modern Hollywood has every reason to assume the audience are idiots - every time a Transformers makes a billion dollars, a French Connection loses its wings.
There are a number of other reasons, too - in the 70's most studios were still independently owned. (Corporations were just starting to buy studios right about then.) So there was still a guy or a couple of guys with the power to say "that won't make a dime, but it'll be a great picture and we should make that" or maybe "oh what the hell, let's make that and see what happens".
Another factor - in the 70's studios were in deep deep financial trouble (hence the corporate buyouts) and were more willing to take risks in general, because the old stuff wasn't working. Big expensive epics and musicals were tanking and crazy shit like Easy Rider were massive hits. So it was a time when studios let young lunatics like Scorcese and Coppola and Friedkin make movies because what the hell, right?
Even after the corporate buyouts - when movie studios became just a minor division of companies that mostly made jet engines or Coca-Cola or whatever - it took a while for the corporate mentality to really take hold. A lot of people credit/blame Frank Wells (who became president of Disney in the '80's) for popularizing the model that all the studios use now.
Wells shocked everyone by moving the Disney marketing department into the executive studio offices. Before Wells, a studio marketing department would get the order "we made another movie, so figure out how to sell it." Wells changed that to "we're thinking of making a movie, figure out if it will sell".
It was in this era that Katzenberg wrote his infamous "Some Thoughts on Our Business" memo. If Eisner had appointed Katzenberg president when Wells died in 1994, maybe movie history would have gone a different way.
But because of Wells' new policy - or in spite of it, who knows - Disney under Wells became hugely successful and eventually other studios started working the same way. And still do to this day, even though half the time the marketing department is spectacularly wrong about what will sell.
But what do you expect them to do - go back to just making movies that might be good? That's just crazy talk.