In ye olden Broadway days, the days of Gershwin, Berlin, Cohan, Rodgers and Hart, Cole Porter - in the glory days of Tin Pan Alley - Broadway musicals dictated the popular songs of the day. Especially prior to the collaboration of Rodgers and Hammerstein on Oklahoma!, which started the trend of integrating songs more thoroughly in character and plot, most showtunes of the early 20th century were generic enough in their relation to plot that they were easily extracted for a larger audience (and even almost frighteningly interchangeable within the show itself - as evidenced in film adaptations like Anything Goes or Babes in Arms, which rewrote plots entirely while maintaining the original score).
I hasten to add, before Jerome Kern takes a hit out on me, that these songs were well-written. Many of them are still considered classics, long after the shows originally containing them are forgotten and unrevived. [Did you know "The Lady is a Tramp" is from Babes in Arms, though it was inserted into the film Pal Joey? Did you know it was written for a woman? You did? Good for you, now go sit in the corner; I'm still talking] Other fun examples include "My Funny Valentine," "Bewitched, Bothered and Bewildered," "I Get a Kick Out of You," "Night and Day," "Smoke Gets in Your Eyes," etc. These were good songs, these were popular successful songs, and they were written for the theater (these scribes also found success composing songs for film musicals as well, including hits like "Puttin' on the Ritz" and "The Way You Look Tonight."
While the musicals of Rodgers and Hammerstein did not signal an abrupt end to the designation of showtunes as popular music, they were the first cobblestones that led down that eventual path. As the songs in musicals became more and more integrated in plot, they became harder and harder to extract with any hope of a popular life outside the Broadway venue. Eventually it got to the state where perhaps one song would be considered the "breakaway hit," and often when a show would be adapted to a film, a new song would be written to fill that need, both to entice a cover by a pop artist, as well as to maintain eligibility at the Academy Awards. By the time the 90s rolled around (aka, when I actually became aware of such shenanigans as "Broadway" and "pop songs") the idea of a showtune being played on the radio was laughable - the best shot was the I Want song or the love ballad from whatever Disney film was premiering that year.
Sadly, this meant that I was terminally uncool, as showtunes were the majority of my listening experience. Whatever, radio station K92! I don't need your approval!
Songwriters over the past few decades have tried to buck this stigma. Baron Andrew Lloyd Webber (he's a baron now! A BARON. I don't even.) composed a number of quite popular sung-through pop operas, containing their own handfuls of breakaway songs. Frank Wildhorn appeared on the scene in the late nineties to try to continue this endeavor, but all his attempts were met with scathing hostility from the New York critics (Webber, though popular with the "plebes," is also grandly sneered at by aficionados and music scholars). Jonathan Larson seemed poised on the brink of bridging Broadway and rock in a way Tommy and Hair had not quite achieved, and might have been able to do much more for that union, but he tragically passed on the eve of Rent's phenomenal success.
And then the aughts came and gave Broadway a new shiny gift: the Jukebox Musical. No, it didn't originate in the aughts. Jukebox Musicals appeared now and again through the past several decades. But Mamma Mia's explosive success in 1999 in London and then New York reignited the flame and made it a popular venture for Broadway producers searching for a money-making show. Jukebox Musicals are another big target for Broadway's Protectors, despite their frequent success [for those taking issue with me that a huge pile of Jukebox Musicals have bellyflopped - which they have - the fact that MM is still running and Jersey Boys took home the Tony means that this genre isn't going away any time soon, so we may as well call it successful. I can also point slyly at the popularity of Moulin Rouge and Glee]. Land on the right formula with your Jukebox Musical, and you've got good revenue for the next several years (at least).
People (well, those in love with "The Golden Age of Broadway") decry the presence of the Jukebox Musical as the true death of Broadway musicals. [for those wanting the editor's honest opinion here? I don't care for Jukebox Musicals that much myself. I think it's lazy writing, crutching heavily on the audience's already-in-place affection for the songs. I like a good clever original score, integrated into a well-told multidimensional story with fleshed-out characters. But whether I like them or not is not the point here.] But they also called Broadway's death when Lloyd Webber and Wildhorn kept churning out their pop musicals. They said the same thing when books were being adapted into musicals (especially non-cheery ones like Les Miserables or Jane Eyre). They said the same thing when films were being adapted into musicals. They said the same thing when they started mic'ing all the actors. The truth is, people like to predict the end, and look back fondly at days past as "The Golden Age," assuming everything and anything written back then is by definition superior to anything we can produce now, which frankly is utter bollocks. The only way to have a perfect Golden Age past is to forget or conveniently ignore all the mediocrity that was also meandering around at the time. I shudder to think what they'll recall and forget about this decade in fifty years' time.
Moving on: What no one seems to have noticed about the Jukebox Musical trend is the actual renaissance it has achieved. This sub-genre, more than any other aspect of modern musical theater, is a return to Broadway's origins: Popular Music and the songs you hear on a Broadway stage are now again one and the same. Their actual relation to plot and character is once again sketchy and interchangable. We're back in the days of Tin Pan Alley - they just did it in reverse, using songs that were already popular and inserting them into a shallow plot with one-dimensional characters. The style and format of the songs themselves has changed, but their use and aim is exactly the same.
We're back to Old School Broadway. We just did it backwards, choosing songs that have already been written for other purposes.
In discussing this exciting new thesis of mine with others, I discovered a funny contradiction: when I mentioned my comparison to someone who prefers Broadway and the theater to pop music, she remarked that while they were doing similar things, the songs being used now in Jukebox Musicals aren't as good - the quality has gone down; however, when I discussed it with someone who doesn't like musicals much in general, he remarked that the difference now is "integrated" musicals don't have good or memorable songs anymore, and that already-existing pop songs are their best bet. My response to both of them was that it's really a subjective thing, the arguable quality (though my second conversationalist is, ahem, wronger) of one form of songwriting versus another.
The plus side of all this is that Broadway has a slight potential to land back somewhere in the popular realm. The minus side is a distinct downslide in overall quality of story and character being portrayed. Broadway is risking losing its substance, something it worked hard to acquire over time, especially sitting around as the bastard redheaded stepcousin of the entertainment industry. Broadway is, when we've reached the bottom line, a business - something that has to make more money than it loses to survive. But if all art and heart get lost, what's left to keep fighting for? Broadway has been struggling for decades to reconcile these two aims (but they're not alone here; so has Hollywood).
My optimistic (foolhardy?) conclusion:
I heard an actor say during a closing night curtain call speech, in regard to our current Broadway climate, that there's room for all of us at the table (I paraphrase). He also remarked that he thinks we're in a Golden Age right now and that fifty years from now we'll look back and say we were lucky to be here. Jukebox Musicals are not high art, but they're not the only thing on the table. There's a lot to choose from.