I recently watched "I Am Big Bird: The Caroll Spinney Story" documentary and I had this impression that the show shifted towards a younger audience with Elmo. I wonder that this was probably related to a bigger audience on this age range.
I understood the documentary to say that Sesame Street noticed that their audience was getting younger. In response to this age shift they introduced Elmo; a character that resonated to this new growing demographic.