"education and knowledge are products just like any other"
Not in a democracy. For a democratic system to work, the entire population needs to be educated well enough to make a rational choice when they are asked to vote.
If you want a system where knowledge is a "product," what you want is a plutocracy: a system where the wealthy rule, because only the wealthy are educated enough to rule. I cannot speak for you, but I live in a country that was founded on a principle of representative democracy and which has a constitution that is meant to prevent the establishment of any sort of aristocracy, plutocracy included.
"it is necessary to ensure that others invest the time and incur the opportunity cost of doing things that further human knowledge"
That is not what the subscription fees for academic journals are for; the authors of scientific articles are almost never paid by the publishers who collect fees for access to those articles. The purpose of journal fees is and has always been to monetize the publishing industry itself; once upon a time, before there was an Internet or a Web, that was the best known method of mass-distributing scientific journals and other academic writing.
The "opportunity cost" of research is paid for by the government and by the researchers themselves. The overwhelming majority of professional researchers are being paid less to do their research than they would to work in private industry. Graduate students, whose work constitutes a sizable portion of the research work that is done, are paid almost nothing -- barely enough to live on. There is no "return on investment" to speak of; all a researcher gains from publishing a paper, in strict economic terms, is a padded CV that might help that researcher advance their career (but which will do little to help them make as much money as they would in a corporate job). Researchers are willing to forgo a higher salary for various reasons, but there is a common theme, which is a high philosophical ideal: the work is more intellectually interesting, there is a chance to contribute to the greater pool of human knowledge, working as a university professor allows one to impart knowledge onto the next generation, there are bigger problems to solve than what corporations are doing, etc.
"I would rather that research and knowledge is entirely funded by private money and the private sector than by government fiat."
There are two problems with that:
1. The private sector tends to focus education on vocational training. There is certainly a place in the world for such training, and we could not have a society of specialized workers without some form of vocational training, but we need more from our education system than job training. Again, if we want to even pretend that we are a democracy, we have to at least have a populace that is able to read and understand the implications of major political issues.
2. Private sector research tends to shy away from big, risky lines of work that may have no payoff at all. There are few exceptions to this rule; the most prominent is Microsoft Research, which is one of the only "academic" research labs in private industry and which is a place where researchers are doing things nobody else is willing to touch. The development of things like the Internet, nuclear power, the space program, genome mapping, and other "big ideas" would almost certainly have not been possible without government funding, because the key, foundational work of these things had no commercial value whatsoever, with commercial value being realized only after a great deal of risky research.
"failure to publish may mean losing your job but then that is because publish is part and parcel of your job"
Except that people are not paid to "write papers," they are paid to "do research," which is supposed to be reported on by writing papers. Researchers who are under pressure to publish more papers can always use "old tricks" like taking a single good result and splitting it into multiple small results that can be published one at a time, or publishing a good result and then publishing multiple tiny variations on that result. As I noted above, researchers do not generally choose their careers because of the pay; it is equally true that researchers do not choose their careers because they are excited by the idea of publishing a lot of journal articles.
"this still doesn't constitute coercion"
I think you should look up the definition of "coercion" in the dictionary:
Threatening someone with unemployment if they fail to do a particular thing is coercion by definition.
"validation in an "industry" where there are few objective measures of a person's relative performance as a researcher"
Counting the number of journal articles a person published is not even a remotely objective method of judging their performance as a researcher. It is on the level of counting the number of lines of code a person writes as a measure of their performance as a programmer. The only reason it is used as a measure of anything is because the people in charge of hiring decisions often lack the time or expertise needed to closely examine a researcher's work, so they just combine the number of papers published with the perceived prestige of the journals those papers were published in and use that as an approximation. I have seen good work get rejected from top journals, and I have seen mediocre work get accepted to those same journals, just like I have seen 10 lines of code that does more than 1000 lines of code in the same language.
I doubt there is a market-based solution to this, because it is not an economic problem. It is an academic problem that stems from the policies and approach we take to judging the quality of research for employment purposes. When a university is considering hiring a professor, they do not ask their faculty to read that professor's published work thoroughly; instead, they ask the faculty to judge whether or not that professor will bring in grant money (it is unlikely that private sector funding would truly change this; you see similar patterns of behavior out of the private sector, on both sides), if that professor can increase or maintain the school's prestige, etc. Teaching is low on the list of priorities, and a person who spends all their time chasing after big, difficult, and risky problems has a lower chance of being hired than a person who follows tried-and-true approaches to research strategy.
Not in a democracy. For a democratic system to work, the entire population needs to be educated well enough to make a rational choice when they are asked to vote.
If you want a system where knowledge is a "product," what you want is a plutocracy: a system where the wealthy rule, because only the wealthy are educated enough to rule. I cannot speak for you, but I live in a country that was founded on a principle of representative democracy and which has a constitution that is meant to prevent the establishment of any sort of aristocracy, plutocracy included.
"it is necessary to ensure that others invest the time and incur the opportunity cost of doing things that further human knowledge"
That is not what the subscription fees for academic journals are for; the authors of scientific articles are almost never paid by the publishers who collect fees for access to those articles. The purpose of journal fees is and has always been to monetize the publishing industry itself; once upon a time, before there was an Internet or a Web, that was the best known method of mass-distributing scientific journals and other academic writing.
The "opportunity cost" of research is paid for by the government and by the researchers themselves. The overwhelming majority of professional researchers are being paid less to do their research than they would to work in private industry. Graduate students, whose work constitutes a sizable portion of the research work that is done, are paid almost nothing -- barely enough to live on. There is no "return on investment" to speak of; all a researcher gains from publishing a paper, in strict economic terms, is a padded CV that might help that researcher advance their career (but which will do little to help them make as much money as they would in a corporate job). Researchers are willing to forgo a higher salary for various reasons, but there is a common theme, which is a high philosophical ideal: the work is more intellectually interesting, there is a chance to contribute to the greater pool of human knowledge, working as a university professor allows one to impart knowledge onto the next generation, there are bigger problems to solve than what corporations are doing, etc.
"I would rather that research and knowledge is entirely funded by private money and the private sector than by government fiat."
There are two problems with that:
1. The private sector tends to focus education on vocational training. There is certainly a place in the world for such training, and we could not have a society of specialized workers without some form of vocational training, but we need more from our education system than job training. Again, if we want to even pretend that we are a democracy, we have to at least have a populace that is able to read and understand the implications of major political issues.
2. Private sector research tends to shy away from big, risky lines of work that may have no payoff at all. There are few exceptions to this rule; the most prominent is Microsoft Research, which is one of the only "academic" research labs in private industry and which is a place where researchers are doing things nobody else is willing to touch. The development of things like the Internet, nuclear power, the space program, genome mapping, and other "big ideas" would almost certainly have not been possible without government funding, because the key, foundational work of these things had no commercial value whatsoever, with commercial value being realized only after a great deal of risky research.
"failure to publish may mean losing your job but then that is because publish is part and parcel of your job"
Except that people are not paid to "write papers," they are paid to "do research," which is supposed to be reported on by writing papers. Researchers who are under pressure to publish more papers can always use "old tricks" like taking a single good result and splitting it into multiple small results that can be published one at a time, or publishing a good result and then publishing multiple tiny variations on that result. As I noted above, researchers do not generally choose their careers because of the pay; it is equally true that researchers do not choose their careers because they are excited by the idea of publishing a lot of journal articles.
"this still doesn't constitute coercion"
I think you should look up the definition of "coercion" in the dictionary:
http://dictionary.reference.com/browse/coercion
Threatening someone with unemployment if they fail to do a particular thing is coercion by definition.
"validation in an "industry" where there are few objective measures of a person's relative performance as a researcher"
Counting the number of journal articles a person published is not even a remotely objective method of judging their performance as a researcher. It is on the level of counting the number of lines of code a person writes as a measure of their performance as a programmer. The only reason it is used as a measure of anything is because the people in charge of hiring decisions often lack the time or expertise needed to closely examine a researcher's work, so they just combine the number of papers published with the perceived prestige of the journals those papers were published in and use that as an approximation. I have seen good work get rejected from top journals, and I have seen mediocre work get accepted to those same journals, just like I have seen 10 lines of code that does more than 1000 lines of code in the same language.
I doubt there is a market-based solution to this, because it is not an economic problem. It is an academic problem that stems from the policies and approach we take to judging the quality of research for employment purposes. When a university is considering hiring a professor, they do not ask their faculty to read that professor's published work thoroughly; instead, they ask the faculty to judge whether or not that professor will bring in grant money (it is unlikely that private sector funding would truly change this; you see similar patterns of behavior out of the private sector, on both sides), if that professor can increase or maintain the school's prestige, etc. Teaching is low on the list of priorities, and a person who spends all their time chasing after big, difficult, and risky problems has a lower chance of being hired than a person who follows tried-and-true approaches to research strategy.