It would be wonderful if we could get some standardized language for defining test cases, and some tooling for converting to working (or mostly working) code for your language and implementation (at the API level, at least). If publicly developed per protocol/library, I think it would yield a substantial benefit for robustness of common protocols we have.
Want to develop a DNS library or server? Read the RFCs and download the public test suite (which may get regular updates as people fill out corner cases, put in tests for common bugs/exploits, etc). Occasionally run public test suite plus any of your own tests you have added to confirm things function as you expect. There's so much duplication of effort in this area currently, and i'm sure the majority of test suites for related applications have at least a few tests that would be useful to others.
I agree it would be useful to have. There's all kinds of stuff like that in academia. So many different forms and tools that the problem would likely be the time available to assess them. On top of nobody using or replicating them as usual. :)
Note: sklogic did mention doing something like that by making sure all the programs' had a scriptable API then using basic, shell input and output to test them. There were acceptance, testing tools I used a long time ago that did something similar using XML. Not sure the efficiency of this method but it should work.
Want to develop a DNS library or server? Read the RFCs and download the public test suite (which may get regular updates as people fill out corner cases, put in tests for common bugs/exploits, etc). Occasionally run public test suite plus any of your own tests you have added to confirm things function as you expect. There's so much duplication of effort in this area currently, and i'm sure the majority of test suites for related applications have at least a few tests that would be useful to others.