(Hi Simon, I am laughing as I write - I just submitted an article from your blog minutes ago. Then stumbled into this submission, and just before writing this reply, I checked the profile of "simonw"... I did not know it was your username here.)
Well, assuming one normally queries for information, if the server gives false information then you have failure and risk.
If one were in search for supplemental reasoning (e.g. "briefing", not just big decision making or assessing), the server should be certified as trustworthy in reasoning - deterministically.
It may not be really clear what those «plenty [] other tasks that they ARE useful for» could be... Apart from, say, "Brian Eno's pack of cards with generic suggestion for creativity aid". One possibility could be as a calculator-to-human "natural language" interface... Which I am not sure is a frequent implementation.
Many of the most interesting uses of LLMs occur when you move away from using them as a source of information lookup - by which I mean pulling directly from information encoded into their opaque model weights.
Anything where you feed information into the model as part of your prompt is much less likely to produce hallucinations and mistakes - that's why RAG question answering works pretty well, see also summarization, fact extraction, structure data conversion and many forms of tool usage.
Uses that involve generating code are very effective too, because code has a form of fact checking built in: if the model hallucinates an API detail that doesn't exist you'll find out the moment you (or the model itself via tools like ChatGPT Code Interpreter) execute that code.
Well, assuming one normally queries for information, if the server gives false information then you have failure and risk.
If one were in search for supplemental reasoning (e.g. "briefing", not just big decision making or assessing), the server should be certified as trustworthy in reasoning - deterministically.
It may not be really clear what those «plenty [] other tasks that they ARE useful for» could be... Apart from, say, "Brian Eno's pack of cards with generic suggestion for creativity aid". One possibility could be as a calculator-to-human "natural language" interface... Which I am not sure is a frequent implementation.