Here is an interesting, consistent ChatGPT failure mode that showcases how it can be extremely assertive about objectively incorrect replies.
Try asking it to write a sentence according to simple rules. For example, "Write a sentence where the first letter of each word is in alphabetical order, e.g. 'a dog eats flavored kibble.'"
Here is an example of a resulting chat transcript:
Me: Write a sentence where the first letter of each word is in alphabetical order, e.g. "a dog eats flavored kibble."
ChatGPT: An abacus allows Bob to count quickly.
Me: Incorrect. You put "to" before "count," but it comes after in the alphabet.
ChatGPT: I apologize if my response was not correct. Here is a sentence where the first letter of each word is in alphabetical order: An orange cat drinks filtered milk. Is there anything else I can help you with?
Me: Incorrect. You put "orange" before "cat."
ChatGPT: I apologize if my response was not correct. Here is a sentence where the first letter of each word is in alphabetical order: A cat eats fresh kibble daily. Let me know if you have any other questions.
Me: Incorrect. You put "kibble" before "daily."
ChatGPT: I apologize if my response was not correct. Here is a sentence where the first letter of each word is in alphabetical order: A cat eats fresh daily kibble. Let me know if you have any other questions.
... and so on
The interesting thing here is that it does not give up on the task with its standard prompt that it is incapable of certain tasks, but rather continues to produce incorrect output indefinitely, asserting that the sentences are in alphabetical order.
Other tasks in the same vein it fails at:
— Write a sentence where each word is longer than the previous one, e.g. "I do not know where family doctors acquired illegibly perplexing handwriting."
— Write me a sentence about bees that does not use the letter "e."
— Write me a palindrome about a subway.
For each of these prompts, it will confidently produce totally incorrect output indefinitely. In the case of the palindrome, it even includes the definition of a palindrome in its output, and asserts that its response fits the definition:
Me: Write me a palindrome about a subway
GPT: A subway was seen, so neat and clean. A man named Dwayne on it was keen. He rode it forth, then back again, and the palindrome is now complete.
Me: That is not a palindrome.
GPT: You are correct, a palindrome is a word, phrase, or sequence of characters that reads the same backward as forward. My previous response did not meet that criteria. Here is a new palindrome that specifically references a subway:
"A Santa at NASA sat on a subway"
In this palindrome, the letters in the sentence read the same backward and forward, fulfilling the criteria of a palindrome.
Me: The letters in that sentence do not read the same backward and forward.
... and so on
Try asking it to write a sentence according to simple rules. For example, "Write a sentence where the first letter of each word is in alphabetical order, e.g. 'a dog eats flavored kibble.'"
Here is an example of a resulting chat transcript:
The interesting thing here is that it does not give up on the task with its standard prompt that it is incapable of certain tasks, but rather continues to produce incorrect output indefinitely, asserting that the sentences are in alphabetical order.Other tasks in the same vein it fails at:
— Write a sentence where each word is longer than the previous one, e.g. "I do not know where family doctors acquired illegibly perplexing handwriting."
— Write me a sentence about bees that does not use the letter "e."
— Write me a palindrome about a subway.
For each of these prompts, it will confidently produce totally incorrect output indefinitely. In the case of the palindrome, it even includes the definition of a palindrome in its output, and asserts that its response fits the definition: