1a. generate two numbers using: (random() % BIGNUM)
1b. ask the LLM to multiply them together
Any human who has learned multiplication can do this. AFAIU, LLMs cannot unless the computation exists within the training set. They have zero arithmetic reasoning capability.
I just asked ChatGPT: "19191920 multipled by 10292111772"
It said:
--------------
To multiply 19191920 by 10292111772, you can use the standard long
multiplication method as follows:
19191920
x 10292111772
-------------
19191920000 (the product of 19191920 and 1)
153535360000 (the product of 19191920 and 8)
1535353600000 (the product of 19191920 and 2)
-------------
196837644266310240 (the final product)
Therefore, the result of multiplying 19191920 by 10292111772 is 196837644266310240.
----------------------------------
This is completely wrong. It is not doing arithmetic, and it is not capable of doing arithmetic.
Many humans would not be able to solve that problem, especially those that are younger or have lower IQs, and obviously those that have not not been taught multiplication. That doesn't mean they are devoid of all reasoning ability.
Furthermore GPT-4 is able to multiple slightly shorter pairs of numbers, and experiments on smaller neural nets, such as one using a dataset consisting of 20% of all combinations of two four digit numbers, quickly learn to generalize to successfully multiply any four digit pair even though they aren't in the training set.
Try again, give an answer that actually works this time.
Any human who has learned multiplication can do this. AFAIU, LLMs cannot unless the computation exists within the training set. They have zero arithmetic reasoning capability.
I just asked ChatGPT: "19191920 multipled by 10292111772"
It said:
--------------
----------------------------------This is completely wrong. It is not doing arithmetic, and it is not capable of doing arithmetic.