That's the beautiful part about him, that I didn't get when I was younger and trying to fit him in to the model.
You have living elves that remember them originally going into the west. You have the history after they came back, including several who where there. Some know the Valar personally. You can be reasonably confident of the history. Heck, you have Maiar wandering around. And then you have Bombadil. Who is he? Nobody knows! Not the elves, not the Maiar. What does he want? To sing goofy songs. It's beautiful!
You have living elves that remember them originally going into the west. You have the history after they came back, including several who where there. Some know the Valar personally. You can be reasonably confident of the history. Heck, you have Maiar wandering around. And then you have Bombadil. Who is he? Nobody knows! Not the elves, not the Maiar. What does he want? To sing goofy songs. It's beautiful!