Ive been working on something similar, have you had any issues with the LLM not giving you back a full response for all the input? Ive been using Chat gpt but even on the same request sometimes id give it 20 things to rank, and Id just get back 3 results
No, it's been working without problems so far. I'm using Anthropic for what it's worth. I ask the LLM to first do some reasoning and then return a JSON array on the final line. Sometimes I've seen some Markdown backticks there, but no irregularities more than that.
