Would something like AWS SQS not scale for something like this? We currently push about 25k daily transactions over SQS, obviously no where near the scale of this, just wondering about what limitations we will bump into potentially.
The limitations are most likely on price. For the 200B messages they've already processed in the last 3 months, that would be $100,000 total on just the SQS FIFO queue, or $33,333 per month. And that's not counting data transfer.
As long as everything is in ec2 data transfer will be free. You're cost calculations are also off base. You'll need to send, receive and delete every message that you process via SQS. These can all be done in batches of 10. So it's 200B * 3/10 * .50 / million, which comes out to 60k over 3 months. Still not cheap, Kinesis is probably the better option in this case if you want an AWS managed service.
SQS queues deduplicate over a 5 minute window. This is claiming a much larger window.
Either way your listener(s) still has to have its own deduplication. Ensuring a message ends up on the queue only once, and ensuring it's processed exactly once, are two different problems that require separate handling (and, the former is what most out of the box systems claim to solve, while the latter is more important, and, frankly, completely negates the need of the former).
Oh, right, they added their FIFO queues. My bad; thanks for the correction. Worth noting, though, that AWS's own services can't talk to FIFO queues. If you want to wire up SNS or Lambda dead-letter queues to a FIFO queue, you are out of luck.
