The observation about the "block-diagonal patterns" in RAG isn't new and has been exploited / explored before:

- https://arxiv.org/abs/2410.07590 (literally titled "Block-Attention for Efficient RAG")

- https://arxiv.org/abs/2409.15355v3

- https://arxiv.org/abs/2212.10947

The REFRAG paper does not cite any of these.