Investigating the Effects of Sparse Attention on Cross-Encoders
CoRR(2023)
摘要
Cross-encoders are effective passage and document re-rankers but less
efficient than other neural or classic retrieval models. A few previous studies
have applied windowed self-attention to make cross-encoders more efficient.
However, these studies did not investigate the potential and limits of
different attention patterns or window sizes. We close this gap and
systematically analyze how token interactions can be reduced without harming
the re-ranking effectiveness. Experimenting with asymmetric attention and
different window sizes, we find that the query tokens do not need to attend to
the passage or document tokens for effective re-ranking and that very small
window sizes suffice. In our experiments, even windows of 4 tokens still yield
effectiveness on par with previous cross-encoders while reducing the memory
requirements to at most 78
for passages / documents.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要