Towards curriculum learning of multi-document summarization using difficulty-aware mixture-of-experts

Abstract: With the advent of information explosion, automatic multi-document summarization has attracted widespread attention from the natural language processing. Multi-document summarization (MDS) poses challenges with larger search and problem spaces, since lengthy inputs often contain varying degrees of redundancy and contradiction. To tackle these challenges, we propose a novel difficulty-aware framework that enhances MDS by integrating curriculum learning paradigms into mixture-of-experts architectures. During the curriculum learning stage, a multi-document difficulty metric works in tandem with the curriculum scheduler to train experts specialized in distinct problem subspaces. We first employ the proposed multi-document difficulty metric to partition the MDS dataset into subsets with distinct levels of difficulty. Then, we adopt the difficulty-aware curriculum arrangement to train specialized expert models on different subsets. During the mixture-of-experts stage, we utilize a difficulty-aware mixture-of-experts structure to combine different models for improving multi-document summarization. Extensive experimental results on two MDS datasets indicate that the proposed method achieves state-of-the-art performance among length-constrained methods and delivers competitive results compared to other strong baselines with greater parameters. These results demonstrate the effectiveness of combining curriculum learning and mixture-of-experts, implying a promising direction for multi-document summarization.

Keywords: Multi-document summarization; Text summarization; Curriculum learning; Mixture-of-experts; Natural language processing

Recommended citation: Zhang M, Cheng L, Guan W, et al. Towards curriculum learning of multi-document summarization using difficulty-aware mixture-of-experts[J]. Applied Soft Computing, 2025: 114088.
Download Paper