@inproceedings{miyamoto-etal-2023-dynamic,
    title = "Dynamic Structured Neural Topic Model with Self-Attention Mechanism",
    author = "Miyamoto, Nozomu  and
      Isonuma, Masaru  and
      Takase, Sho  and
      Mori, Junichiro  and
      Sakata, Ichiro",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.366",
    doi = "10.18653/v1/2023.findings-acl.366",
    pages = "5916--5930",
    abstract = "This study presents a dynamic structured neural topic model, which can handle the time-series development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a self-attention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.",
}

This study presents a dynamic structured neural topic model, which can handle the time-series development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a self-attention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.
Miyamoto, Nozomu Isonuma, Masaru Takase, Sho Mori, Junichiro Sakata, Ichiro
Dynamic Structured Neural Topic Model with Self-Attention Mechanism