AI2 at ACM UIST 2023
New Intelligent Reading Interfaces Research and The Semantic Reader Open Research Platform
This year at the 36th Annual ACM UIST Conference, researchers from AI2 will present papers on two new AI-powered intelligent reading interfaces for scholarly documents — Papeo and Synergi. Coinciding with the conference, we are also releasing to the broader research community The Semantic Reader Open Research Platform, a set of open-source libraries that powered the development of Papeo, Synergi, and a total of more than 10 reading interfaces research we have published in recent years.
Highlighted UIST Papers from AI2 Researchers
(*AI2 affiliation, +Work completed during a research internship at AI2)
Papeos: Augmenting Research Papers with Talk Videos
Tae Soo Kim+, Matt Latzke,* Jonathan Bragg,* Amy X. Zhang,* Joseph Chee Chang.*
Read more about Papeos, including interactive demos and talk videos.
Research consumption has largely involved reading academic papers, which are often static, dense, and formally written. Alternatively, pre-recorded conference presentation videos have recently become more widely available. In contrast to research papers, talk videos offer a more dynamic, concise, and colloquial style for conveying research. Despite their complementary benefits, talk videos are potentially under utilized after the conferences.
Our research delves into the potential advantages of blending academic papers with talk videos to create a more engaging and dynamic research consumption experience. Through formative and co-design studies, we designed Papeos, a novel reading and intelligent authoring interface that allow authors to augment their papers by segmenting and localizing talk videos alongside relevant paper passages. Papeos enables readers to visually skim a paper through clip thumbnails, and fluidly switch between consuming dense text in the paper or visual summaries in the video. Our comparative lab study showed that Papeos reduced mental load, scaffolded navigation, and facilitated more comprehensive reading of papers.
Synergi: A Mixed-Initiative System for Scholarly Synthesis and Sensemaking
Hyeonsu B. Kang, Sherry Tongshuang Wu, Joseph Chee Chang,* Aniket Kittur.
Read more about Synergi, including demo and talk videos.
Efficiently reviewing and weaving together scholarly literature is crucial for advancing scientific knowledge. However, the ever-expanding volume of publications and the weight of accumulated knowledge pose a formidable challenge to the synthesis of research threads in the literature. While significant research has been devoted to helping scholars interact with individual papers, building research threads scattered across multiple papers remains a challenge. One core challenge here is that top-down synthesis methods and large language models have their limitations in terms of personalization and adaptability, whereas bottom-up synthesis is laborious and time-consuming.
In response to these challenges, we introduce an innovative approach involving mixed-initiative workflows. We introduce Synergi, a novel computational pipeline and interface that allows users to select descriptions of cited papers in a reading interface as seeds, and exploit both the larger citation graphs and LLMs to expand and summarize an additional set of relevant research threads and papers for users to explore. Synergi expowers scholars to start with an entire threads-and-subthreads structure generated from papers relevant to their interests, and to iterate and customize on it as they wish. Our evaluation showed that Synergi helps scholars efficiently make sense of relevant threads, broaden their perspectives, and increase their curiosity. We discuss future design implications for thread-based, mixed-initiative scholarly synthesis support tools.
The Semantic Reader Open Research Platform
More broadly, Papeos and Synergi are a part of the Semantic Reader Project, a collaborative effort of NLP + HCI researchers from non-profit, industry, and academic institutions to create AI-powered interactive and intelligent reading interfaces for scholarly papers. Our research led to the creation of Semantic Reader, an application that incorporates novel research features as they mature, and is used by tens of thousands of scholars each week on the Semantic Scholar website.
Finally, coinciding with UIST 2023, we are also releasing The Semantic Reader Open Research Platform. With the platform, we open source NLP toolkits and React UI libraries that we have developed to power more than a dozen intelligent reading interface prototypes. More specifically, we are releasing PaperMage (to appear at EMNLP 2023), a library for processing and analyzing scholarly PDFs, and PaperCraft, a React UI component for building augmented and interactive reading interfaces. We also included documentations on how to use these resources to build research prototypes. For example, on the website you can find a detailed tutorial on building Paper Plain, an intelligent reading interface with in-situ term definitions, passage summaries, and suggested key questions to guide laypeople when reading medical papers. Paper Plain was published in ACM TOCHI 2023, and will be presented at the upcoming ACM SIGCHI 2024.
We hope by releasing these resources, they can enable the broader research community to explore exciting challenges around novel research support tools. Join us in designing the future of scholarly reading interfaces with our open source libraries!
References
PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents. Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Chee Chang, Russell Authur, Erin Bransom, Stefan Candra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, Amanpreet Singh, Chris Wilhelm, Angele Zamarron, Marti A. Hearst, Daniel S. Weld, Doug Downey, Luca Soldaini. (To appear) Conference on Empirical Methods in Natural Language Processing: Demos. 2023.
Follow @allen_ai and @semanticscholar on Twitter/X, and subscribe to the AI2 Newsletter to stay current on news and research coming out of AI2.
