academics

NLP has become much more interesting!

ehudreiter

At Aberdeen, we have a weekly reading group which has run (with some interruptions) for almost 20 years. Members propose papers for the group, I seem to end up doing this around once a month or so. There was a time around 5 years ago when I found it difficult to propose papers which I was excited by, but in the second half of 2023 I’ve been spoilt for choice, there are lots of exciting papers out there. Which is a very welcome change!

Perhaps because of chatGPT and LLMs, and the push to real-world usage, I’m seeing a lot more work on high-quality evaluation (as opposed to yet more poorly-valided metrics), real-world NLG issues (which do not manifest in artificial leaderboards), and scientific understanding of the fundamentals (as opposed to tweaking models to slightly increase performance). Just as importantly, I am seeing more high-quality science I can trust (a lot of ACL-type papers are scientifically dubious). Which is great!

Papers mentioned in blogs

I have talked about some of the 2023 papers which impressed me in previous blogs, including

I’d like to think that some of my own papers are also scientifically important, including the ones discussed in

Other really interesting papers

There are many other really interesting papers I’ve read in 2023 (often in our reading group) which are not mentioned in the above blogs. A few such papers which made a real impression on me are listed below (there are many others).

G Abercrombie et al (2023). Mirages. On Anthropomorphism in Dialogue Systems. Proc of EMNLP-2023

There has been a long-standing assumption that dialogue systems should be made as human-like as possible. Indeed media stories about AI and dialogue systems highlight this, as do many evaluations. But human-likeness may *not* be a good thing in real-world dialogue systems!

D Demszky et al (2023). Using large language models in psychology. Nature Reviews Psychology

The NLP community has often been somewhat insular, and one of the really encouraging developments (at least to me) in 2023 was the blossoming of inter-disciplinary papers which linked NLP to other scientific fields. This was my favourite 2023 paper in this genre, really interesting to see how psychologists thought LLMs would impact their research.

G Lapalme (2023). Data-to-Text Bilingual Generation. Arxiv.

The NLG community is almost completely focused on neural and ML techniques, so it was very refreshing to see a paper on an open-source software package for rule-based NLG, including many demos and examples. I loved the concluding comment “we found [symbolic NLG] easier, faster and more fun than fine-tuning the parameters of a learning algorithm or tweaking prompts for an LLM.”

M Karpinska and M Iyyer (2023). Large language models effectively leverage document-level context for literary translation, but critical errors persist. Proc of WMT-2023.

This was a great example of a very careful high-quality scientific analysis of what LLMs could and could not do in a specific machine translation context. Findings are very interesting, but to me the key contribution is to show what a very high-quality evaluation in this space looks like (professional translators asked to evaluate 18 language pairs using a carefully designed annotation protocol).

R van Noorden (2023). Medicine is plagued by untrustworthy clinical trials. How many studies are faked or flawed? Nature

This is a medical paper, not a CS one, but it still made a huge impression on me. I’ve always had great respect for medical experiments and evaluation, but this article highlighted the huge problems seen in medical experiments. Especially the work of Carlisle, who found that he was **20** times more likely to identify a paper as seriously flawed when he had access to detailed experimental data (compared to when he just read the paper itself). What percentage of ACL/NLP papers would be assessed as seriously flawed by reviewers who had access to data (and enough time to check it)???

Looking forward to 2024

I am looking forward to 2024, and expect to continue to see exciting high-quality research on topic I care about!

Leave a comment