Neuro-Symbolic Creative Artificial Intelligence for Humor

Author: Thomas Winters
Publication Date: 2023-12
Publication Venue: PhD thesis
Abstract: Large transformer-based language models, e.g. BERT and GPT-3, outperform previous architectures on most natural language processing tasks. Such language models are first pre-trained on gigantic corpora of text and later used as base-model for finetuning on a particular task. Since the pre-training step is usually not repeated, base models are not up-to-date with the latest information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art Dutch language model, which was trained in 2019. First, the tokenizer of RobBERT is updated to include new high-frequent tokens present in the latest Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the RobBERT model using this dataset. To evaluate if our new model is a plug-in replacement for RobBERT, we introduce two additional criteria based on concept drift of existing tokens and alignment for novel tokens.We found that for certain language tasks this update results in a significant performance increase. These results highlight the benefit of continually updating a language model to account for evolving language use.

Read paper

Citation

APA

Winters, T. (2023). Neuro-Symbolic Creative Artificial Intelligence for Humor [Phdthesis]. KU Leuven.

Harvard

Winters, T. (2023) Neuro-Symbolic Creative Artificial Intelligence for Humor. phdthesis. KU Leuven.

Vancouver

Winters T. Neuro-Symbolic Creative Artificial Intelligence for Humor [phdthesis]. KU Leuven; 2023.

BibTeX

@phdthesis{winters2023phdthesis,
author = {Winters, Thomas},
language = {eng},
title = {Neuro-Symbolic Creative Artificial Intelligence for Humor},
school={KU Leuven},
year = {2023},
abstract = {In the rapidly evolving domain of artificial intelligence, large language models have shown significant progress in generating text for creative applications, including humor. However, they lack the controllability, efficiency and explainability which symbolic AI methods offer. In this thesis, we explore how symbolic and subsymbolic AI methods can be used, extended and integrated for applications in creative AI, particularly for Dutch humor, while also going beyond that. To achieve this, the thesis tackles three challenges: 1) adapting symbolic generative techniques for creative and humor-related tasks; 2) creating high-performance general Dutch language models and making them capable of executing humor-related tasks; and 3) integrating neural and symbolic paradigms for enhanced performance and scalability.
This thesis presents a set of novel methods, models and frameworks to meet these challenges. The first challenge is addressed through augmentations to existing symbolic text generators such as templates, grammars and Markov chains, enhancing their adaptability, learnability and stylistic control for creative and humor-focused imitative text generation. These methods led to several practical implementations, such as Twitterbots and a slideshow generator, which serve as proofs of concept and evaluation methods. For the second challenge, we introduce several state-of-the-art Dutch large language models called the RobBERT models. Using these, we show how they can detect humor and be used with symbolic methods to generate humor. For the final challenge, this thesis introduces DeepStochLog, a framework that integrates stochastic definite clause grammars with neural network capabilities, providing a high-performance framework for scalable neuro-symbolic logic programming.
Our empirical findings show that the extended symbolic generators are effective in creative contexts and that the generated slideshows are quality-wise on par with human-created ones. The RobBERT language models achieved outstanding state-of-the-art results on most Dutch language tasks and also surpassed other models at humor detection. Moreover, DeepStochLog exhibits state-of-the-art performance on most neuro-symbolic tasks while displaying higher scalability than existing methods.
In summary, this research demonstrates the continuing relevance of symbolic methods in the modern generative landscape and offers significant contributions to the Dutch natural language processing community through the RobBERT models. The introduction of the DeepStochLog framework provides a new view on integrating logic and neural networks using the more scalable grammar semantics, thereby laying the foundation for the future interplay between symbolic and neural systems. This thesis thus lays the groundwork for future endeavors at the intersection of neural and symbolic paradigms, opening new avenues for creative and humorous applications in artificial intelligence.},

}