Neuro-Symbolic Creative Artificial Intelligence for Humor
- Author: Thomas Winters
- Publication Date: 2023-12
- Publication Venue: PhD thesis
- Abstract: Large transformer-based language models, e.g. BERT and GPT-3, outperform previous architectures on most natural language processing tasks. Such language models are first pre-trained on gigantic corpora of text and later used as base-model for finetuning on a particular task. Since the pre-training step is usually not repeated, base models are not up-to-date with the latest information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art Dutch language model, which was trained in 2019. First, the tokenizer of RobBERT is updated to include new high-frequent tokens present in the latest Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the RobBERT model using this dataset. To evaluate if our new model is a plug-in replacement for RobBERT, we introduce two additional criteria based on concept drift of existing tokens and alignment for novel tokens.We found that for certain language tasks this update results in a significant performance increase. These results highlight the benefit of continually updating a language model to account for evolving language use.
Citation
APA
Winters, T. (2023). Neuro-Symbolic Creative Artificial Intelligence for Humor [Phdthesis]. KU Leuven.
Harvard
Winters, T. (2023) Neuro-Symbolic Creative Artificial Intelligence for Humor. phdthesis. KU Leuven.
Vancouver
1.
Winters T. Neuro-Symbolic Creative Artificial Intelligence for Humor [phdthesis]. KU Leuven; 2023.
BibTeX
Related talks
Related projects

Babbly
A programming language for efficiently building complex text generators

DeepStochLog
Neural Stochastic Logic Programming

GITTA
Discovering textual structures to create generators

RobBERT
State-of-the-art Dutch language model

RobBERT Humor Detection
Distinguishing jokes from generated non-jokes, creating the first Dutch humor detectors

Talk Generator
Automatic slideshow generation about any topic

TorfsBot
Twitterbot automatically imitating Rik Torfs, the previous rector of KU Leuven

Twitterbots
Various little Twitterbots I made over the years

TorfsBot Or Not?
Is the tweet from Rik Torfs or TorfsBot? A daily Twitter Turing test