redefine-math

redefine-math (Xudong Shen) General description In this task, the author tests whether language models are able to work with common symbols when they are redefined to mean something else. The author finds that larger models are more likely to pick the answer corresponding to the original definition rather than the redefined meaning, relative to smaller models. This task demonstrates that it is difficult for language models to work with new information given at inference… See the full description on the dataset page: huggingface.co/datasets/inverse-scaling/redefine-math.