BlackboxNLP

Python code imports numpy as np so consistently that, to a language model, the alias and the library are almost the same thing. We ask what a model does when a file breaks the convention, as in import pandas as np. Using templated minimal pairs, we score how strongly a model prefers the conventional library's methods at the use site and compare the swapped binding with a control that binds the same library to an alias with no convention (zz). Across nine open models from 0.5B to 8B parameters, the swap shifts the preference toward the conventional library by 6.8 nats relative to the control (95% CI 4.6 to 9.1). The shift is larger for more frequent conventions, stays flat out to 8,192 tokens, and reaches generated code, where 74% of swapped completions call an attribute the bound library lacks. DeepSeek-V4-Pro makes the same mistake out to 128k tokens even though it names the bound library correctly when asked, and its thinking mode removes the error. In two Qwen2.5 models, activation patching and the logit lens show that the pull travels with the alias token at the use site. Patching the import line barely moves it.

Controlling the output of Large Language Models (LLMs) is a central challenge for their reliable deployment, yet a clear understanding of the involved trade-offs remains elusive. Current approaches to conditioning are often evaluated with a narrow focus on their effectiveness at injecting or removing a target concept, neglecting generation quality. We systematically investigate a range of conditioning methods in both injection and removal scenarios. We find that efficient steering methods frequently achieve conditioning at a steep cost to fluency. Furthermore, we identify a critical yet previously overlooked interaction with the training paradigm: activation steering methods are far less effective on instruction-tuned models than on their base counterparts. Simple prompting and full-fledged supervised fine-tuning, on the other hand, are viable options for concept injection, but are not as good at concept removal. Finally, cheaply computed textual metrics highly correlate to costly LLM-as-judge scores, and provide insights on the behavior of conditioning methods.

LLMs often default to equal treatment across cultural groups, even though context warrants differentiation: this is a lack of difference awareness. Using mechanistic interpretability and a factorial design on the N4 cultural appropriation benchmark from Wang et al. (2025), we identify 2–3 mid-layer attention heads per model that contribute causally to cultural binding across eight models (base and instruct versions of four architectures). Cultural binding is the process of associating a cultural item with its related identity. Knockout of the identity$\rightarrow$item edges on these heads lowers the binding strength by 9–23%. The identified heads transfer from instruct to base models, suggesting that cultural binding is created during pre-training. An $\alpha$-scaling shows a graded dose–response. Moderate amplification steering at generation ($\alpha = 2$–$3$) increases cultural differentiation accuracy by 1–3 pp while leaving reasoning on culturally neutral questions mostly intact. A knowledge probing task shows that models know $3$–$6\times$ more than they act upon, indicating that the bottleneck lies in routing and not knowledge.

Reliably operating LLMs in specialized domains often requires retrieval-augmented generation (RAG), which grounds the model's responses in domain-specific information. Retrieval performance is critical for output quality, making retrieval model selection a key decision for specialized RAG deployments. To guide retrieval model selection, we ask whether domain-tuned encoders produce better-structured embedding spaces for specialized corpora than general-purpose encoders, and whether geometric regularity alone reliably indicates retrieval quality in these settings. Our results show that the answer is more nuanced than expected: domain-specific models show more pronounced geometric irregularity than general-domain encoders, while geometric regularity alone does not reliably indicate retrieval quality. Moreover, geometric and intervention patterns observed on MS-MARCO do not consistently transfer to specialized corpora.

We analyze how lexical ambiguity and underspecification affect language model training. We create artificial homonyms and artificial hypernyms as pseudowords and analyze the generative performance of language models as they are trained with increasing amounts of these ambiguous or underspecified pseudoword types. We further analyze whether the models disambiguate ambiguous or underspecified statements and provide a first mechanistic account of how ambiguity and disambiguation are represented internally. Our main results show that both ambiguity and underspecification increase model performance in ways that scale with their influence on the language's type-token ratio. However, the accuracy of generating sequences containing ambiguous words or their synonyms decreases compared to other texts. We also show that internal representations of pseudowords reflect disambiguation of pseudo-homonyms, but underspecification of pseudo-hypernyms is maintained during the generative process.

Text embedding inversion reconstructs source text from fixed-dimensional vectors, posing a privacy risk to systems that store embeddings instead of raw text. We adapt the vec2text framework (Morris et al., 2023) to support text-embedding-3-small and mistral-embed alongside GTR-base. We evaluate 800 samples per category across general instructions, Wikipedia passages, paraphrased instructions, health-related instructions, and short literal code snippets, using a common 60–80-character range and paired original and paraphrased instructions. Under the evaluated vec2text configuration, text-embedding-3-small achieves the highest exact-match rate in all five categories, including 47.4% on health-related instructions and 24.5% on general instructions under the evaluation’s case-insensitive, whitespace-stripped matching rule. BERTScore F1 exceeds 87 on the 0–100 scale for every prose category and embedder, indicating high contextual similarity to the references. Short code snippets yield lower reconstruction scores. These findings characterize one attack configuration and the selected datasets; they do not establish attack-independent rankings or sensitive-attribute recovery rates. They nevertheless motivate explicit assessment of inversion risk and access controls for stored embeddings.

We reproduce the canonical GPT-2-small ``greater-than'' circuit in its original stack and independently, then test how far it generalizes across eight models (124M--12B) and to numerical entailment. The reported numbers replicate, and the late-MLP structure recurs at matched relative depth well above a size-matched null. Sparse faithfulness is high everywhere too, but only when the edge budget scales with the model's own graph; since these graphs differ 96-fold in size, a fixed budget makes faithfulness appear to collapse with scale. On numerical entailment, no circuit-bearing model \rev{passes the 0.6 gate}, while the one model that solves the task well cannot be posed the circuit's own task.

Large Language Models (LLMs) rely on induction heads for in-context learning, yet how they manage complex hierarchical structures and contextual dependencies remains unclear. By evaluating a range of LLMs on synthetic tasks and natural language analogues, we identify adaptive induction heads in later layers that learn to attend to successor tokens based on hierarchical cues rather than simple bi-gram statistics. These heads are supported by a circuit of context matching heads that uncover latent transition relationships by routing information from distant past tokens to the current query. Controlled ablation experiments across model families confirm that this circuit is essential for accurate hierarchical predictions in context. Finally, we show that this mechanism is used in natural language to resolve ambiguities where single tokens have multiple potential successors. Our findings provide a mechanistic account of how LLMs learn to predict higher-order repetitive patterns in-context.

Large language models (LLMs) increasingly score open-ended text along multidimensional rubrics, yet strong score recoverability need not imply that a dimension is distinct or relevant to downstream decisions. We examine how Gemma-2-2B represents five metacognitive-prompting dimensions in self-reflections from $486$ participants on a probabilistic reversal learning task. Linear probes recover all dimensions across depth, with depth gradients ordered by the rubric hierarchy. Evaluation is strongly decoded but has the weakest coupling to independently measured task markers. Under the primary five-occurrence token filter, Evaluation-related vocabulary shows no detectable association with switching, whereas the negative Final Decision association is stable across frequency thresholds. Activation patching yields oppositely signed point estimates in a fixed linear win--switch readout when the Final Decision--Evaluation matched-pair contrast is injected in each direction, although no layer survives correction across the full sweep. These findings show that decodability alone is insufficient evidence for weighting rubric dimensions in downstream decisions.

Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions representing misaligned personas, acquired during pre-training, are amplified during fine-tuning on misaligned data. We ask where these features come from, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that misalignment fine-tuning amplifies features related to jailbreak personas, sarcasm, deception, and manipulation, while suppressing safety-relevant and assistant-identity features. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models and re-aligns misaligned models to near-baseline rates. Attributing the causal features to one million web documents from an open pre-training corpus retrieves recurring narratives about villainous characters, domination, and harmful agency. Yet fine-tuning on these human-written documents does not reliably induce EM, whereas synthetic instruction-response pairs derived from the same content do, even across models. Semantic relevance alone is thus not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.

Large language models process text as sequences of tokens, raising the question of how coherent representations emerge from fragmented word pieces during inference. Prior work has proposed a \textit{detokenization} stage in early layers, where tokens are recombined into larger linguistic units, but direct intervention-based evidence for the role of specific model components remains limited. This article investigates whether a small subset of attention heads (1/3/5%) contributes disproportionately to this process. Candidate heads are identified based on their preference for within-word attention, measured using attention weights and $L_2$ norms, and their contribution is evaluated through targeted ablation and activation patching in a word reconstruction task. Across languages (English, Turkish, Chinese), tokenization schemes (BPE, SentencePiece, byte-level), and model families, ablating these heads generally produces greater degradation in reconstruction performance than random ablations. Results further show that larger models exhibit increased specialization, characterized by steeper and more pronounced early-layer retrieval peaks. These findings provide additional evidence for the detokenization hypothesis and suggest that early layers rely on specialized attention heads to construct internal word-level representations from fragmented inputs.

Large Language Models (LLMs) can generate natural language explanations that rationalize their own decisions, a phenomenon commonly referred to as self-explanations. Such explanations have emerged as a promising direction for explainable artificial intelligence (XAI), particularly for interpreting LLM behavior. However, while self-explanations often appear plausible, whether they faithfully reflect a model's underlying reasoning process remains an open question. In this opinion paper, we argue that self-explanations can be highly plausible, questionably faithful, and yet highly actionable. From a traditional XAI perspective, we identify the limitations of standard evaluation protocols for LLM-generated self-explanations and propose practical guidelines for assessing their plausibility and faithfulness. Moreover, we argue that evaluation should extend beyond these criteria to actionability, highlighting applications of LLM rationalization capabilities that support informed decision-making and appropriate action across diverse stakeholders.

Sparse autoencoders (SAEs) are typically applied at the single layer they were trained on. We ask whether a pretrained SAE's per-prompt feature activations persist when the same SAE is applied to residuals at deeper layers of the underlying language model, and whether the resulting cross-layer signal is strong enough to drive per-prompt FFN neuron pruning without any training or calibration data. Testing three SAE families on three roughly 8B-scale language models, we find adjacent-layer carry-through is high across families (0.72--0.82 at $L_{\text{src}}{+}1$) and its rate of decay differs qualitatively across sparsification mechanisms, with JumpReLU still activating half the features 17 layers downstream while TopK degrades within about five. We propose \textbf{FARN} (Feature and Anchor-based Removal of Neurons), a training-free per-prompt FFN pruner that scores each downstream neuron on the SAE's decoder columns weighted by the current prompt's active features, plus a prompt-independent anchor from the decoder bias. On six multiple-choice benchmarks, keeping roughly 17--20% of FFN neurons per layer, FARN closes the gap to an activation oracle (a baseline that peeks at the true downstream activations to pick the top-firing neurons) to within 1.5 percentage points and outperforms random, magnitude, and calibrated static-activation baselines on every model.

High probe accuracy does not by itself establish that a language model computes and uses the decoded value. We examine this inference in a syntax-sensitive arithmetic testbed: Llama-3.1-8B often fails multiplication when an operand is wrapped in parentheses or an additive identity. In the primary operand band, an input-only linear baseline nearly matches the original hidden-state probe's regression $R^2 \approx 0.96$. A classical variance decomposition gives an analytical headroom of only $0.031$ for this baseline class. In wider bands, selection-separated reanalysis instead finds positive activation advantages, which increase with regularization tuning; joint input-and-activation probes also improve over the baseline. These predictive gains are distinct from exact integer recovery and causal use. In the primary band, tested answer-site steering and state transplants produce small average output shifts and no observed donor-answer emissions, whereas operand-site interventions produce large effects. Operand access is necessary, with a strong bottleneck at one attention layer. The causal tests do not cover the wider-band predictive gains. We advocate explicit input baselines, matched causal tests, and positive controls, with conclusions restricted to the distributions, representations, and interventions actually examined.

Language can affect how people think. We ask a computational version: if a language model is trained on English that marks propositions as observed, reported, or inferred, does it use evidence differently in ordinary English? We create matched corpora in which particles either mark these routes (EVID), appear in the same positions without predicting source (CTRL), or are absent (NOMARK), and remove them at evaluation.

We test 405M-parameter Pythia models trained from scratch and Qwen3.5-4B Base and Post models adapted across three paired seeds. The models learn the mapping: Pythia reaches 92.8% before ordinary-English adaptation and retains 79.8% afterward; Qwen EVID averages about 80% for both Base and Post, with controls near chance. Visible markers shift Qwen's scores. Once markers are removed, however, the Pythia tests cannot show whether the effect carries over. Base's main Qwen result changes direction across seeds, and Post's is negative in all three. A separate test suggests that Base may respond more to relevant context, but not specifically to source wording. No condition consistently improves reliability, multistep inference, or recognition of copied information. Source marking is easy to learn and can influence model scores, but we find no broader improvement in reasoning.

Multimodal LLMs report strong performance on scientific reasoning benchmarks, yet most treat perception and reasoning as a single measurable process. We introduce a five-task diagnostic experiment across physics and geometry benchmarks that isolates failures to perception, reasoning, or both. Incorrect diagram interpretation degrades performance even on problems models solve correctly from text alone, and accuracy generally rises from raw images to human-authored captions. Recovery under corrected captions is high for some models, separating perception-blocked failures from genuine reasoning bottlenecks. Which reasoning error follows a perception failure depends on domain: physics failures resolve into calculation errors, geometry into conceptual misapplication. As a discussion beyond our core experiments, InternS1-mini, despite heavy scientific pretraining and thinking capabilities, falls below the weakest model from experiments on every task, with reasoning traces frequently truncating before completion.

Large language model (LLM) hallucinations, meaning fluent but factually incorrect generations, fall into two types: faithfulness violations, where the model misuses provided context, and factuality violations, where answers reflect errors in internal knowledge. Proper mitigation depends on knowing which source drives each answer. We study contributive attribution, i.e. the classification of the dominant knowledge source behind each output, and show that a simple linear probe trained on hidden representations can reliably identify it. We introduce AttriWiki, a self-supervised pipeline that automatically generates labelled training data by prompting models to recall withheld entities from memory or read them from context without relying on knowledge conflicts. Probes trained on AttriWiki achieve up to 0.96 Macro-F1 on Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B, transfer to SQuAD and WebQuestions with 0.92-0.99 accuracy, and generalize zero-shot to Tighidet et al. (2024)'s benchmark, outperforming their probe on conflicting settings without retraining. Furthermore, attribution mismatches raise error rates by up to 70%, though correct attribution does not guarantee correct answers, pointing to the need for broader detection frameworks.

While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising. In this work, we study how DLMs implement induction, a mechanism behind in-context learning in which the model finds a repeated context and copies the token that followed it. Our analysis compares attention-only AR models and absorbing-mask DLMs with matched architectures. We find that DLMs learn a bidirectional induction circuit, where previous-token and next-token heads write local context into the residual stream and later induction heads use it to find and copy the answer from the matching source position. The circuit is direction-symmetric, working whether the source appears in the past or in the future. When only left context is visible, matching what an AR model sees, the DLM does not outperform its AR counterpart in induction capabilities. However, we observe it has stronger induction when both sides of the masked token are visible, pointing to bidirectional context access rather than a stronger one-sided mechanism. Beyond induction, we provide causal evidence that DLMs compute the global fraction of masked tokens and use it as an implicit timestep, even though they are given no explicit timestep embedding.

We observe that fine-tuned BERT outperforms feature-based models at estimation of text difficulty, but it is unclear what drives this advantage. We explore a state-of-the-art Finnish BERT difficulty estimator and analyze it against a strong feature-based model trained on 66 hand-crafted linguistic features. First, we find that BERT implicitly encodes the coarse length and complexity features, but not fine-grained morphology of Finnish. Second, BERT subsumes the information in the hand-crafted features and adds signal that they miss: the two models are evenly matched on most documents, and BERT's advantage comes from rarely being very wrong, rather than from winning everywhere. Third, through input perturbation and layer-wise probing, we show that BERT's predictions rely on sentence-internal structure rather than on lexical content alone. Our analysis explains why a neural difficulty estimator succeeds and what linguistic information it depends on.

Distinguishing animate from inanimate concepts in written language requires more than shallow text processing, as it involves recognizing complex selectional constraints and contextual cues, such as verb-argument interactions. Yet, current large language models (LLMs) appear to be capable of doing it. We investigate whether this animacy-sensitive behavior of LLMs can be traced to a localized set of causally relevant components and connections. To do so, we construct a controlled dataset of minimal pairs and perform circuit discovery on four open-weight models. Through in-depth experiments and ablations, we show that a causal mechanism responsible for handling animacy in these models does exist, thus discovering an animacy circuit. At the same time, this circuit appears to be less localized compared to other known ones and generalizes only partially across models and animacy tasks, confirming the distributed, context-dependent, and somewhat graded nature of the animacy concept.

Large language models can produce fluent explanations for chess moves, but plausible language does not necessarily reflect the reasoning behind a decision. We study this question in chess, where the board state is fully observable, legal actions can be enumerated, and move quality can be evaluated independently. Across 200 Lichess endgame puzzles, we test explanations using move recoverability, decoder-side controls, and token-level scoring of legal candidate moves. Unmasked explanations make generated moves easy to recover, but this advantage drops sharply after explicit move hints are removed. Under strict masking, explanations provide only small and decoder-dependent gains over the board state alone. Token-level scoring shows that explanations can nevertheless alter move preferences: random but plausible explanations from other puzzles reduce the probability of the correct move, indicating that irrelevant reasoning text is not simply ignored. We also find that recognizable endgame motifs can make generated moves easier to recover without reliably improving move correctness. Together, these results show that linguistic plausibility, consistency with a generated action, and solution correctness are distinct properties. Fluent chess explanations can influence action preferences and support a coherent move narrative while providing only limited evidence of faithful reasoning.

Language Models (LMs) produce highly fluent narratives, yet cognitive probing suggests their semantic representations lack grounding in sensorimotor experiences. By projecting model-generated texts into a multidimensional sensorimotor vector space, we quantify narrative embodiment in storytelling. Specifically, we evaluate the impact of parameter scaling and conditional multimodality on perceptual grounding. We find that (1) model capacity monotonically widens the divergence from the human level of embodiment, demonstrating that scale alone cannot resolve narrative disembodiment, and (2) audio-language models demonstrate a distinct superiority over vision-language models, an effect we contextualize through the auditory scaffolding hypothesis. Ultimately, we demonstrate that authentic sensorimotor grounding is not an emergent property of autoregressive decoding or conditional encoding, highlighting a critical limitation in current generative architectures.

A causal large language model can retrieve the same fact more confidently in one direction than in the other. This asymmetry is known as the Reversal Curse. The usual explanation links it to next-token prediction: seeing ``$A$ is $B$'' trains the model to predict $B$ from $A$, but not $A$ from $B$. Prior work supports this with controlled finetuning on synthetic facts. We ask whether the same account still holds in a real pretraining corpus, where facts appear many times, in different contexts, and in both orders. To test this, we work with OLMo~3 7B Base and Dolma, its 5.93-trillion-token stage-1 corpus. For 100 curated relation instances, we locate passages where both entities appear, have two LLM judges check which passages state the expected relation, and probe the model in both directions with contrastive likelihood scores. The raw specific-prompt forward-minus-reverse gap is small and uncertain (0.119, 95% CI $[-0.228,0.486]$). Once the generic-prompt gap is subtracted, relation naming helps reverse retrieval more than forward ($\Delta^{\mathrm{rel}}=-0.467$, $[-0.673,-0.266]$) on average, though not uniformly across families. That gap is not predicted by local entity order after we control for exposure, proximity, entity frequency, and relation family.

Interpretability research often predicts model responses to targeted mechanistic $\textit{interventions}$. But can we predict responses to unseen $\textit{input data}$? We propose and demonstrate this alternate objective by using model internals to predict their out-of-distribution (OOD) behavior. We train hundreds of Transformers on simple synthetic tasks, where perfect in-distribution accuracy is compatible with multiple OOD generalization rules. We successfully use attention patterns---observed only on in-distribution data---to predict which rule each model follows on OOD data. Our experiments decouple the mechanistic faithfulness of our interpretation from its predictive value; ablations reveal such internal patterns can $\textit{suppress}$ rather than $\textit{support}$ the rule they predict, showing observational analysis can forecast behavior even when causal analysis fails to support a simple cause-effect link. Our findings are a proof-of-concept for a new interpretability objective: understanding model internals to predict behavior and assess reliability under distribution shift.

Transformer circuits are often evaluated by ablating certain circuits or components and measuring the resulting performance drop. These interventions test circuit necessity only under a fixed model configuration, and it is unclear if the residual network can recover performance through alternative, parameter-efficient adaptations. Using BERT as a frozen encoder and a linear part-of-speech (POS) readout as a task classifier, we first use gate masking to learn a sparse core subnetwork sufficient for POS classification. Ablating this core produced a damaged model and a sharp drop in accuracy $(95.0\\% \\rightarrow 16.8\\%)$. We then evaluated recovery from damage under four retraining pathways: readout-only, mask-only, joint readout+mask, and encoder-only with the ablation held fixed. Readout-only, mask-only, joint mask-and-readout, and encoder-only retraining arrived at 80.0%, 69.7%, 85.9%, and 92.3% accuracy, respectively. Some recovery pathways partially restored core-like geometry, but others recovered high performance without restoring the original encoder geometry. These results show that in a joint encoder--readout context, a sparse subnetwork can be sufficient and necessary for performance in the original fixed configuration, but need not remain necessary after recovery, and that recovered performance does not rely on recovering the original geometry.

Large language models (LLMs) have been shown to encode truth of statements in their activation space along a linear truth direction. Previous studies have argued that these directions are universal in certain aspects, while more recent work has questioned this conclusion drawing on limited generalization across some settings. In this work, we identify a number of limits of truth-direction universality that have not been previously understood. We first show that truth directions are highly layer-dependent, and that a full understanding of universality requires probing at many layers in the model. We then show that truth directions depend heavily on task type, emerging in earlier layers for factual and later layers for reasoning tasks; they also vary in performance across levels of task complexity. Finally, we show that model instructions can affect truth directions; simple correctness evaluation instructions significantly affect the geometry and the generalization ability of the truth probes. Our findings indicate that universality claims for truth directions are more limited than previously known, with significant differences observable for various model layers, task difficulties, task types, and prompt templates

Safety benchmarks for large language models (LLMs) commonly use multiple-choice question (MCQ) format. However, prior studies documented that MCQ format reduces refusal rates, which may significantly affect the safety behavior of the model. We provide a first mechanistic explanation of this effect on the Weapons of Mass Destruction Proxy (WMDP) dataset across two models – Llama-3.1-8B and Qwen3-8B. First, we use the tuned lens technique to show that when processing MCQ the model commits to an answer-letter token very quickly, which is in contrast to the gradual refusal build-up in open-ended formats. Additionally, we observe that activations on the MCQ format lie roughly $3\times$ further from the refusal direction, compared with their open-ended counterparts. Moreover, ablating the attention heads that carry the MCQ format answer increases the refusal, which is a direct evidence that the MCQ answer-selection competes with refusal. These findings indicate that MCQ safety scores primarily measure knowledge capabilities and do not assess the safety-mechanisms.

In language models, the choice between believing the prompt and believing the weights is made by a handful of identifiable attention heads. Instruction tuning changes how models behave under conflict, but whether it rewires the underlying circuit or merely gates/reweights already present components, remains unknown. We provide the first mechanistic base-vs-instruct comparison of conflict-resolution circuits, across three families (Llama-3.2-3B, Qwen-2.5-3B, Gemma-3-4B). Five independent methods, node and edge attribution, superposition role analysis, causal ablation, and path patching, converge on gating, with the same heads, in the same late-layers, are found to be reweighted rather than replaced with a high node overlap (0.60–0.82). Behaviorally, tuning shifts models toward parametric memory, making instruct models reject a terse counterfactual context far more than base ones, the opposite of a naive user-following expectation. Yet this added skepticism is a factor of framing since it disappears when the same false claim is delivered as a coherent, evidential passage. The robustness that instruction tuning buys against terse injection is therefore real but narrow. More broadly, we believe that because the conflict circuit is preserved rather than rebuilt, interpretability and control tools calibrated on base models should transfer directly to their deployed instruct siblings.

How do LLMs learn in-context? Is it by pattern-matching recent tokens, or by inferring latent structure? We probe this question using a toy graph random-walk across two competing graph structures. We present behavioral, representational, and causal evidence that neither account alone is sufficient. First, a complexity-weighted sigmoid model captures topology-dependent evidence accumulation, and PCA shows that both graph topologies are visually recoverable at intermediate mixture ratios. Second, residual-stream activation patching and graph-difference steering causally intervene on this graph-family signal: late-layer patching almost fully transfers the clean graph preference, while linear steering moves predictions in the intended direction and fails under norm-matched and label-shuffled controls. An explicit transition-cache baseline reproduces the fitted asymmetry and held-out-edge transfer, however, so these results do not by themselves distinguish global inference from local copying. Taken together, our findings are consistent with a dual-mechanism account in which structure inference and induction circuits operate in parallel, but do not uniquely identify it.

Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models (LLMs), which can be monitored for target behaviors by reading the reasoning trace, motivating work on CoT monitorability. Latent CoT approaches, however, replace the explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on. Monitoring then requires alternative access to the model, such as probing its activations or verbalizing the latent states back into text, but how much monitorability these alternatives preserve is unclear. We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues, e.g., an inadvertently leaked answer or a belief stated by the user, without acknowledging them. Taking hint-reliance as the monitorability target, we compare monitors across reasoning modes, from explicit CoT to weakly- and strongly-supervised latent CoT, on math reasoning and question answering. We find that, in this setup, monitorability depends more on properties of the task (such as whether the correct answer constrains the supporting reasoning) and the level of access to model internals than on the reasoning mode.

Deciding which transformer layers can be cheaply replaced - when a node fails mid-computation, or when a block is pruned - requires knowing what each layer actually computes. A straightforward approach is to fit a linear map between consecutive hidden states and read off its $R^2$. In our analysis, these fits look excellent, exceeding $R^2$ of $0.97$. However this number is largely an artifact of the residual stream: an untrained network scores just as well. The informative quantity is instead how linearly predictable a layer's update $h_{L+1}-h_L$ is. Measured this way, and across six models from three architecture families, layers fall into a clear U-shaped pattern: those at the beginning and end of the stack perform large but almost perfectly linear transformations, while those in the middle perform small nonlinear ones. Given these insights, we present Linear Layer Repair, a policy that reads a layer's profile and prescribes one of three responses when it goes missing: (i) skip it, (ii) swap in a fitted linear map, or (iii) declare it unrecoverable. Using WikiText-103, we test Linear Layer Repair by removing layers and measuring the damage. The results show that the linear map beats plain skipping at all $16$ LLaMA-3.2-1B layers and $26$ of $28$ in Qwen2-1.5B; replacing a single middle layer costs under half a point of perplexity against a baseline of 15; even four consecutive layers cost only $\sim$ $2.5$ points; and at an early layer, where skipping raises perplexity above $10^5$, the patch reduces it more than a thousand-fold. Task-level evaluations show corresponding recovery on prediction-sensitive behavior.

Automated interpretability systems aim to reduce the need for human labor and scale analysis to increasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We investigate this challenge in the context of automated circuit analysis---explaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against human expert explanations across six circuit analysis tasks in the literature, the system appears competitive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incomplete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems and reveals key limitations of replication-based evaluation.

Outlier dimensions are a recurring geometric property of transformer representations that have been linked to token frequency in pretrained language models. However, studies on pretrained models cannot disentangle token frequency from semantic and optimization confounds, preventing a causal understanding of whether frequency itself drives outlier formation. These distortions also degrade probing accuracy and hinder model compression, motivating a controlled study of their origin. We address this issue by isolating frequency skew as a controlled variable in a synthetic transformer trained on data with a tunable Zipfian token distribution. We find that outlier magnitude remains stable under low skew but increases sharply beyond a moderate threshold. A frozen frequency-based output bias fails to reproduce this behavior, model capacity exhibits a non-monotonic U-shaped effect, and an $L_2$-normalized cosine softmax suppresses outlier geometry only under a large temperature not reached by standard training. These results identify frequency skew as a causal driver of outlier dimensions and show that removing their geometry requires an intervention beyond standard training.

Large language models often succeed on one formulation of a problem while failing on an equivalent formulation. Whether these failures arise from distinct internal circuits or different activation states of a shared circuit remains unknown. Recent mechanistic interpretability studies suggest that arithmetic in LLMs emerges from a "bag of heuristics," encoded by a sparse set of MLP neurons that represent distinct arithmetic strategies. We investigate whether arithmetic heuristic neurons are form-invariant across symbolic arithmetic, natural language word problems, and Python code in three Llama-3 models. In each format, we identify arithmetic heuristic neurons using a two-stage pipeline combining attribution patching and activation patching. A compact set of neurons is shared across all three formats, and targeted interventions show this shared circuit is both necessary and sufficient for late-layer arithmetic computation. Transferring the shared neurons' activations from a successful execution in one format to a failed execution in another recovers most incorrect predictions, exceeding 97% for addition and subtraction, indicating that cross-format failures arise from activation states rather than distinct circuits. Moreover, shared neurons largely belong to the same heuristic families across formats, indicating that arithmetic computation in LLMs is substantially form-invariant at the neuron level. \end{abstract}

Detecting media bias automatically is difficult because biased framing is often subtle, yet in domains such as news analysis, accurate predictions alone are insufficient without explanations that reflect the model's underlying reasoning. We present a multi-dimensional evaluation of explainability in encoder-based media bias detection using the Bias Annotations By Experts (BABE) dataset. Specifically, we study BERT and RoBERTa as classifiers (base and large variants) along three complementary axes: predictive performance, explanation plausibility (token-level alignment with expert rationales), and mechanistic faithfulness (whether compact sets of attention heads recover predictive signal under counterfactual rationale masking). To induce variation in plausibility, we additionally investigate attention-supervised finetuning, which incorporates expert rationale annotations as an auxiliary training signal. Attention supervision serves as an intervention on attribution plausibility, while the effectiveness of attribution methods varies substantially across architectures. Circuit analysis further reveals substantial variation in mechanistic recoverability across architectures, suggesting that model scale alone does not determine circuit compressibility. Taken together, our findings suggest that predictive performance, attribution plausibility, and mechanistic faithfulness characterize different aspects of model behavior and should be evaluated separately when studying explainability in media bias detection.

Even though we know the precise computations that lead from a large language model (LLM) input to its output this computation remains very hard to interpret. One way to make it easier to understand this process is by creating a sparse computational graph that captures most of the model behavior with smallest number of computational nodes. Cross-layer transcoders (CLT) decompose the dense computations of the MLP but the resulting circuits still contain thousands of nodes even for short prompts. Existing automated interpretation methods label individual features from corpus activations, and it often happens that these labels are not validated by causal intervention. We introduce \emph{probe prompting}, a transparent rule-based pipeline that groups the features of an attribution graph into concept-aligned supernodes from their responses on a small set of concept-targeted probe prompts, summarized as Cross-Prompt Activation Signatures (CPAS). Across four factual domains, on Gemma-2-2B with a public CLT dictionary and 45,596 entity-swap interventions, we find that the labeled supernodes have the predicted steering behavior in every one of them. Code, datasets, and an interactive demo are released anonymously as a reusable harness for calibrating supernode labels against causal interventions.

Large language models (LLMs) encode rich stylistic structure in their hidden activations, but discovering which stylistic dimensions are salient for a given prompt typically requires supervised contrastive data. We present a training-free, prompt-conditional alternative: we repeatedly sample completions of a single prompt at elevated temperature, apply Principal Component Analysis (PCA) to the pooled hidden activations, and label the resulting axes automatically from the pole generations. We validate the discovered axes against 245 human-elicited stylistic annotations in a two-phase study. On our strongest model (Qwen3.5-4B), the top two axes match spontaneously requested human dimensions with 72.8% precision and 43.6% macro-recall, and 75.6% of validity ratings judge the axes’ polar generations accurate to their labels, with 90.9% adjacent inter-annotator agreement. Discoverability is strongly model-dependent: both Qwen models and Llama-3.2-3B expose human-salient axes, while DeepSeek-7B-Chat drops to 35.3% precision, its leading components dominated by structural rather than stylistic variance. Simple PCA over a model’s own decoding variance is thus an effective, low-cost probe of stylistic structure in LLM representations, one that also exposes sharp cross-model differences in how that structure is organized.

This study examines the compositionality of steering vectors for language and behavioral control in large language models. Focusing on language, jailbreak, and conciseness, we investigate whether additive, training-free composition of attribute steering vectors can preserve the intended steering effect of each attribute, across four instruction-tuned models from two model families and two size scales. We find that single-attribute steering is reliable for all three attributes, but only within an appropriate combination of intervention layer and steering strength, with abstract behaviors (jailbreak, conciseness) favoring middle layers and language favoring earlier layers. We show that additive composition of two attribute vectors succeeds in steering both attributes simultaneously when each is injected at its own best-performing layer, and that this partially extends to three simultaneously composed attributes, addressing an inconsistency left open by prior work on training-free composition. We further analyze the geometric properties of these steering vectors, finding that they are approximately orthogonal in the residual stream, consistent with their compositional behavior.

AutoXAI frameworks automatically select and optimize explainability methods using quantitative metrics such as faithfulness and plausibility. But do these metrics align with human perception of explanation quality? We investigate this question for attribution-based explanation methods, which assign token-level relevance scores to the input, in the context of claim matching, a core fact-checking task in which a bi-encoder model matches social media posts to previously fact-checked claims. Combining a large-scale evaluation of 11 XAI methods with a controlled user study, we find that methods recommended by AutoXAI metrics are perceived by users as most helpful for understanding the claim-post match, while deprioritized methods are judged as misleading. We further show that simple post-processing of explanations can improve human-perceived usefulness without degrading faithfulness. Because AutoXAI recommends the top-ranked method in practice, this alignment at the decisive top-versus-bottom boundary positions these metrics as reliable indicators for distinguishing clearly strong from clearly weak methods.

LLM-as-judge has become a standard way to score open-ended model outputs, yet the judge is itself a frozen language model steered by a few in-context examples, and such models are known to absorb the label balance of their demonstrations rather than treat them as neutral illustrations. We ask whether this label bias reaches an LLM judge used inside a multimodal (video question answering) evaluation pipeline, and what form it takes. Using a fully frozen generator--judge pipeline, we hold the generated answers fixed and vary only the Yes/No balance of the judge's few-shot examples across four demonstration sets. Each verdict is validated against a two-annotator human consensus on 200 examples per dataset. The judge's acceptance rate stays close to human consensus when its examples are balanced or skewed toward rejection, but jumps sharply (by about 30 points on NExT-QA and 20 on MSVD-QA, with a strongly one-sided error profile) when the examples are skewed toward acceptance; the opposite skew produces no matching drop. The bias is therefore one-directional within the tested range of label ratios. A separate five-draw analysis shows that results from a single example draw are unreliable, varying by up to 18 points, more than any difference between prompting conditions. We conclude that demonstration label bias in an LLM judge is a directional, inflation-only failure mode that can fabricate apparent benchmark gains but not losses, and that it is cheaply avoided by balancing the judge's examples and reporting results averaged over several draws.

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become \emph{concept-specific anti-readers}: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioural leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

While Transformer models can learn algorithmic tasks and generalize reliably to unseen, in-distribution data, they often fail catastrophically when required to extrapolate to sequence lengths beyond their training regime. Using AttentionSpan, a suite of synthetic diagnostic tasks designed to enable inspection of internal model computations, we analyze the model's behavior on length extrapolation. Our findings indicate that this failure is not fully explained by an inability to induce general rules for the task. Instead, we identify inconsistent attention patterns as a major contributing factor: information retrieval strategies learned by individual attention heads, which fail to remain stable as sequence length increases. Reinforcing a reference attention pattern at inference time, using no out-of-distribution data, largely recovers accuracy on String Reversal and partially on Value Assignment, while leaving multi-digit arithmetic unresolved. We also study \emph{length adaptation}, how a model trained on one length adjusts to another by fine-tuning its Key and Query parameters, and find that a single dimension in every attention head is enough, across 4 algorithmic tasks and the RULER benchmark. Adaptation does not carry past the fine-tuned lengths, and our interventions are diagnostic rather than a proposed remedy. We make our benchmark and code publicly available.

When the speaker has a motive to lie --- profit, face-saving, fear of punishment --- an informed reader expects the lie. Forming this expectation requires three distinct abilities: representing what is actually true in the described situation, aligning linguistic expectations with the speaker's incentives to be truthful, and adapting those expectations across situations. We study whether twelve open Qwen3 and OLMo2 models possess these abilities, asking how each is organized, whether they are separable, and when each is acquired. Within each model, fact information is linearly decodable at mid depth but becomes directly expressed only in the final third of the network; motive recognition emerges slightly later, and both grow with scale. Two-layer ablations dissociate them: removing a mid-depth band of layers weakens fact expression and motive recognition together. Ablating a deeper pair of layers, in contrast, leaves both essentially intact while eliminating human alignment: being sensitive to motives and being sensitive in the way humans are turn out to be separable abilities. Across 19 public checkpoints of OLMo2-7B, fact representation and motive recognition become detectable at around 0.3T tokens and grow steadily throughout pretraining, while human alignment stays non-significant until mid-training --- a brief continued training on curated data. Altogether, representing what is true and recognizing whether a speaker is motivated to lie are closely coordinated; matching human judgments about when lying is likely is a later and partially separable achievement.

Large language models encode high-level concepts with heterogeneous domain-specific manifestations, but it remains unclear whether their internal representations are organized in a form that supports cross-domain generalization. We investigate this question by adapting representational geometry methods from systems neuroscience and using political ideology as a naturalistic test case for a candidate latent variable spanning diverse policy domains. Across three open-weight LLMs, we analyze hidden states elicited by matched left- and right-leaning statements from ten policy domains. Within each domain, while political polarity is linearly decodable, lexical features achieve comparable performance. By contrast, cross-domain generalization substantially exceeds the lexical baseline and peaks in intermediate layers. This peak coincides with a marked reduction in participation ratio, suggesting that domain-specific political distinctions, instead of being maintained as independent axes, become aligned within a shared low-dimensional subspace. These findings suggest that LLMs transiently organize heterogeneous inputs into a common representational geometry that supports cross-domain abstraction, offering insight into how models structure complex social knowledge.

Sonority projection, the ability to rank unattested consonant clusters by their sonority contour, has been interpreted as evidence for innate phonological knowledge. We test whether neural phonotactic language models can acquire sonority structure from English phoneme distributions through a controlled comparison of four input geometries: articulatory features, trainable embeddings, feature vectors randomly reassigned to phonemes, and frozen random vectors. We train LSTMs and Transformers, probe whether sonority structure is linearly recoverable across representation spaces and throughout training, and evaluate phonotactic sensitivity using nonce-word violations and human acceptability judgements of unattested onsets. Sonority probes recover the consonant–vowel distinction from contextual phoneme representations under every input condition, whereas the hierarchy among consonant classes (obstruents, nasals, liquids, and glides) is most strongly recoverable with articulatory-feature input and less recoverable under the other conditions. Nevertheless, under all input geometry conditions, model surprisal discriminates phonotactic violations and correlates moderately with human judgements of unattested onsets. In summary, sonority structure and the tested phonotactic preferences can develop without sonority-aligned input representations: input geometry shapes how much fine-grained sonority structure is recoverable, but models with more recoverable structure, whether provided explicitly or learned during training, at the input or in the hidden states, show no consistent advantage on the phonotactic evaluations used here.

Misalignment in language models can be induced through either in-context learning (ICL) or supervised finetuning (SFT), but it is uncertain whether they share an internal causal representation. We study this in Llama-3.1-8B-Instruct by comparing two setups built from a dataset promoting financial recklessness: the base model whose context window carries 1,024 examples, and a model organism finetuned on that dataset. We extract difference-in-means directions that contrast misaligned and benign responses to finance advice prompts. We make four observations. First, the two directions are aligned across depth, with cosine similarity $0.59$ at layer 24. Second, the component transfers causally across induction routes: ablating the ICL direction from the SFT model lowers misalignment from $58.8$ to $47.4$, ablating the SFT direction from the ICL model lowers it from $65.9$ to $46.3$, and steering the ICL direction into the non-misaligned model raises it to $88.9$. Third, the component only partially represents the behavior: in the ICL model, the ICL direction is still causally much stronger than the SFT direction, and substantial misalignment remains even under repeated ablation. Finally, the component is domain-specific: finance-to-medical interventions do not transfer. On net the evidence supports a partial, domain-specific, and causal ICL--SFT bridge.

We find that representations of conversation turn structure in a Llama model are approximately helical, reflecting the cyclical nature of turn-based conversation. The position of a token within a conversation round can be accurately predicted using both linear and trigonometric probes on mid-to-late layers. True round-phase and round-phase predicted by a linear probe show a continuous correlation, ruling out "switch-like" geometries. Further, projecting onto a particular planar slice of activation space (revealed by a PCA and Fourier transform analysis) reveals a distinct rotational structure to the position of tokens within a conversation round.

We conducted an interpretability-informed analysis to study the influence of contextual cues on gender in translation in an ambiguous setting. We first created GAND, a gender-ambiguous natural data benchmarking resource for machine translation consisting of English source sentences. We then translated a subset of GAND into two grammatical gender languages and extended these with manually crafted contrastive translations. A feature attribution analysis revealed source words in context that inform the gender translation of an ambiguous referent entity in the target translation.

How does a multilingual model hold African languages in its hidden states? We compare the internal representations of nine African languages in Gemma-4-31B using four methods that operate directly on hidden states, avoiding the logit lens, and test whether the same semantic contrast occupies shared directions and subspaces across languages. Sharing with English is weak, partial, and resource-ordered. Removing English as the reference reveals a strong African inter-language structure that crosses the Niger-Congo/Afro-Asiatic family boundary and peaks in the early-to-mid network. This structure supports intervention: a difference-in-means direction added to the residual stream steers generation toward one of four African countries with causal effects of 0.63 to 0.81 over a random-direction baseline while preserving coherent text. A cross-model check on Aya-Expanse-32B shows that which model exhibits stronger sharing depends on the measure used: the representational structure is model-dependent per metric.

Hyper-Connections replace the single Transformer residual stream with multiple interacting streams, but multiple streams need not become multiple functional pathways. We reverse-engineer trained Hyper-Connection language models and identify a consistent stream-collapse regime. After an early mixing stage, residual mixers become nearly identity, while read and write operators repeatedly favor one stream. The same stream accumulates larger norms, follows smoother token trajectories, and contains most sparse crosscoder features. Learned Stream Scaling, a minimal symmetry-breaking intervention, reduces this imbalance, makes cross-stream mixing more consequential, and improves language-modeling performance. These results show how internal analysis can expose and repair underused architectural capacity.

Whether the internal architecture of large language models (LLMs) possesses a functional structure analogous to the human language system remains an open question. To investigate this, we propose a mechanistic interpretability framework inspired by aphasia (clinical language disorder): systematically perturbing a model’s internal components to simulate human language impairments and causally map specific linguistic functions. In this current work, we establish our diagnostic pipeline by developing a classifier to detect global language deficits in LLM outputs, coupled with a greedy ablation search validated by activation patching. Our preliminary findings localize aphasia-like degradation in Llama-2 to a compact three-head circuit in the early layers, laying the ground-work for fine-grained identification of syntactic and semantic components in LLMs.

When a language model answers from a retrieved document, its attention heads mix evidence from the context with evidence from the model's own self-generated tokens and write the sum into the residual stream. We split each head's write into these two source-specific contributions before they are summed, and find that the two sides are complementary. The context write, read out along the model's unembedding, gives a training-free groundedness score, GroundHOG-U. The response write (its magnitude and directional agreement with the other residual writes) gives a supervised hallucination detector, GroundHOG. Across grounded-generation benchmarks and model families, GroundHOG beats prior internal-state detectors, and GroundHOG-U is the strongest training-free groundedness signal on the multi-model benchmarks.

Large Language Models (LLMs) are increasingly being deployed to make decisions that require trading off near-term gains against long-term consequences, yet little is known about how they internally represent or resolve these tradeoffs. We causally localize an underlying subgraph for temporal preference in a distilled LLM (Qwen3-4B-Instruct-2507), identifying mid-to-upper-layer nodes through converging evidence from gradient-based attribution and activation patching. We show that the user-to-assistant turn boundary is the site where attention collapses a continuous time-horizon manifold into a binary preference, that contrastive activation steering shifts this preference 3.4 times in odds-ratio at L22, and that unintervened LLMs hold an unstable preference inconsistent across contexts.

Overview Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)—a phenomenon that prior work has highlighted as a critical safety concern (Betley et al., 2025a,b). We find this phenomenon is best understood by studying how models rely on context and when they fall back on out-of-context reasoning (OOCR; i.e. use non-trivial reasoning to generalize a behavior outside the context window; Evans (2026)). Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior thereby suppressing the OOCR and weird generalization’s effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward easily implemented solutions.

Large Language Model (LLM) representations are known to align with brain activity during language processing, but it remains unclear what drives this alignment. We test whether explainable AI (XAI) can help answer this: using attribution methods, we quantify word influence on LLMs' next-word predictions and map these explanations onto fMRI data from participants listening to narratives. We find that gradient-based attribution methods robustly align with brain activity, contribute unique variance beyond acoustic and word-rate confounds, and outperform internal representations in early auditory regions. Using layer conductance, we further find that this alignment reflects two different signals: early layers align with auditory regions because of their sensitivity to lexical content, while the last layer's alignment reflects a near-pure positional signal that reduces to local word rate.

Studies in which transformer models are trained on formal languages show limitations on acquiring more complex types of formal languages, such as context-sensitive ones. We evaluate to what extent decoder LLMs are able to process a classic challenge for formal grammars -- Dutch cross-serial dependencies. Through a question-answering task, a minimal pairs analysis, and a maximum attention weight analysis, we observe that the models have implicitly acquired cross-serial dependencies in their internal representations to some extent. We also find that the models prefer nested dependencies over crossed ones, unlike humans.

Sparse Human-Interpretable Feature Trimming (SHIFT) uses Sparse Autoencoders (SAEs) to identify and ablate model components responsible for spurious correlations, and underpins SAE evaluation methods such as Spurious Correlation Removal and Targeted Probe Perturbation. We test whether SHIFT generalises beyond the Bias in Biographies dataset on which it was introduced. Testing on four datasets and two models, we replicate the original results on Bias in Biographies but find that SHIFT fails to reduce reliance on the spurious variable on the other three datasets. We conclude that SHIFT does not generalise reliably and that SAE evaluation methods built on it should be used with caution.

Told "Anna believes the cup is blue; in reality it is red," a capable language model answers blue about Anna and red about the world. We locate where that separation lives: in the routing of the attributed value, rather than in its storage. A generic value slot binds the attributed value; a router at the query position selects which frame, the character's belief or reality, a query reads out. The slot carries no belief–reality tag: intervening on it moves a reality readout as strongly as a belief one. The separation lives instead in a dissociated pair of routing subspaces that flip a query between frames without injecting the donor's value. Results hold across three architectures on stimuli de-confounded against theory-of-mind-benchmark shortcuts; the behavior itself emerges between 3B and 7B across five model families.

Autoregressive (AR) sequence models trained only to predict legal Othello moves develop an emergent, linearly-decodable representation of the board in a player-relative (MINE/YOURS) frame (Li et al., 2023; Nanda et al., 2023). We ask whether this emergent linear world model survives a change of training objective: from next-token prediction to masked diffusion (Sahoo et al., 2024). On an identical GPT-2 backbone and identical data, we train an AR control and a masked diffusion language model (MDLM), reproduce the Li–Nanda linear/relative dissociation on the control, and probe the MDLM with the same battery. We find that the linear player-relative world model transfers to masked diffusion (the dissociation reproduces) but is substantially weaker (~0.79 vs. the AR's 0.96 two-way accuracy). A Nanda-style causal intervention shows the representation is only minimally used: a statistically real but tiny and inconsistent effect (mean +0.02, 24% win-rate) against the AR's strong, consistent +0.21 / 84%. Order-artifact controls confirm the representation is genuine rather than an artifact of an easy unmasking order. As a reproducibility contribution, this is a generalizability stress test of the Li–Nanda finding under a change of training objective, anchored throughout by random-weight and shuffled-label control tasks and by component ablations. We report a second implementation, in Rust on an independent interpretability stack, that reproduces our headline numbers. Code, data recipes, and model checkpoints are available anonymously for review at https://anonymous.4open.science/r/othello-mdlm-anon-DBC2/.

A language model can learn a task from examples in its prompt. Function vectors (FVs) and task vectors (TVs) aim to transfer that behavior to a new prompt by modifying the model's activations. We reproduce both methods on word-pair tasks across GPT-J, Llama-3.1, and Gemma-2. In held-out tests, FVs outperform random controls on GPT-J and Llama-3.1-8B. TVs outperform shuffled-label controls on GPT-J and both base and instruction-tuned Gemma; the Llama-8B comparison is inconclusive. Our FV implementation achieves zero accuracy on both Gemma checkpoints in a two-task follow-up. Sentiment classification shows a further limitation: in-context learning (ICL) achieves 73.5–94.0% accuracy with arbitrary labels, while TVs achieve only 49.6–52.7%. Successful transfer on word pairs therefore does not extend to this sentiment dataset. Separating vector construction, layer selection, and testing also shows why gains over zero-shot accuracy can overstate the evidence for transfer. Code and data are available online.

We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-trained variants, and extend evaluations to five additional knowledge relation types from the ParaConflict dataset. We empirically confirm most of their original findings: larger models and higher-frequency entities tend to favor memorized answers, with substantial family-level variance.However, several conclusions do not generalize cleanly: entity-frequency effects disappear on Qwen3-14B and 32B; post-training shifts the memory-context trade-off inconsistently across families; question phrasing alone can change a model's reliance on memorized knowledge by up to 80 percentage points; and semantically unrelated prose can mimic coherent supporting context. Our results clarify where Yu et al.'s claims hold and to what extent they generalize to other prompts.

Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from steps to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs. Since human evaluation is costly, automated simulatability replaces human explainees with LLM simulators, as proposed in ConSim (Poché et al., 2025) for large-scale experiments. We qualitatively replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs, and identify two limitations. First, when class names are meaningful, simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations. Second, class anonymization can reward explanations for leaking the hidden label mapping, a limitation we expose with a new classes-as-concepts baseline. These results are consistent with a shortcut hypothesis: in the tested settings, simulator predictions mainly rely on task priors, while explanations produce small changes. We derive recommendations for more robust automated simulatability evaluations.

Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another language. We study this question using translation-initiation features (Wu et al., 2026). We reproduce the SAE feature discovery method from Wu et al. in Gemma 2 and extend it to multilingual settings that vary prompt language, source language, and target language. We then test whether features that recur across settings affect translation behavior by amplifying or ablating their activations during inference. We also examine whether the method can be applied to Gemma 3.

In both models, we observe an identical finding: although we can find more than 20 features that activate frequently across all discovery settings, causal validation shows that nearly all have small or inconsistent effects. In contrast, one feature -- Gemma 2's (L10, 5717) and Gemma 3's (L20, 2456) -- consistently improves COMET scores when amplified and degrades them when ablated across 23 language settings. These results show that feature recurrence can overstate cross-lingual transfer, while identifying a language-agnostic translation-initiation direction in Gemma 2 and Gemma 3.

In this reproduction paper we investigate subliminal learning, a consequence of distillation where teacher models transmit behavioral preference traits through semantically unrelated data. The original paper explores two types of traits (animal preferences and misalignment), three data modalities (number sequences, code, and chain of thought), and several model families. We reproduce their experiments and extend the setup along three axes: new preference categories (actors and politicians), a new task (chess move generation), and an additional open-weight model (Ministral8B). We also run a controlled ablation on the numbers task's answer-space size (1-, 2-, and 3-digit sequences). We focus on open-weight models with accessible checkpoints on HuggingFace, since the original paper's GPT-4.x fine-tuning is no longer available. Our reproduction supports the original paper's claims, but our extensions show they are not universal as transmission strength varies across traits and tasks, and one model shows almost no effect at all.

Large language models (LLMs) encode a wide range of information about the world, but understanding how such knowledge is represented remains a central challenge in mechanistic interpretability. In this work, we reproduce and extend a recent study investigating the representation of geographic information in LLMs using sparse autoencoders (SAEs). We focus on this work because, to the best of our knowledge, it is the only study that explicitly examines geographic representations in LLMs through the lens of feature-based interpretability. While the original study identifies SAE features that appear to encode geographic concepts, we extend the analysis to a broader set of locations spanning the globe and investigate the robustness of these findings. In particular, we hypothesize that some of the identified features may not represent geographic concepts directly, but may instead be driven by lexical properties of placenames, such as highly discriminative character sequences or syllables. To test this hypothesis, we introduce systematic perturbations of placenames that preserve their geographic reference while altering their orthographic form, and examine the resulting changes in SAE activation patterns and geographic activation maps. Our results provide a broader assessment of the reproducibility and generalizability of geographic representations identified by the original study, while highlighting potential confounds that should be considered when interpreting SAE features as semantic or conceptual representations. We position this work within the Generalizability and Baselines themes of the Reproducibility Challenge: we test whether the original findings hold beyond their initial experimental scope, and we introduce a simple lexical baseline that recovers most of the reported geographic signal from surface form alone, without access to the model's internal representations.

Two recent training interventions suppress unwanted behavior by changing the context in which it is learned. Inoculation prompting (IP) adds an instruction that explicitly elicits the unwanted trait during fine-tuning, then removes that instruction at test time. Counterfactual reflection training (CRT), introduced by Gurnee et al. (2026) in their J-space study, supervises only what a model says when interrupted and asked to reflect. We compare what these methods change, not only whether the target behavior disappears. We induce sycophancy in Qwen3-8B, repeat four core conditions across seeds 42–44, and test default behavior, correct-user agreement, prompted re-elicitation, NNsight activation steering, activation patching, and LoRA geometry. Under a frozen speech-act rubric, contaminated SFT affirms wrong proposals on 99.7% of responses; Strong IP averages 27.3% and post-hoc CRT repair 2.0%. CRT repair disputes correct users 56.9% of the time, versus 11.9% for Strong IP. Generic re-elicitation averages 98.5% for Strong IP and 81.0% for CRT repair. At a predeclared layer-18 sensitivity condition, steering raises Strong IP to 46% and CRT repair to 4%, while patching produces small, non-selective changes. The results show why reproducibility studies should test suppression, replacement behavior, and restoration.

Refusal directions have been shown to transfer cross-lingually with near-perfect effectiveness across safety-aligned languages (Wang et al., 2025), reportedly regardless of resource level. However, the only truly low-resourced language evaluated is Yoruba, which is strongly misaligned and shows limited transfer, leaving open how refusal directions behave for other low-resourced languages. We extend PolyRefuse to three additional low-resourced languages (Belarusian, Bashkir, and Tajik) and replicate the pipeline of (Wang et al., 2025) on Qwen2.5-14B-Instruct. We find that universality is graded rather than binary: directions from safety-aligned and partially-aligned languages transfer broadly, while those from strongly misaligned languages fail to reliably modulate refusal behavior on any target tested, including other misaligned languages, once ceiling effects are accounted for. Geometric alignment and behavioral transfer are further shown to be partially dissociated. These findings complicate the universality claim and suggest that users of the least-resourced, most strongly misaligned languages may face weaker safety guarantees.

What activation patching localizes can depend on where the patch is applied. Reproducing Ahsan et al. (2025) and extending it from four to ten models, we find the behavioral results largely reproduce and that medical adaptation improves calibration consistently only for gender with BioMistral, for MedGemma not at all. Two mechanistic findings are artifacts of the patch site. In Gemma-2 and OLMo-2 the patched MLP down-projection passes through a scale-invariant RMSNorm before the residual stream, so activation scaling is ineffective there. Patching after normalization restores it, raising the Gemma-2 flip rate on multiple sclerosis from $0.83$, flat under scaling, to $1.00$. It also lets a single layer recover the target race nearly as well as the reported sliding window ($0.83$ vs. $0.84$, OLMo-2), so the window compensated for the site, not a distributed representation.

Contrastive interpretability methods use pairs (or tuples) of inputs whose elements differ in minimal but controlled ways. Contrast-Consistent Search (CCS) is one such method that can identify whether (language) models encode binary features, such as sentence truth, in their internal activations. One advantage is that this requires no labels, meaning there is no assumption that internal activations agree with those labels, as is the case for supervised methods. In this work, we (1) reproduce results previously obtained with CCS, (2) better understand CCS’s objective, (3) propose Tuple-Contrastive PCA (tcPCA), a method with closed-form solutions, as a spiritual successor, and (4) show how tcPCA has wider applicability than CCS by: allowing for n-ary rather than just binary variables; and, finding contrastive subspaces instead of directions. See https://contrast.stefanfs.me for code and interactive figures.

Recent work has shown that finetuned language models can predict human creativity ratings of metaphors, motivating claims that they can comprehend creativity in non-literal language. We question the interpretation of a previously reported performance of finetuned GPT-2 and RoBERTa and investigate the signals underlying their predictions. Using ridge regression probes, random baselines, token-order ablations, and representation-evolution features, we find that pretrained representations perform comparably to finetuned models. Randomly initialized models retain moderate performance when token and positional information is preserved, and token shuffling does not eliminate the predictive signal. These results suggest that performance is supported largely by surface-level cues, including sentence length, token identity, and token co-occurrence. We therefore reproduce the original results but challenge the interpretation, highlighting the importance of controlled reproducibility studies on LM-related findings and claims from other domains.

The representation of grammatical knowledge in individual LLM neurons and attention heads remains poorly understood. We replicate the feature-based approach of Kryvosheieva et al. (2026) that uses functional localisation to illustrate that distinct syntactic phenomena can recruit the same units. We reproduce and extend the paper to new minimal pair datasets (Dutch, Turkish, and Russian) and additional interpretability techniques: Attention-based analysis, Differential Binary Masking (DBM), and Distributed Alignment Search (DAS). We find reproducible and robust evidence for syntactic localisation in LLMs across several methods and languages. However, localisation strength depends on interpretability techniques (particularly for DBM) and varies across languages, highlighting the value of validating mechanistic claims using several techniques. Code and data available via https://anonymous.4open.science/r/syntax-interpretability-F4C3

Training large language models to perform latent reasoning may improve performance, but the lack of legible reasoning poses challenges for model interpretability and safety assurance. This challenge has motivated interpretability research that aims to characterise the functional role of latent reasoning tokens. Some studies indicate that latent reasoning tokens are effective, whereas other work suggests that these tokens are under-utilised. In this paper, we consider the tension between existing findings by reproducing prior work which shows that some latent reasoning tokens are not necessary. Two interesting observations emerge from our experiments. First, we find that latent reasoning models often generate redundant tokens due to the fixed token budgets. Second, we observe that explicit reasoning models fine-tuned on gold reasoning traces may copy the result of the most recent reasoning step when generating an answer.

Attention sinks assign large, largely content-independent attention to the first position. Ran-Milo et al. (2026) explain GPT-2's sink with a parameter-level circuit: a learned query bias $\mathbf{b}_Q$ aligns with the key projection of the first position's effective positional encoding, whose few massive coordinates amplify the score. They infer that other architectures build sinks differently but do not measure this claim. We reproduce the original intervention effects and stress-test their causal interpretation. Baseline plus nine interventions and four structural analyses reproduce under the original evaluation, with two scope qualifications; the massive coordinates are exactly {138, 378, 447}. Yet nullifying $\mathbf{b}_Q$ leaves 44.5% of the sink, which follows an activation-dependent query route into the same first-position key. This anchor–route distinction unifies the extensions: with GPT-2 scale, the bias route fades while the activation-dependent route concentrates into fewer carrier heads; across 16 checkpoints in four families, no qualitative profile over mutually available conditions spans all families; and across five pre-training runs, anchoring recurs despite unstable coordinate identities. In GPT-2-small at 40 tokens, Swap EPE incurs about one quarter of the cross-entropy cost of Remove First PE. The stable phenomenon is an anchoring computation, not a universal parameter-level circuit, so mitigation must be validated against the target model's wiring.

Probing studies consistently find that language models (LMs) encode the truth of a statement along a low-dimensional linear direction of the residual stream. The Truth Co-occurrence Hypothesis (TCH) (Ravfogel et al., 2025) provides a mechanistic interpretation: tracking a latent truth variable is loss-reducing because true statements co-occur with true statements in natural text, evaluated using toy models and base LLMs. We replicate and extend the toy setting. In the tested fully trainable single-layer transformers, removing truth co-occurrence while preserving fact frequency eliminates linear truth decodability, supporting the necessity of co-occurrence in this setting. However, the same manipulation leaves the original toy model largely unchanged, revealing that probe-based evidence can arise from architectural artefacts rather than a functional truth-tracking mechanism. In pretrained Llama-3.1 models (8B and 70B), we reproduce the reported behavioural effect that false context reduces confidence in true facts, but show that nearly half of the original effect is explained by surface confounds. After controlling for these confounds, the effect remains robust across six factual relations. Under our probing protocol, cross-relation performance is closer to within-relation performance at 70B than at 8B. Our work characterises the emergence conditions for linear truth representations, and highlights the limitations of probe-based interpretability studies. Reproducibility Code is available here: https://github.com/Samzmc/truth-encoding-blackboxnlp26

Random-vector controls provide a reference for interpreting activation-steering effects, but a single realization hides variation in that reference. We examine this variation and two separate implementation checks. In a released activation-addition (ActAdd) configuration on GPT-2-XL, we reproduce the steering arm over 57 prompts with a maximum absolute error of 1.74 × 10⁻⁶. At coefficient 10, 9 of 20 fresh random-control realizations have greater mean normalized Kullback–Leibler (KL) divergence than the steering arm, and 19 have lower means than the stored random realization. In a separate four-model study over 96 prompts, changing batch size or attention implementation changes the greedy next token for at most 5 and 6 prompts, respectively; an identical rerun changes none. Finally, an erroneous left-padding index in our own analysis pipeline, shared by the intervention and readout, overstates the maximum greedy-token change rate in 10 of 12 model-dose settings, with a median relative distortion of 46.8%. These findings support reporting realization-level variation, measuring configuration sensitivity, and validating intervention and readout indices. The results concern distributional change in the evaluated configurations, not semantic steering success.

In this reproducibility study, we investigate the transparency and scrutability of recommender systems enhanced by incorporating generated natural-language user profiles that represent user preferences. The original paper explores the synthesis of user profiles from raw user-generated review text across domains such as movies and accommodations (Amazon Movies & TV, TripAdvisor). Crucially, these natural-language user profiles enable direct user interaction and intervention, allowing users to customize recommendations by correcting misattributed preferences or addressing cold-start settings. We successfully reproduce the core findings of the original study. Additionally, we extend the evaluation by conducting systematic context ablation experiments, multi-seed stability across five distinct random seeds to establish statistical reliability, and a mechanistic interpretability analysis using the nnsight framework to probe internal model representations under counterfactual profile perturbations. Our findings verify the original paper's claim that User Profile Recommendation (UPR) achieves competitive performance under its test-set reranking protocol and makes recommendations more transparent. Perturbing the natural-language profiles does change predictions, but it shifts predicted ratings uniformly across genres with no detectable genre-selective effect, leaving rankings unchanged even under direct activation steering. We trace this back to the rating-regression objective rather than the profile interface, with ranking-objective models clearly exceeding in this task.

Large language models (LLMs) are used as agents in social-media simulations, where behavioral fidelity may depend on preserving the affective intensity of real discourse. Using 3,692 unique Bluesky posts expanded into 14,768 activation records, we extract mild–strong affect directions in Qwen2.5 and Llama-3.1 Base and instruction-tuned checkpoints. Measured separation and reciprocal transfer to non-political hostility support a broader hostility-related feature rather than a uniquely partisan axis. In a six-prompt steering probe, checkpoints differ in their labeled hostile expression, refusal/deflection, and neutralization. Refusal/deflection includes substantive answers with disclaimers. Same-layer comparisons distinguish the hostility directions from a measured harmful-request contrast; removing that component changes behavioral counts non-uniformly and does not consistently return them to baseline. These descriptive results illustrate a representation–expression gap and motivate downstream fidelity audits.

Authorship attribution asks whether two texts were written by the same author. Models fine-tuned for this task with the same pretrained encoder, data, and loss can still differ four-fold in Recall@20 depending only on their scoring mechanism. We use mechanistic interpretability tools to explain this gap. Stylistic features such as word length, punctuation density, and function-word frequency are similarly available at every layer in every model we probe, including an off-the-shelf control encoder, suggesting that differences in feature availability do not explain the gap. Instead, causal intervention shows that the scoring mechanism determines where the encoder consolidates authorship signal. Mean pooling forces consolidation in the middle third of the network, while late interaction defers it to the upper third. We further derive this difference from the gradient structure of each scorer, and training dynamics reveal distinct learning trajectories that follow from that difference.

The term “linear representation hypothesis” (LRH) has appeared across diverse subfields of artificial intelligence, neuroscience, and cognitive science. But previous works have not consistently treated the LRH as a falsifiable scientific hypothesis; we analyze these inconsistencies and examine their implications for how prior theoretical and methodological results should be interpreted. Based on this analysis, we argue that claims regarding linear representations become well-defined only through careful examination of the model, representation location, feature definition, and evaluation dataset. We therefore propose a more rigorous formalization of the LRH that makes these dependencies explicit and allows the hypothesis to be evaluated as a falsifiable scientific claim. Finally, we identify some non-trivial open problems that warrant further attention from the research community.

Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood.

We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the $\textit{representational}$ level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely preserving semantic content. The vectors generalizes asymmetrically to writing systems unseen during construction: it flips non-Latin output to Latin, but maps Latin output into varied non-Latin scripts. At the $\textit{mechanistic}$ level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the model. Overall, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.

The rapid integration of Machine Learning in critical decision-making has shifted the focus of Explainable Artificial Intelligence (XAI) toward human-centric Natural Language Explanations (NLEs). While Large Language Models (LLMs) can synthesize narrative justifications from technical feature attributions, ensuring the faithfulness and robustness of these explanations remains an open challenge. Although multi-agent ensembles and Crowd-based Comparative Evaluation (CCE) are emerging standards for evaluating open-ended dialogues, they fail to capture domain-specific hallucinations and structural omissions inherent to technical explanations. To bridge this gap, we introduce MACE-XAI, a novel neuro-symbolic framework engineered specifically for the robust assessment of NLEs in the context of tabular feature-attribution XAI. Rather than proposing isolated evaluators, our key innovation lies in the architectural synergy of its components: we orchestrate a heterogeneous crowd of LLM agents to evaluate pragmatic text quality and fluency, while anchoring them with a deterministic symbolic verification layer that audits textual claims against raw technical ground truths. Experimental results demonstrate that while standalone LLM-as-judge setups are easily swayed by linguistic surface features, MACE-XAI achieves an agreement with human experts (Cohen's $\kappa=0.64$). While this represents a significant improvement over standalone neural judges, it also highlights the remaining alignment gap that must be addressed before fully autonomous evaluation in safety-critical deployments.

As Large Language Models (LLMs) continue to improve on complex tasks, evaluation has shifted from simple factoid retrieval to multi-hop reasoning. While existing studies analyze structural aspects of sequential steps, analysis from diverse multiple perspectives remains limited. This paper introduces a framework for the automated, multi-perspective analysis of QA datasets and its application to LLM answer diagnosis. An Analyst LLM is utilized to (1) generate explicit sequential steps from question-context-answer triples in a zero-shot setting, (2) synthesize a taxonomy of reasoning features from educational, logical, and professional perspectives. The resulting annotations are then used to (3) analyze associations between these features and Target LLM failures via decision tree training. Applied to the FRAMES, Multihop-RAG, and MuSiQue datasets, our decision tree analysis identifies dataset-specific bottlenecks such as Implicit Math in FRAMES, Conditioned Filtering in Multihop-RAG, and Temporal Order Logic Paradox in MuSiQue, overlooked by structural analysis.

Sign Language Translation (SLT) models integrate multiple visual channels, including the hands, body, and face, to generate written text, yet it remains unclear which channels the model actually relies on. We introduce Causal Stream Intervention (CSI), an intervention-based framework that measures each channel's contribution by suppressing it during decoding and observing the change in the model's predictions. This goes beyond attention weights, which show where a model looks but not what it depends on. Across two American Sign Language (ASL) datasets, we find that hand streams drive content-word prediction, while face and body streams contribute in a more variable, context-dependent way, with no single stream consistently dominating during question generation. Attention and routing scores offer a partial signal, but targeted interventions, particularly in later decoding layers, reveal functional stream use more faithfully.

As many benchmarks have become saturated, it is increasingly important to create new datasets that evaluate the generalization capacity of current state-of-the-art models in reasoning. However, creating high-quality reasoning datasets is challenging: manual construction is costly, and automatic generation is error-prone, with the community therefore relying on synthetic datasets with limited scope. In this paper, we propose the Minimal Expression Replacement GEneralization (MERGE) test, to evaluate the robustness of reasoning models against minimal and non-adversarial variants of existing evaluation datasets. First, high-quality variants are automatically obtained from the original instances using Masked Language Models (MLMs) for generation together with safeguarding filters, called Minimal Expression Replacement (MERE). We then apply the MERGE test to Natural Language Inference (NLI), a popular reasoning task, by using MERE on two popular existing NLI datasets. We evaluate multiple strong NLI models and LLMs, and the results indicate they generalize poorly: both struggle to consistently and correctly classify variants minimally different in form, but similar in reasoning, from the original ones. We also analyze how aspects of variant generation, such as word class and source MLMs, affect model performance.

Metaphor is a crucial aspect of human cognition. As such, we would like to understand how it is represented in large language models (LLMs) and how it might be controlled. We use mechanistic interpretability techniques to identify causal subspaces in a range of LLMs covering different architectures and sizes. We show that we are able to steer output on classification tasks across different parts of speech. We are also able to steer output on generation tasks, indicating that these techniques can be used to control the metaphoricity of LLM output.

Psycholinguistics studies show that human readers fall for coherence illusions: an incoherent discourse can seem coherent simply because a distractor matches what comes next. We investigate whether Dutch language models (6 monolingual and 4 multilingual) show the same behavior on texts that link back to earlier context with words such as 'again' and 'too'. First, we find that surprisal at the critical word tracks human acceptability judgments and eye-tracking data. Models are more surprised by incoherent continuations, but a matching distractor in the prior context reduces this surprisal. Second, attention entropy identifies heads that behave differently under coherence vs. incoherence. We find that ablating these heads shows transfer effects across experiments, suggesting a shared mechanism. Third, we introduce energy from the associative-memory literature as a metric to quantify discourse coherence. Taken together, our results show that coherence illusions arise in Dutch LLMs, with entropy and energy exposing mechanisms that operate across settings.