Publications related to Activation Scaling for Steering and Interpreting Language Models

Related publications (22)

XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models

Self-supervised pretrained models exhibit competitive performance in automatic speech recognition (ASR) on finetuning, even with limited in-domain supervised data. However, popular pretrained models are not suitable for streaming ASR because they are train ...

Institute of Electrical and Electronics Engineers2025

Adapting Fine-Grained Cross-View Localization to Areas Without Fine Ground Truth

Zimin Xia

Given a ground-level query image and a geo-referenced aerial image that covers the query’s local surroundings, fine-grained cross-view localization aims to estimate the location of the ground camera inside the aerial image. Recent works have focused on dev ...

Springer Science and Business Media Deutschland GmbH2025

Diffusion Stochastic Optimization for Min-Max Problems

Ali H. Sayed, Haoyuan Cai

The optimistic gradient method is useful in addressing minimax optimization problems. Motivated by the observation that the conventional stochastic version suffers from the need for a large batch size on the order of O(ϵ-2) to achieve an ϵ-stationary solut ...

2025

Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR

Karl El Hajal

Automatic speech recognition (ASR) systems are well known to perform poorly on dysarthric speech. Previous works have addressed this by speaking rate modification to reduce the mismatch with typical speech. Unfortunately, these approaches rely on transcrib ...

Institute of Electrical and Electronics Engineers Inc.2025

First and Zeroth-Order Implementations of the Regularized Newton Method with Lazy Approximated Hessians

Nikita Doikov

In this work, we develop first-order (Hessian-free) and zeroth-order (derivative-free) implementations of the Cubically Regularized Newton Method for solving general non-convex optimization problems. For that, we employ finite difference approximations of ...

SPRINGER/PLENUM PUBLISHERS2025

Knowledge Distillation for Face Recognition Using Synthetic Data With Dynamic Latent Sampling

Hatef Otroshi Shahreza

State-of-the-art face recognition models are computationally expensive for mobile applications. Training lightweight face recognition models also requires large identity-labeled datasets, raising privacy and ethical concerns. Generating synthetic datasets ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2024

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

Xiuying Wei, Caglar Gulcehre, Skander Moalla

State-of-the-art results in large language models (LLMs) often rely on scale, which becomes computationally expensive. This has sparked a research agenda to reduce these models' parameter counts and computational costs without significantly impacting their ...

2024

Settlement Prediction for Concrete Face Rockfill Dams Considering Major Factor Mining Based on the HHO-VMD-LSTM-SVR Model

Sen Zheng

Some important discoveries have been revealed in some studies, including that the settlement of concrete face rockfill dams (CFRDs) may cause cracks in the concrete face slabs, which may lead to dam collapse. Therefore, deformation behavior prediction of C ...

MDPI2024

EdgeFace: Efficient Face Recognition Model for Edge Devices

Hatef Otroshi Shahreza

In this paper, we present EdgeFace - a lightweight and efficient face recognition network inspired by the hybrid architecture of EdgeNeXt. By effectively combining the strengths of both CNN and Transformer models, and a low rank linear layer, EdgeFace achi ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2024

Distributed Adaptive Learning Under Communication Constraints

Ali H. Sayed

We consider a network of agents that must solve an online optimization problem from continual observation of streaming data. To this end, the agents implement a distributed cooperative strategy where each agent is allowed to perform local exchange of infor ...

Ieee-Inst Electrical Electronics Engineers Inc2024

Finding Actual Descent Directions For Adversarial Training

Volkan Cevher, Fabian Ricardo Latorre Gomez, Igor Krawczuk, Thomas Michaelsen Pethick, Leello Tadesse Dadi

Adversarial Training using a strong first-order adversary (PGD) is the gold standard for training Deep Neural Networks that are robust to adversarial examples. We show that, contrary to the general understanding of the method, the gradient at an optimal ad ...

2023

SKILL: Structured Knowledge Infusion for Large Language Models

Martin Jaggi

Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better internalize knowledge from a structured data, such as a knowledge graph, or from ...

ASSOC COMPUTATIONAL LINGUISTICS-ACL2022

Generalisable Methods for Early Prediction in Interactive Simulations for Education

Tanja Christina Käser Jacober, Christian Giang, Jade Maï L Cock

Interactive simulations allow students to discover the underlying principles of a scientific phenomenon through their own exploration. Unfortunately, students often struggle to learn effectively in these environments. Classifying students’ interaction data ...

2022

Learning of Continuous and Piecewise-Linear Functions With Hessian Total-Variation Regularization

Michaël Unser, Shayan Aziznejad, Joaquim Gonçalves Garcia Barreto Campos

We develop a novel 2D functional learning framework that employs a sparsity-promoting regularization based on second-order derivatives. Motivated by the nature of the regularizer, we restrict the search space to the span of piecewise-linear box splines shi ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Predicting in Uncertain Environments: Methods for Robust Machine Learning

Paul Thierry Yves Rolland

One of the main goal of Artificial Intelligence is to develop models capable of providing valuable predictions in real-world environments. In particular, Machine Learning (ML) seeks to design such models by learning from examples coming from this same envi ...

EPFL2022

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

Andreas Krause, Maryam Kamgarpour

We consider model-based multi-agent reinforcement learning, where the environment transition model is unknown and can only be learned via expensive interactions with the environment. We propose H-MARL (Hallucinated Multi-Agent Reinforcement Learning), a no ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Visual Correspondences for Unsupervised Domain Adaptation on Electron Microscopy Images

Pascal Fua, Mathieu Salzmann, Carlos Joaquin Becker, Róger Bermúdez Chacón, Okan Altingövde

We present an Unsupervised Domain Adaptation strategy to compensate for domain shifts on Electron Microscopy volumes. Our method aggregates visual correspondences—motifs that are visually similar across different acquisitions—to infer changes on the parame ...

Institute of Electrical and Electronics Engineers2020

Short-term energy use prediction of solar-assisted water heating system: Application case of combined attention-based LSTM and time-series decomposition

Dolaana Khovalyg, Amirreza Heidari

With improved insulation of building envelopes and the use of low-temperature space heating systems, the share of energy use for domestic hot water (DHW) production in buildings has increased significantly, and nearly become the most energy-expensive servi ...

2020

On enforcing the necessary conditions of optimality under plant-model mismatch - What to measure and what to adapt?

Martand Singhal

Industrial processes are run with the aim of maximizing economic profit while simultaneously meeting process-critical constraints. To this end, model-based optimization can be performed to ensure optimal plant operations. Usually, inevitable model inaccura ...

EPFL2018

Dimensionally Tight Bounds for Second-Order Hamiltonian Monte Carlo

Nisheeth Vishnoi, Oren Rami Mangoubi

Hamiltonian Monte Carlo (HMC) is a widely deployed method to sample from high-dimensional distributions in Statistics and Machine learning. HMC is known to run very efficiently in practice and its popular second-order "leapfrog" implementation has long bee ...

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)2018