Research Mission
Seeking the core rules of interaction with the world
Robots that learn from few demonstrations need to understand which actions a scene allows and what those actions will do. I study two forms of this knowledge, affordances and equivariances, and combine analytical models with learning to identify them from small amounts of data.
Full statement
Affordances describe which actions a scene supports, such as pulling a drawer by its handle or stepping onto a curb. Equivariances describe how the result of an action changes when the scene is transformed, for example rotated or mirrored, so that one experience covers many. Large networks can pick up both, but only implicitly and from large datasets. Hand-built physics models state them explicitly, but only for systems simple enough to model by hand. Combining the two keeps what is learned in a form that can be checked and reused on new tasks.
Part of the inspiration is Conway's Game of Life and especially Particle Life, where a small table of attraction rules between particle types produces complex, organized behavior. Efficient behavior in robots can also rest on a small set of rules.
Research Map
An arrow from one topic to another means the first helps with the second. Pick a topic to see its connections, or pick an arrow to see the reason for it.
Perceptual priors Priors from infant development, built into a model's structure and loss.
- helps with AffordancesMany infant priors concern what can be done with objects, so they act as priors over affordances.
- helps with ArchitectureInfant priors suggest what a model should represent and what its loss should reward.
Foundation models Large pretrained models have learned many rules implicitly. Distilling them can make those rules explicit.
- helps with Symbolic distillationFoundation models are trained on far more data than one robot can collect. Distilling them can recover important rules that were previously out of reach.
Information theory Entropy as a measure of how important each feature is for finding these rules.
- helps with Symbolic distillationEntropy measures how important each feature is, so distillation can build rules from the features that matter most.
Architecture Choosing architectures for the structure they can represent as well as their accuracy.
- from Perceptual priorsInfant priors suggest what a model should represent and what its loss should reward.
- helps with AffordancesAn output shaped like the affordance, such as SPARTA's traversability over heading, lets the model represent it directly.
- helps with EquivariancesEquivariant layers enforce a symmetry by construction, which reduces the data needed.
Symbolic distillation Train a small, structured network, then extract closed-form rules from it.
- from Foundation modelsFoundation models are trained on far more data than one robot can collect. Distilling them can recover important rules that were previously out of reach.
- from Information theoryEntropy measures how important each feature is, so distillation can build rules from the features that matter most.
- helps with AffordancesDistilling a network that has learned which actions succeed gives an explicit affordance model.
- helps with EquivariancesDistillation can turn symmetries a network picked up into explicit equivariances.
Koopman theory Find a space in which nonlinear dynamics act linearly, where structure is easier to see.
- helps with AffordancesIn a space where the dynamics are linear, the effect of an action is easier to predict, so affordances may be easier to identify there.
- helps with EquivariancesSymmetries of the dynamics show up as properties of the linear Koopman operator and can be read off in closed form.
Affordances Which actions a scene supports, linking what a robot sees to what it can do.
- from Perceptual priorsMany infant priors concern what can be done with objects, so they act as priors over affordances.
- from ArchitectureAn output shaped like the affordance, such as SPARTA's traversability over heading, lets the model represent it directly.
- from Symbolic distillationDistilling a network that has learned which actions succeed gives an explicit affordance model.
- from Koopman theoryIn a space where the dynamics are linear, the effect of an action is easier to predict, so affordances may be easier to identify there.
- from Reinforcement learningTrial and error in reinforcement learning produces the experience from which affordances can be learned.
- helps with Reinforcement learningKnowing what a scene affords rules out actions that will fail, as in SACRED.
Equivariances How an outcome changes when the scene is transformed, so one experience covers many.
- from ArchitectureEquivariant layers enforce a symmetry by construction, which reduces the data needed.
- from Symbolic distillationDistillation can turn symmetries a network picked up into explicit equivariances.
- from Koopman theorySymmetries of the dynamics show up as properties of the linear Koopman operator and can be read off in closed form.
- helps with Reinforcement learningA policy that respects a task's equivariances transfers to rotated or mirrored versions without new data.
- helps with Hybrid modelsKnown equivariances can form the analytical part of a hybrid model.
Reinforcement learning Where these rules have been tested so far. The aim is constraints that make any learning method efficient.
- from AffordancesKnowing what a scene affords rules out actions that will fail, as in SACRED.
- from EquivariancesA policy that respects a task's equivariances transfers to rotated or mirrored versions without new data.
- from Hybrid modelsA hybrid model predicts the effect of an action accurately from little data, so a learner needs fewer demonstrations.
- helps with AffordancesTrial and error in reinforcement learning produces the experience from which affordances can be learned.
Hybrid models An analytical part fixes known properties of the problem in advance and makes established tools usable. A network learns the rest.
- from EquivariancesKnown equivariances can form the analytical part of a hybrid model.
- helps with Reinforcement learningA hybrid model predicts the effect of an action accurately from little data, so a learner needs fewer demonstrations.
Perceptual priors → Affordances
Many infant priors concern what can be done with objects, so they act as priors over affordances.
Perceptual priors → Architecture
Infant priors suggest what a model should represent and what its loss should reward.
Foundation models → Symbolic distillation
Foundation models are trained on far more data than one robot can collect. Distilling them can recover important rules that were previously out of reach.
Information theory → Symbolic distillation
Entropy measures how important each feature is, so distillation can build rules from the features that matter most.
Architecture → Affordances
An output shaped like the affordance, such as SPARTA's traversability over heading, lets the model represent it directly.
Architecture → Equivariances
Equivariant layers enforce a symmetry by construction, which reduces the data needed.
Symbolic distillation → Affordances
Distilling a network that has learned which actions succeed gives an explicit affordance model.
Symbolic distillation → Equivariances
Distillation can turn symmetries a network picked up into explicit equivariances.
Koopman theory → Affordances
In a space where the dynamics are linear, the effect of an action is easier to predict, so affordances may be easier to identify there.
Koopman theory → Equivariances
Symmetries of the dynamics show up as properties of the linear Koopman operator and can be read off in closed form.
Affordances → Reinforcement learning
Knowing what a scene affords rules out actions that will fail, as in SACRED.
Equivariances → Reinforcement learning
A policy that respects a task's equivariances transfers to rotated or mirrored versions without new data.
Equivariances → Hybrid models
Known equivariances can form the analytical part of a hybrid model.
Reinforcement learning → Affordances
Trial and error in reinforcement learning produces the experience from which affordances can be learned.
Hybrid models → Reinforcement learning
A hybrid model predicts the effect of an action accurately from little data, so a learner needs fewer demonstrations.
Sources of structureMethodsCore: rules of interactionEfficient learningHas related past work
Research Topics
Affordances
Gibson introduced affordances as the action possibilities an environment offers an agent. A handle can be pulled, a ledge can be stepped on, and a gap can be passed through. Because they are defined by what the agent can do, affordances describe directly how the world responds to action.
They help in two ways. They are compact enough to learn once and reuse across tasks, and once known they rule out actions that will fail, so a learner spends fewer trials on them. Traversability, whether a robot can cross a patch of ground at a given heading, is one example.
- SACRED: affordance-guided explorationMaster's research. Affordances learned from images steer a Q-learner toward actions likely to work
- Density estimation for affordance detectionChoosing an online density estimator for SACRED's affordance models
- SPARTA: smooth state-dependent traversabilityCoRL 2025. Traversability as a function of heading, an affordance of terrain
- Gibson (1979). The Ecological Approach to Visual PerceptionIntroduced affordances: what the environment offers an animal for action.
- Şahin et al. (2007). To afford or not to affordA formal definition of affordances for robot control.
- Khetarpal et al. (2020). What can I do here? A theory of affordances in reinforcement learningAffordances as a way to restrict the actions a learner considers.
Equivariances
Many tasks are symmetric. Grasping a mug works the same way after the mug is rotated, and terrain that is easy to cross heading north is usually easy heading south. An equivariant model changes its output in a known way when its input is transformed, so what it learns in one configuration carries over to the transformed ones.
Most equivariant methods build a chosen symmetry into the network ahead of time. I am interested in finding which equivariances a task actually has from data, and representing them explicitly so they can be checked, combined with affordances, and reused.
- SPARTAA periodic (Fourier) basis builds the wrap-around in heading into the model
- Cohen & Welling (2016). Group equivariant convolutional networksBuilt equivariance to rotations and reflections into convolutional networks.
- van der Pol et al. (2020). MDP homomorphic networksSymmetries of a task built into the policy in reinforcement learning.
Koopman theory
Koopman operator theory states that nonlinear dynamics can be represented linearly, given the right space of observables and the operator that acts on it. Finding that space and operator turns nonlinear behavior into linear behavior that can be analyzed in closed form.
This makes Koopman theory a way to look for structure. Affordances and equivariances may be easier to find in a space where the dynamics are linear than in the original one.
In our hybrid Koopman work, the operator is found by semidefinite programming and neural networks learn only the maps into and out of the lifted space. With the structure in the analytical part, the networks have less to learn and need less data.
- Hybrid Koopman operator learningCDC 2025, co-first author, oral presentation. SDP finds the operator, a network learns only the maps
- Koopman (1931). Hamiltonian systems and transformation in Hilbert spaceRepresented nonlinear dynamics as a linear operator on a space of functions.
- Mezić (2005). Spectral properties of dynamical systems, model reduction and decompositionsSpectral analysis of the Koopman operator for model reduction.
- Lusch, Kutz & Brunton (2018). Deep learning for universal linear embeddings of nonlinear dynamicsA deep network learns the space in which the dynamics become linear.
Symbolic distillation
This runs in the opposite order. A small network with a suitable structure is trained first, and the function it learns is then distilled into a symbolic expression. Tools for this include sparse regression over a library of candidate terms (SINDy), symbolic regression (AI Feynman), and pruning a KAN down to a formula.
Either order can work. What I care about is the result, an affordance or equivariance written in a form that can be inspected and reused, found from less data than a network or a hand-built model would need on its own.
- Schmidt & Lipson (2009). Distilling free-form natural laws from experimental dataRecovered physical laws from motion data by symbolic regression.
- Brunton, Proctor & Kutz (2016). Sparse identification of nonlinear dynamicsSparse regression selects the few terms of a governing equation.
- Cranmer et al. (2020). Discovering symbolic models from deep learning with inductive biasesTrained a graph network, then distilled its parts into equations.
Architecture
The architecture decides what structure a network can represent. Equivariant layers enforce a symmetry by construction. An output layer shaped like the target also helps. SPARTA predicts a Fourier series over heading instead of a single number, which makes the learned affordance easier to read and needs less data to fit.
- SPARTAThe network outputs a Fourier series over heading, not one number
- Hybrid Koopman operator learningStructure goes in the operator, learning goes in the maps
- Battaglia et al. (2018). Relational inductive biases, deep learning, and graph networksArchitectures as inductive biases, with graph networks as a general form.
Perceptual priors
Infants learn quickly partly because they start with expectations about objects, contact, and motion. Many of these expectations concern what can be done with things, so they work as priors over affordances.
Building such priors into a model's architecture and loss, instead of waiting for it to rediscover them from data, should make training more data efficient. Our ICDL 2022 paper used an infant-style curriculum in this way and reached up to 10× better sample efficiency.
- Sensorimotor abstraction on curriculaICDL 2022. An infant-style curriculum, up to 10× more sample-efficient
- Baillargeon, Spelke & Wasserman (1985). Object permanence in five-month-old infantsFive-month-old infants expect a hidden object to continue to exist.
- Spelke (1990). Principles of object perceptionInfants perceive objects as cohesive, bounded bodies that move only on contact.
- Spelke & Kinzler (2007). Core knowledgeA small set of systems for objects, agents, number and space, present early in life.
- Lake et al. (2017). Building machines that learn and think like peopleArgues that machines should start from such core knowledge to learn from little data.
Foundation models
Large pretrained models encode a lot about how objects behave, including many affordances and symmetries, but only implicitly. Distilling them can recover important rules that no single robot dataset is large enough to reveal. They can also supply training signal for smaller models, and they give a check on whether rules found by other methods agree with what these models learned.
- Bommasani et al. (2021). On the opportunities and risks of foundation modelsDefined foundation models and surveyed their capabilities and limits.
- Hinton, Vinyals & Dean (2015). Distilling the knowledge in a neural networkTransfers what a large model has learned into a smaller one.
Information theory
Information theory measures how much each feature tells a model about the outcome. Entropy and KL divergence rank features by importance, so rule discovery can focus on the features that matter and keep the resulting rules short.
- Multi-intention inverse RLMaximum-entropy IRL, extended to several reward functions at once
- Density estimation for affordance detectionComparing estimators by JS and KL divergence
- DefendaGANA KL diversity term keeps generators from collapsing
- Shannon (1948). A mathematical theory of communicationDefined entropy as a measure of information.
- Tishby, Pereira & Bialek (1999). The information bottleneck methodKeeps only the information in a feature that is relevant to the target.
Hybrid models
A hybrid model combines an analytical component with a learned one. The analytical part does two things. It derives important properties of the problem in advance, such as the dimension of the state or the equivariances and affordances it has. It also makes established tools usable, such as semidefinite programming or model-based planning, which apply to analytical models but not to networks. A network models the residual that is hard to write down. With these properties fixed in advance, the data needed grows more slowly with the size of the state.
Hybrid models are a means to efficient learning, and the rules discovered elsewhere on this page are what they are built from.
- Hybrid Koopman operator learningOptimization handles the structure, learning handles what is hard to write down
- ENCORE: cost-optimal fault recoveryFirst author. A learned repair model inside model-based mission rollouts
- Raissi, Perdikaris & Karniadakis (2019). Physics-informed neural networksKnown physical laws added to the loss of a network.
- Zeng et al. (2020). TossingBot: learning to throw arbitrary objects with residual physicsA network learns a correction on top of a physics model of throwing.
Reinforcement learning
The aim is to find constraints that make any learning method more efficient, not reinforcement learning in particular. My background is in reinforcement and imitation learning, so that is where these rules have been tested so far. RL is also a basis for training, since trial and error produces the experience from which rules such as affordances can be learned.
Affordances narrow the set of actions worth trying, and equivariances let one experience count for its rotated or mirrored versions. Both reduce the data a learner needs. SACRED showed this for affordances, and the infant-style curriculum gave up to 10× better sample efficiency.
- Multi-intention inverse RLLearning several rewards from mixed demonstrations
- SACREDAffordance priors made Q-learning much faster than undirected exploration
- Sensorimotor abstraction on curriculaUp to 10× sample efficiency and transfer without retraining
- TA, graduate RL and MLUMass COMPSCI 687 and 589
- Sutton & Barto (2018). Reinforcement learning: an introductionThe standard text on reinforcement learning.
- Mnih et al. (2015). Human-level control through deep reinforcement learningDeep reinforcement learning from pixels on Atari games.