For a list of all posts go here.
Building Proto-Affective Agents with Active Inference
Most artificial agents today are built around a simple idea: maximize reward.
Whether in reinforcement learning, game-playing systems, or autonomous robotics, the agent is typically given a predefined objective and learns policies that optimize it. While this approach has produced impressive results, it often leaves open a deeper question:
How can behavior emerge from the need to maintain oneself in the world, rather than from externally defined rewards?
This question led me to explore Active Inference — a framework originating in theoretical neuroscience that models biological organisms not as reward maximizers, but as systems that continuously minimize prediction errors while maintaining preferred states.
Instead of explicitly coding emotions, drives, or goals, I wanted to investigate whether simple emotional and motivational dynamics could emerge naturally from:
self-maintenance,
uncertainty,
prediction errors,
and preferred states.
The result so far is a series of minimal simulations that progressively evolve from a simple persistent self-model toward exploratory, proto-affective behavior.
This post describes the first two scenarios.
Scenario 1 — The Minimal Self
The first experiment began with the simplest possible Active Inference organism.
The agent exists in a two-dimensional grid world and possesses only:
beliefs about its own position,
noisy observations generated from those beliefs,
and a weak prior that its inferred state should persist.
At this stage:
there are no targets,
no threats,
no rewards,
and no external objectives.
The entire system revolves around minimizing prediction errors between:
inferred hidden states,
and noisy sensory consequences.
The core loop is extremely simple:
Generate noisy observations from current beliefs.
Compute prediction errors.
Compute Variational Free Energy (VFE).
Update beliefs to reduce prediction errors.
Repeat.
The interesting part emerged when experimenting with priors.
Without a prior on self-state persistence, the system naturally collapses toward zero. Since the observations are attenuated versions of beliefs, minimizing prediction errors alone gradually dampens all internal dynamics.
To prevent this collapse, I introduced a weak self prior:
a preferred inferred state toward which the system is softly attracted.
This simple modification creates a minimal form of self-maintenance.
Importantly, the “self” here is not a symbolic entity or explicit representation. It is simply:
a persistent inferred state stabilized by priors.
This distinction turned out to be surprisingly profound.
The system only begins to exhibit coherent persistence once there is a meaningful separation between:
the current inferred state,
and the preferred inferred state.
Without that distinction, there is no homeostasis, no regulation, and no meaningful self-maintenance.
Figure 1 — Scenario 1: trajectory and Variational Free Energy over time. The weak self prior prevents collapse of inferred self-state, allowing persistent self-maintenance dynamics to emerge.
Scenario 2 — Hunger, Uncertainty, and Proto-Exploration
Once the minimal self became stable, the next step was introducing motivation.
Rather than hardcoding rewards or explicit goals, I introduced the idea of “hunger” as:
increased probability of preferred states associated with nutrition.
At this stage, “nutrition” remained abstract. It could represent:
food,
water,
information,
or knowledge.
The important point was that the agent now possessed uncertain beliefs about meaningful external states.
Initially, the target’s position was completely unknown.
Instead of assigning a fixed target location, target beliefs were initialized as noise, representing uncertainty about where meaningful states might exist.
The agent now maintained:
beliefs about itself,
beliefs about targets,
separate prediction errors,
and separate Variational Free Energies.
The total agent free energy became:
An important design choice was avoiding fixed target locations. Instead, the system sampled distributed target priors across the world, encoding:
“meaningful states may exist somewhere.”
This produced surprisingly rich dynamics.
The agent began exhibiting:
wandering,
oscillatory movement,
switching motivational tendencies,
and persistent exploration-like behavior.
At this point, I deliberately avoided labeling emotions explicitly.
One particularly interesting observation was that unresolved prediction errors could plausibly correspond to multiple emotional interpretations simultaneously.
For example:
unresolved uncertainty may resemble curiosity,
but also anxiety,
vigilance,
anticipation,
or exploratory tension.
This suggested a more interesting possibility:
emotions may emerge as modes of uncertainty regulation, rather than predefined symbolic states.
The same underlying dynamical signal can produce very different affective behaviors depending on:
precision weighting,
temporal persistence,
controllability,
and stability of the self-model.
To track these dynamics, I introduced several internal indicators:
unresolved target prediction errors,
motivational drive strength,
and surprise.
Surprise was defined as temporal change in free energy:
This produced clear spikes whenever internal beliefs became suddenly invalidated.
Although the system still lacks true environmental sensing, it already exhibits a form of internally generated exploratory pressure:
unresolved uncertainty itself destabilizes the system,
driving persistent wandering even in the absence of visible targets.
This is particularly interesting because biological organisms rarely remain perfectly still when no explicit target or threat is visible. Exploration itself appears to be an intrinsic regulatory process.
Figure 2 — Scenario 2: exploratory trajectories generated by uncertain target beliefs and distributed motivational priors. Persistent wandering emerges without explicit reward maximization or path-planning routines.
Toward Epistemic Behavior
The next scenario will introduce a major transition:
actual hidden targets,
visibility radius,
and position-dependent observations.
At that point:
movement will change information,
uncertainty reduction will become spatially meaningful,
and exploration will become genuinely epistemic.
This is where curiosity may begin to emerge not merely as unresolved internal tension, but as active information-seeking behavior.
What makes this direction especially fascinating is that none of these behaviors are explicitly programmed as emotions.
Instead, they emerge progressively from:
prediction error minimization,
preferred states,
uncertainty regulation,
and self-maintaining inference dynamics.
The long-term goal is not to build an “emotional AI” in the conventional sense, but to explore whether affective organization itself can emerge from the mathematics of self-maintaining inference.
I suspect we are only at the very beginning of that exploration.
No comments:
Post a Comment