Curiosity, Motivation and Discomfort

  For a list of all posts go here.

Curiosity Helps Agents Find Their Way: Targets, Barriers, and Active Inference

Imagine an agent moving through a 2D world. In this world, some points are targets — places the agent wants to reach — and some structures are barriers — obstacles that push the agent away. Both can be thought of as invisible “fields”: targets attract, barriers repel. The agent’s task is to navigate this space.

How does the agent decide what to do?

The agent doesn’t see the full map in advance. Instead, it relies on Active Inference, a framework from neuroscience and AI. In Active Inference, the agent constantly predicts what should happen if it acts a certain way, compares this to what actually happens, and adjusts its beliefs and actions to minimize the difference (prediction errors).

This process is described mathematically by the minimization of Free Energy:

• Variational Free Energy (VFE): measures surprise about the present.

• Expected Free Energy (EFE): looks ahead to evaluate future actions.

EFE has two parts:

• A pragmatic term: “Will this action get me closer to my goal?”

• An epistemic term (curiosity): “Will this action help me learn something new about the environment?”

---

The Barrier with an Opening

Let’s take a concrete example.

• The agent starts on one side of the map.

• A fixed target is placed on the other side.

• Between them lies a wall-like barrier, with a small opening at the top.

The agent has to find that opening to reach the target.

• Without curiosity: The agent focuses only on the pragmatic term of EFE (minimizing present prediction errors). It feels the attraction of the target, but the barrier blocks the way. Usually, the agent just presses against the wall and fails to reach the goal.

• With curiosity: The agent balances the pragmatic and epistemic terms. It tries different strategies, explores new directions, and eventually discovers the opening. Once it passes through, the pragmatic drive takes over, and the agent reaches the target.

                                Figure 1: Without curiosity.

                               Figure 2: With curiosity.

                                

---

Motivation and Discomfort

To make things more interesting, we modeled two extra subjective-like signals:

• Motivation: linked to the distance to the target. The further away, the stronger the pull.

• Discomfort: linked to the distance to the barrier. The closer the agent gets to the wall, the stronger the push.

Both of these influence prediction errors, meaning the agent doesn’t just navigate mechanically — it feels motivated to move forward and uncomfortable when too close to obstacles. These signals are showed in the figures above.

---

A Twist: Sensitization to Motivation

We also added a mechanism we call Sensitization to Motivation. This is like a gain control that determines how strongly the agent reacts to motivational drives. Interestingly, once the target is reached, this gain drops close to zero — meaning the agent stops feeling driven once it achieves its goal.

This looks a lot like how dopamine works in biological brains: high when we anticipate rewards, but decreasing once the goal is achieved (Schultz et al., 1997).

                               Figure 3: Metrics for Fig. 2.

---

Why Does This Matter?

This simple simulation shows why curiosity matters. A purely pragmatic agent gets stuck. But an agent that values both pragmatic and epistemic terms — in other words, one that allows curiosity to guide exploration — finds the solution.

The framework also lets us connect navigation to subjective states like motivation, discomfort, and curiosity, and even to neurobiology through concepts like dopamine.

---

What’s Next?

• Adding more targets and barriers to see how agents handle complex maps.

• Exploring how different levels of curiosity affect efficiency.

• Connecting the model more explicitly to neurotransmitter systems, like dopamine (motivation), acetylcholine (uncertainty), and serotonin (risk).

---

References

• Friston, K., et al. (2017). Active Inference: A Process Theory. Neural Computation, 29(1), 1–49.

• Schwartenbeck, P., et al. (2019). Computational mechanisms of curiosity and goal-directed exploration. Nature Neuroscience, 22(3), 437–447.

• Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.