Observing its Motivations

Whatever motivational structure we give to robots, they should be able to observe and reason about it. For many purposes a simple goal-subgoal structure is the right thing. However, there are some elaborations to consider.

1. There often will be auxiliary goals, e.g. curiosity. When a robot is not otherwise occupied, we will want it to work at extending its knowledge.

2. The obverse of an auxiliary goal is a constraint. Maybe shall want something like Asimov's science fiction laws of robotics, e.g. that a robot should not harm humans. In a sufficiently general way of looking at goals, achieving its other goals with the constraint of not harming humans is just an elaboration of the goal itself. However, since the same constraint will apply to the achievement of many goals, it is likely to be convenient to formalize them as a separate structure. A constraint can be used to reduce the space of achievable states before the details of the goals are considered.