Discovering Hierarchy in Reinforcement Learning: Automatic Modelling of Task-Hierarchies by Machines through Sense-Act Interactions with their Environments

We are relying more and more on machines to perform tasks that were previously the sole domain of humans. There is a need to make machines more self-adaptable and for them to set their own sub-goals. Designing machines that can make sense of the world they inhabit is still an open research problem. Fortunately many complex environments exhibit structure that can be modelled as an inter-related set of subsystems. Subsystems are often repetitive in time and space and reoccur many times as components of different tasks. A machine may be able to learn how to tackle larger problems if it can successfully find and exploit this repetition. Evidence suggests that a bottom up approach, that recursively finds building-blocks at one level of abstraction and uses them at the next level, makes learning in many complex environments tractable. This book describes a machine learning algorithm called HEXQ that automatically discovers hierarchical structure in its environment purely through sense-act interactions, setting its own sub-goals and solving decision problems using reinforcement learning.