When agent performs actions suggested by the current policy it is said to be in exploitation mode. When agent tries actions it would have never done otherwise and thereby gain knowledge of the environment it is said to be in exploration mode. During exploration mode the agent tries different actions to gain knowledge of the environment and use this knowledge of environment to learn optimal policy. Therefore the crucial question is asked, “How much time should the agent spend on exploration versus exploitation?” This question created the exploration-exploitation dilemma, which is still an open research problem.
Why the Dilemma Matters?
To obtain a lot of reward, an agent must prefer actions that it has tried in the past and found effective in producing reward. But to discover such actions, it has to try actions that it has not selected before. In other words, the agent has to exploit what it has already experienced in order to obtain reward but it also has to explore in order to make better action selections in the future. The choice of how to handle this trade-off can depend on the problem being solved and environment. Popular strategies are greedy 𝜖 and greedy 𝜖 simulated with annealing.
Popular Strategies to Address the Dilemma
1. Greedy 𝜖
It contains a hyperparameter 𝜖 which can have any values between 0 to 1. When the 𝜖=0 the agent is in exploitative mode and when 𝜖=1 the agent is in explorative mode. That is the value of 𝜖 is the probability of agent performing random action instead of action suggested by policy. Agent performing random action is explorative. Therefore, if 𝜖=0.4 then agent explores 40% and exploits 60%. This is formalised as:
2. Greedy 𝜖 simulated with annealing.
It is a variation of Greedy 𝜖 where 𝜖 starts at 1 and decays gradually until it reaches 0 or close to 0. This is because when the agent first starts learning, it has no knowledge of the environment and thus must explore a lot. But over time, as the agent gains familiarity with environment, it should switch to exploit so it can start using the knowledge it gained during exploration.
Conclusion
The exploration-exploitation dilemma is central to the design of effective reinforcement learning algorithms. Balancing these two aspects is crucial as it directly impacts the learning speed and the quality of the policy developed by the agent. Strategies like epsilon-greedy and its variations help manage this balance, enabling the agent to both discover new potential rewards and capitalize on known opportunities.
Comments