Developing agents that can execute multiple skills by learning from
pre-collected datasets is an important problem in robotics, where online
interaction with the environment is extremely time-consuming. Moreover,
manually designing reward functions for every single desired skill is
prohibitive. Prior works targeted these challenges by learning goal-conditioned
policies from offline datasets without manually specified rewards, through
hindsight relabeling. These methods suffer from the issue of sparsity of
rewards, and fail at long-horizon tasks. In this work, we propose a novel
self-supervised learning phase on the pre-collected dataset to understand the
structure and the dynamics of the model, and shape a dense reward function for
learning policies offline. We evaluate our method on three continuous control
tasks, and show that our model significantly outperforms existing approaches,
especially on tasks that involve long-term planning.