Use of a relational reinforcement learning algorithm to generate dynamic activity-travel patterns

Abstract

In the course of the past decade activity-based models have entered the area of transportation modelling. Such models simulate the generation of individual activity-travel patterns while deciding simultaneously on the different dimensions of activity-travel behaviour, such as the type of activity, the activity location, the transport mode used to reach this location, the starting time and duration of the activity, etc. However, as real-world activity-travel patterns prove not to be static due to short-term adaptation and long-term learning, the scheduling algorithm needs to be adapted in order to be able to account for these dynamics. Short-term adaptation refers to within-day rescheduling as the result of the occurrence of unexpected events in the course of the execution of individual planned activity programmes, for instance congestion, or unexpected changes in the duration of an activity. Long-term learning denotes the change in activity-travel behaviour caused by the occurrence of key events, such as residential relocation and obtaining one’s driving license. In order to capture these dynamics, the current research will implement a technique originating from the area of artificial intelligence, in particular on a reinforcement learning technique extended with inductive learning. This method will be based on Q-learning in which the estimation of the traditional Q-function has been substituted by inductive learning. The approach aims at generalizing the (state, action, Q-value)-triplet by the induction of a regression tree. The major advantage of this technique consists of the fact that the Q-function no longer needs to be represented by means of a reward table which grows exponentially as the number of (state, action)-pairs rises due to both an increase in the number of exploratory and/or explanatory variables and an increase in the granularity of those variables. This technique will be examined for its applicability to the current research area. To start with, the key component of the reinforcement learning algorithm, the reward function, will be elaborated. This function will be founded on the starting time and weekday, the activity type, the activity duration, the waiting time and the activity history in order to reflect the underlying needs of the agents. Subsequently the parameters of the algorithm will be tuned. Afterwards the improved reinforcement learning technique will be implemented to simulate observed activity sequences of sixteen full-time working individuals being part of a four-headed household. The results will be explored, showing that the agent is able to determine autonomously activity sequences and to take into account temporal constraints and limits with respect to rather fixed activities, including work and night’s sleep. To end with, the generated activity schedules will be validated. The simulated activity patterns will be compared to actual, revealed patterns by means of the distance method SAM (Sequence Alignment Method). This method calculates the distance between the generated and the actual activity schedule, reflecting as such the (dis)similarity of these patterns.