Offline model-based reinforcement learning with causal structured world models

Abstract

Model-based methods have recently been shown promising for offline reinforcement learning (RL), which aims at learning good policies from historical data without interacting with the environment. Previous model-based offline RL methods employ a straightforward prediction method that maps the states and actions directly to the next-step states. However, such a prediction method tends to capture spurious relations caused by the sampling policy preference behind the offline data. It is sensible that the environment model should focus on causal influences, which can facilitate learning an effective policy that can generalize well to unseen states. In this paper, we first provide theoretical results that causal environment models can outperform plain environment models in offline RL by incorporating the causal structure into the generalization error bound. We also propose a practical algorithm, oFfline mOdel-based reinforcement learning with CaUsal Structured World Models (FOCUS), to illustrate the feasibility of learning and leveraging causal structure in offline RL. Experimental results on two benchmarks show that FOCUS reconstructs the underlying causal structure accurately and robustly, and, as a result, outperforms both model-based offline RL algorithms and causal model-based offline RL algorithms.

Graphical abstract

Keywords

reinforcement learning / offline reinforcement learning / model-based reinforcement learning / causal discovery

1 Introduction

Offline Reinforcement Learning (RL) is a learning paradigm where policies are learned entirely from previously collected data. Offline RL is gaining popularity since it enables RL algorithms to scale to several real-world applications, e.g., autonomous driving [1] and healthcare [2], where trial-and-error is too expensive. In the offline setting, Model-Based Reinforcement Learning (MBRL) is a popular framework that involves learning a predictive environment model for policy optimization [3]. The effectiveness of this approach relies on the accuracy of the learned environment model.

However, current offline MBRL approaches usually have poor generalization because the environment models tend to capture spurious correlations that only exist in offline data, resulting in erroneous predictions. For instance, in autonomous driving, if offline data is acquired from a driver who always turns on the wiper and brake pedals on rainy days, such a preference will result in a spurious correlation between “the wiper is turned on” and “the speed is dropped” in the data, which will also be captured by the environment model. Once we employ this environment model for policy learning, the agent will likely urge the driver to turn on the wiper when the vehicle’s speed is too high because it believes that “the wiper is turned on” has an effect on “the speed is dropped”, which is not sensible and potentially hazardous. Intuitively, leveraging the causal structure of observed variables can avoid taking spurious correlations as causal influences and thus facilitate the learning of an environment model with enhanced generalizability. Recent empirical evidence also indicates that inducing the causal structure is important to improve the generalization ability of deep learning models [4-7]. Despite such evidence, it is still unknown whether and how the causal structure improves model generalization in offline RL.

For this purpose, we first provide theoretical support for the aforementioned intuition: we show that a causal environment model can outperform a plain environment model on generalization for offline RL. From the causal perspective, we categorize the variables in states and actions into two groups: causal variables and spurious variables. We then formalize the process of learning an environment model with both types of variables in states. On the basis of the formalization, we quantify the effect of spurious dependencies on the generalization error bound and thereby demonstrate that integrating causal structures can assist in minimizing this bound.

We also propose a practical offline causal MBRL algorithm, FOCUS, to illustrate the feasibility of learning causal structure in offline RL. Learning the causal structure from offline data, also known as causal discovery from observations [8], is a crucial phase of FOCUS. However, causal discovery from observations requires a huge number of hypothesis testing, which is computation-consuming. To tackle this problem, we utilize the time-series property in RL data to reduce the number of hypothesis testing. Specifically, we incorporate the constraint that the future cannot cause the past in the PC algorithm [8], which seeks to uncover causal relationships based on inferred conditional independence relations. Consequently, we can reduce the number of conditional independence tests and determine the causal direction. In addition, we employ kernel-based conditional independence tests [9], which can be applied to continuous variables without assuming a specific functional form between the variables or a particular data distribution.

In conclusion, this paper makes the following key contributions.

● It theoretically demonstrates that a causal environment model outperforms a plain environment model in offline RL with respect to the generalization error bound.

● It proposes a practical algorithm, FOCUS, and illustrates the feasibility of learning and employing a causal environment model for offline MBRL.

● Our experimental results validate the theoretical claims, showing that FOCUS outperforms baseline models and other existing causal MBRL algorithms in the offline setting.

2 Related work

The RL algorithms with causal structure learning can be roughly divided by the type of their causal discovery methods. First, we will discuss relevant causal discovery methods, followed by related RL algorithms.

2.1 Causal discovery methods

Causal discovery methods can be categorized into two main types: intervention-based methods and observation-based methods. Intervention-based methods determine the causal relationship between variables by intervening on them and obtaining a counterfactual outcome different from what is currently observed. By contrast, observation-based methods rely solely on existing data to discover causal relationships without performing any intervention. Since interventions are not feasible in the offline reinforcement learning environment discussed in this paper, we will primarily focus on observation-based causal discovery methods.

Observation-based causal discovery methods can generally be divided into two categories: constraint-based methods and score-based methods. Constraint-based methods use statistic tests (conditional independent tests) to find the causal skeleton and determine the causal directions up to the Markov equivalence class. For example, [10] is based on explicit estimation of the conditional densities or their variants, which exploit the difference between the characteristic functions of these conditional densities. The estimation of the conditional densities or related quantities is difficult, which deteriorates the testing performance especially when the conditioning set is not small enough. Score-based methods evaluate the quality of candidate causal models with some score functions and output one or multiple graphs having the optimal score [11]. [12] discretizes the conditioning set to a set of bins and transforms conditional independence (CI) to the unconditional one in each bin. Inevitably, due to the curse of dimensionality, as the conditioning set becomes larger, the required sample size increases dramatically.

2.2 Causal discovery in reinforcement learning

Causal discovery methods can be utilized in policy learning and model learning in RL.

In policy learning, [7] proposes a causal imitation learning algorithm, which learns the causal structure between states and actions. However, it assumes that we can query experts for actions and use interventioned data to do causal discovery, which is impractical in offline RL.

In world model learning for RL, various methods have been proposed but can hardly be used in offline RL. For example, [13] (LNCM) views data sampled from different policies as data with soft interventions and use score-based methods with the log-likelihood on “interventional” data as the score function. Its implicit assumption that data is sampled from multiple policies and data has been labeled by its sampling policy is not a general assumption in offline RL, which only holds true in online RL. [14] samples causal structures from Multivariate Bernoulli distribution and scores those structures according to the log-likelihood on interventional data. Based on the scores, it calculates the gradients for the parameters of the Multivariate Bernoulli distribution and updates the parameters iteratively. [6] utilizes the speed of adaptation to learn the causal direction, but it does not provide a complete algorithm for learning causal structure. Given the properties of offline RL that the sampling policy has unknown preferences and interactions with the environment are not available, the above methods are not practical for learning the causal structure in offline RL.

3 Preliminaries

3.1 Markov decision process (MDP)

We describe the RL environment as an MDP with five-tuple $⟨\mathcal{S},\mathcal{A},P,R,\gamma ⟩$ [15], where $\mathcal{S}$ is a finite set of states; $\mathcal{A}$ is a finite set of actions; $P$ is the transition function with $P({\mathbf{s}}^{\prime }|\mathbf{s},\mathbf{a})$ denoting the next-state distribution after taking action $\mathbf{a}$ in state $\mathbf{s}$; $R$ is a reward function with $R(\mathbf{s},\mathbf{a})$ denoting the expected immediate reward gained by taking action a in state s; and $\gamma \in [0,1]$ is a discount factor. An agent chooses actions $\mathbf{a}$ according to a policy $\mathbf{a}\sim \pi (\mathbf{s})$, which updates the system state ${\mathbf{s}}^{\prime }\sim P(\mathbf{s},\mathbf{a})$, yielding a reward $r\sim R(\mathbf{s},\mathbf{a})$. The agent’s goal is to maximize the expected cumulative return by learning a good policy $\underset{\pi ,P}{max}\mathbb{E}[\sum _{t=1}^T{\gamma }^{t}R({\mathbf{s}}_t,{\mathbf{a}}_t)]$. The state-action value $Q_{\pi }$ of a policy $\pi$ is the expected discounted reward of executing action $a$ from state $\mathbf{s}$ and subsequently following policy $\pi$: $Q_{\pi }(\mathbf{s},\mathbf{a}):=R(\mathbf{s},\mathbf{a})+\gamma {\mathbb{E}}_{{\mathbf{s}}^{\prime }\sim P,{\mathbf{a}}^{\prime }\sim \pi }\left[Q_{\pi }\left({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime }\right)\right]$.

3.2 Offline model-based reinforcement learning

In the offline RL setting, the algorithm only has access to a static dataset $\mathcal{D}=\{(s,a,r,s^{\prime })\}$ collected by one or a mixture of behavior policies ${\pi }_B$, and further interactions with the environment is not available. When we use model-based approaches to solve offline RL problems, we will learn a virtual environment model $\hat{P}$ for transition prediction from offline data. With the learned environment model $\hat{P}$, we can define a new MDP $⟨\mathcal{S},\mathcal{A},\hat{P},R,\gamma ⟩$. Similarly, we can also define the value function ${\hat{Q}}_{\pi }$ with $\hat{P}$. A standard model-based RL algorithm (in an online setting) learns a virtual model by fitting it using a maximum-likelihood estimate of the trajectory-based data collected by running the latest policy, which guarantees that the virtual model can always be accurate when the policy keeps exploring [16,17]. In the offline RL setting, where we only have access to the data collected by previous policies, the accuracy of the virtual model in exploring policy cannot be guaranteed. Therefore recent techniques all build on the idea of pessimism that regularizes the original problem based on how confident the agent is about the learned model [3,18]. Specifically, the policy only visits the states where the learned model is confident in predictions.

4 Theoretical advantages of causal world models

In this section, we present theoretical evidence demonstrating the superior performance of a causal environment model over a traditional environment model in the context of offline RL. This underscores the importance of leveraging an appropriate causal structure to minimize the generalization error bounds for offline MBRL algorithms. More specifically, we formalize a process that discerns the environment model burdened with spurious correlations, and subsequently, quantifies their influence. We then evaluate the effects of these spurious correlations in relation to the generalization ability. These evaluation metrics encompass the model prediction error bound and the policy evaluation error bound within the offline RL framework.

For the purpose of simplicity, we operate under the assumption that the causal structure is accurately known. As a result, our theoretical findings do not account for errors that could potentially arise from inaccurate structures. We further assume that the causal relationships are linear and that all causal variables are fully observed. For a comprehensive understanding of our theoretical proof, details are detached in Appendixes A1.

4.1 Problem formalization

Plain world models. In the context of this paper, we use the term “plain world models” to refer to the traditional forward dynamics model, $P_{\theta }({\mathbf{s}}_t|{\mathbf{s}}_{t-1},a_{t-1})$, which ignores the underlying causal structure between the current state-action pair and the future state:

$\begin{array}{r}{P}_{\theta }({\mathbf{s}}_t|{\mathbf{s}}_{t-1},a_{t-1})={\mathrm{\Pi }}_{i=1}^{n}P({\mathbf{s}}_{t,i}|{\mathbf{s}}_{t-1},a_{t-1}),\end{array}$

where ${\mathbf{s}}_{t,i}$ denotes the $i\mathrm{t}\mathrm{h}$ dimension of the state ${\mathbf{s}}_t$.

Causal world models. By contrast, we use the term “causal world models” to refer to the forward dynamics model with the causal structure:

$\begin{array}{r}{P}_{causal}({\mathbf{s}}_t|{\mathbf{s}}_{t-1},a_{t-1})={\mathrm{\Pi }}_{i=1}^{n}P({\mathbf{s}}_{t,i}|{\cup }_{j\in P{a}_i}{\mathbf{s}}_{t-1,j},a_{t-1}),\end{array}$

where $P{a}_i$ denotes the dimensions of state ${\mathbf{s}}_{t-1}$ that are causal parents of variable ${\mathbf{s}}_{t,i}$ and ${\cup }_{j\in P{a}_i}{\mathbf{s}}_{t-1,j}$ denotes the variable set that contains the causal parent of variable ${\mathbf{s}}_{t,i}$.

Model learning. Model learning in MBRL aims to predict the future state given the current state and action, which can be viewed as a supervised learning problem. For simplicity, we formula it as supervised learning, i.e., use $\mathbf{X}$ to represent $({\mathbf{s}}_{t-1},a_{t-1})$ and $Y$ to represent one dimension of ${\mathbf{s}}_t$.

Let $\mathcal{D}$ denotes the data distribution where we have samples $(\mathbf{X},Y)\sim \mathcal{D},\mathbf{X}\in {\mathbb{R}}^n$. The goal of model learning is equivalent to learning a linear function $f$ to predict $Y$ given $\mathbf{X}$. From the causal perspective, $Y$ is generated from only its causal parent variables rather than all the variables in $\mathbf{X}$. Therefore we can category the variables in $\mathbf{X}$ into two groups, $\mathbf{X}=$ $({\mathbf{X}}_{causal},{\mathbf{X}}_{spurious})$.

● ${\mathbf{X}}_{causal}$ represents the causal parent variables of $Y$, that is, $Y={\mathbf{X}}_{causal}{\beta }^{\ast }+{ϵ}_{causal}$, where ${\beta }^{\ast }$ is the ground truth and ${ϵ}_{causal}$ is a noise with zero mean and independent to ${\mathbf{X}}_{causal}$, i.e., ${\mathbf{X}}_{causal}\perp \perp {ϵ}_{causal}$.

● ${\mathbf{X}}_{spurious}$ represents the spurious variables that are independent to $Y$, i.e., ${\mathbf{X}}_{spurious}\perp \perp Y$, but ${\mathbf{X}}_{spurious}$ and ${\mathbf{X}}_{causal}$ have strong relatedness in biased offline data. In other words, ${\mathbf{X}}_{spurious}$ can be predicted by ${\mathbf{X}}_{causal}$ with small error, i.e., ${\mathbf{X}}_{spurious}={\mathbf{X}}_{causal}{\gamma }_{spurious}+{ϵ}_{spurious}$, where ${ϵ}_{spurious}$ is the regression error with zero mean and small variance ${\sigma }_{spu}$.

For clearly representation, we use ${\mathbf{X}}_{cau}\triangleq \mathbf{X}\circ {\omega }_{cau}$ ($\circ$ represents element-wise multiplication) to replace ${\mathbf{X}}_{causal}$, where $cau$ records the indices of ${\mathbf{X}}_{causal}$ in $\mathbf{X}$ and $({\omega }_{cau}{)}_i=\mathbb{I}(i\in cau)$. Correspondingly, we also use ${\mathbf{X}}_{spu}\triangleq \mathbf{X}\circ {\omega }_{spu}$ to replace ${\mathbf{X}}_{spurious}$.

Example: Suppose that $\mathbf{X}=[x_1,x_2,x_3]$ and $(x_1,x_3)$ are causal parent, we have ${\omega }_{cau}=[1,0,1]$ and ${\omega }_{spu}=[0,1,0]$, and then ${\mathbf{X}}_{cau}=[x_1,0,x_3]$, ${\mathbf{X}}_{spu}=[0,x_2,0]$.

According to the definition of ${\mathbf{X}}_{cau}$, we have $Y=(\mathbf{X}\circ {\omega }_{cau}){\beta }^{\ast }+{ϵ}_{cau}$, where ${\omega }_{cau}\circ {\beta }^{\ast }$ is the global optimal solution of the optimization problem:

Definition 1 (Optimization problem 1).

(1)$\begin{array}{r}\underset{\beta }{min}{\mathbb{E}}_{(\mathbf{X},Y)\sim \mathcal{D}}[\mathit{X}\beta -Y{]}^2.\end{array}$

However, in the offline setting, we only have biased data ${\mathcal{D}}_{train}$ sampled by given policy ${\pi }_{train}$, where the optimization problem becomes:

Definition 2 (Optimization problem 2).

(2)$\begin{array}{r}\underset{\beta }{min}{\mathbb{E}}_{(\mathbf{X},Y)\sim {\mathcal{D}}_{train}}[\mathit{X}\beta -Y{]}^2.\end{array}$

4.2 Model prediction error bound

In this subsection, we assume a causal structure of the RL environment and spurious relations in offline data. We highlight that these spurious relationships can contribute to the formation of an ill-posed problem within the model learning process, hence increasing the model prediction error bound. Specifically, we quantitatively measure the impact of spurious relations on model learning as shown in Theorem 1. This theorem provides the relations between the properties of spurious relations and the model prediction error bound.

4.2.1 The spurious theorem

In Lemma 1, we prove that the optimization problem 2 has multiple optimal solutions due to the strong linear relatedness of ${\mathbf{X}}_{spu}$ and ${\mathbf{X}}_{cau}$ in ${\mathcal{D}}_{train}$.

Lemma 1 Given that ${\omega }_{cau}\circ {\beta }^{\ast }$ is the optimal solution of optimization problem 1, suppose that in $D_{train}$, ${\mathit{X}}_{spu}=$ $(\mathit{X}\circ {\omega }_{cau}){\gamma }_{spu}+{ϵ}_{spu}$ where ${\mathbb{E}}_{D_{train}}[{ϵ}_{spu}]=0$ and ${\gamma }_{spu}\ne \mathbf{0}$, we have that ${\hat{\beta }}_{spu}\triangleq {\omega }_{cau}\circ ({\beta }^{\ast }-\lambda {\gamma }_{spu})+\lambda {\omega }_{spu}$ is also an optimal solution of optimization problem 2 for any $\lambda$:

$\begin{array}{rl}& {\mathbb{E}}_{(\mathit{X},Y)\sim D_{train}}[(|\mathit{X}({\omega }_{cau}\circ {\beta }^{\ast })-Y{|}_2)\mid \mathit{X}]\\ =& {\mathbb{E}}_{(\mathit{X},Y)\sim D_{train}}[(|\mathit{X}{\hat{\beta }}_{spu}-Y{|}_2)\mid \mathit{X}].\end{array}$

The most popular method for solving such an ill-posed problem is to add a regularization term for parameters $\beta$ [19]:

Definition 3 (Optimization problem 3).

(3)$\begin{array}{r}\underset{\beta }{min}{\mathbb{E}}_{(\mathit{X},Y)\sim {\mathcal{D}}_{train}}[\mathit{X}\beta -Y{]}^2+k\Vert \beta {\Vert }^2,\end{array}$

where $k$ is a coefficient.

The form of optimization problem 3 corresponds to the form of the ridge regression, which provides a closed-form solution of $k$ by Hoerl-Kennard formula [20]. Given the closed-form solution of $k$, we can derive the optimal solution of optimization problem 3, which corresponds a solution of $\lambda$.

In the following, we will first introduce the solution of $\lambda$ with given $k$ in Lemma 2, and then introduce the model prediction error bound with $\lambda$ in Theorem 1. For ease of understanding, we provide a simple version where the dimensions of ${\mathbf{X}}_{cau}$ and ${\mathbf{X}}_{spu}$ are both one ($|{\mathbf{X}}_{cau}|=$ $|{\mathbf{X}}_{spu}|=1$).

Lemma 2 ($\lambda$ Lemma). Let ${\sigma }_{cau}^2$ denote the variance of the noise ${ϵ}_{cau}$ and ${\sigma }_{spu}^2$ denote the variance of ${ϵ}_{spu}$, given $k$ in optimization problem 3 chosen by Hoerl-Kennard formula that $k=\frac{{\sigma }_{spu}^2}{({\beta }^{\ast }{)}^2}$, we have the solution of $\lambda$ in optimization problem 3 that:

(4)$\begin{array}{rl}\lambda & =\frac{{\sigma }_{cau}^2{\beta }^{\ast }{\gamma }_{spu}k}{{\sigma }_{cau}^2{\sigma }_{spu}^2+{\sigma }_{cau}^2{\gamma }_{spu}^{2}k+{\sigma }_{cau}^{2}k+{\sigma }_{spu}^{2}k+k^2}\\ & =\frac{{\beta }^{\ast }{\gamma }_{spu}}{{{\beta }^{\ast }}^2+{\gamma }_{spu}^2+1+\frac{{\sigma }_{spu}^2}{{\sigma }_{cau}^2}(1+\frac{1}{({\beta }^{\ast }{)}^2})}.\end{array}$

Based on Lemma 2, we can find that the smaller ${\sigma }_{spu}^2$ (it means that ${\mathbf{X}}_{spu}$ and ${\mathbf{X}}_{cau}$ have stronger relatedness in the training dataset $D_{train}$), the larger $\lambda$. And we prove that the coefficient $\lambda$ is bounded:

Proposition 1 Given $\lambda$ as in Lemma 2, we have its bound that $-\frac{1}{2}\le \lambda \le \frac{1}{2}.$

Theorem 1 (Spurious theorem). Let $\mathcal{D}=\{(\mathit{X},Y)\}$ denote the data distribution, ${\hat{\beta }}_{spu}$ denote the solution in Lemma 1 with $\lambda$ in Lemma 2, and ${\hat{Y}}_{spu}=\mathit{X}{\hat{\beta }}_{spu}$ denote the prediction. Suppose that the data value is bounded: $|X_i{|}_1\le X_{max},i=1,...,n$ and the error of optimal solution ${ϵ}_{cau}$ is also bounded: $|{ϵ}_{cau}{|}_1\le {ϵ}_c$, we have the model prediction error bound:

(5)$\begin{array}{r}{\mathbb{E}}_{(\mathbf{X},Y)\sim D}[(|{\hat{Y}}_{spu}-Y{|}_1)\mid \mathit{X}]\le X_{max}|\lambda {|}_1(|{\gamma }_{spu}{|}_1+1)+{ϵ}_c.\end{array}$

Theorem 1 shows that

● the upper bound of the model prediction error $|{\hat{Y}}_{spu}-Y{|}_1$ increases by $X_{max}|\lambda {|}_1(|{\gamma }_{spu}{|}_1+1)$ for each induced spurious variable $X_{spu}$ in the model,

● when $X_{spu}$ and $X_{cau}$ have stronger relatedness (which means a bigger $\lambda$), the increment of the prediction model error bound led by $X_{spu}$ is bigger.

4.3 Policy evaluation error bound

Building on the model prediction error bound established earlier, we proceed to delineate the policy evaluation error bound, a metric that offers a more direct evaluation of the world model in MBRL. In this subsection, we apply the spurious theorem (Theorem 1) to offline MBRL. Our investigation explores how the policy evaluation error bound escalates in response to an increase in the number of spurious variables and the strengthening of spurious relationships.

Assuming that the state value is bounded as well as the reward that $|S_{t,i}{|}_1\le S_{max},R_t\le R_{max}$, and denoting the maximum of $\lambda$ as ${\lambda }_{max}$ and the maximum of $|{\gamma }_{spu}{|}_1$ as ${\gamma }_{max}$, we establish the policy evaluation error bound in Theorem 2.

Theorem 2. (RL spurious theorem) Given an MDP with the state dimension $n_s$ and the action dimension $n_a$, a data-collecting policy ${\pi }_D$, let $M^{\ast }$ denote the true transition model, $M_{\theta }$ denote the learned model that $M_{\theta }^i$ predicts the $i\mathrm{t}\mathrm{h}$ dimension with spurious variable sets $sp{u}_i$ and causal variables $ca{u}_i$, i.e., ${\hat{S}}_{t+1,i}=M_{\theta }^i(({\mathbf{S}}_t,{\mathbf{A}}_t)\circ {\omega }_{ca{u}_i\cup sp{u}_i})$. Let $V_{\pi }^{M_{\theta }}$ denote the policy value of the policy $\pi$ in model $M_{\theta }$ and correspondingly $V_{\pi }^{M^{\ast }}$. For an arbitrary bounded divergence policy $\pi$, i.e. $\underset{S}{max}{D}_{KL}(\pi (\cdot |S),{\pi }_D(\cdot |S))\le {ϵ}_{\pi }$, we have the policy evaluation error bound:

(6)$\begin{array}{rl}|V_{\pi }^{M_{\theta }}-V_{\pi }^{M^{\ast }}|\le & \frac{2\sqrt{2}{R}_{max}}{(1-\gamma {)}^2}\sqrt{{ϵ}_{\pi }}+\frac{R_{max}\gamma }{2(1-\gamma {)}^2}{S}_{max}[n_s{ϵ}_c+\\ & (1+{\gamma }_{max}){\lambda }_{max}{n}_s(n_s+n_a)R_{spu}],\end{array}$

where

$\begin{array}{r}{R}_{spu}=\frac{\sum _{i=1}^{n_s}|sp{u}_i|}{n_s(n_s+n_a)},\end{array}$

which represents the spurious variable density, that is, the ratio of spurious variables in all input variables.

Theorem 2 shows the relation between the policy evaluation error bound and the spurious variable density, which indicates that:

● When we use a non-causal model that all the spurious variables are input, $R_{spu}$ reaches its maximum value ${\overline{R}}_{spu}<1$. By contrast, in the optimal causal structure, $R_{spu}$ reaches its minimum value of $0$.

● The density of spurious variables $R_{spu}$ and the correlation strength of spurious variables ${\lambda }_{max}$ both influence the policy evaluation error bound. However, if we exclude all the spurious variables, i.e., $R_{spu}=0$, the correlation strength of spurious variables will have no effect.

5 FOCUS algorithm

Having established the necessity of a causal environment model in offline RL, we proceed in this section to propose a practical offline MBRL algorithm, FOCUS, to illustrate the feasibility of learning and using causal structure in offline RL. Our algorithm consists of two distinct steps: the discovery of the causal structure from offline data, followed by the effective integration of the identified structure with an offline MBRL algorithm. During the first step, the offline setting imposes limitations on the selection of causal discovery methods. Specifically, methods that involve interventions or randomized experiments are deemed unsuitable as interactions with the environment are prohibited. When considering approaches for observational data, score-based methods necessitate data from distinct sampling policies and a known form of causal mechanisms, neither of which are present in offline data collected from a single policy. Constraint-based methods, on the other hand, do not presuppose any specific form of causal mechanisms, but they fail to distinguish structures within one Markov equivalence class and may prove inefficient due to numerous independence tests. To address these challenges, FOCUS extends the PC algorithm (a derivative of constraint-based methods) and rectifies its shortcomings by incorporating sequential information. In the second step, FOCUS initializes the environment model with the learned causal structure and then proceeds to learn the environment model along with the policy.

5.1 Preliminary

5.1.1 Assumptions

First, we assume the causal Markov condition and faithfulness in the environment transition, with which we can use conditional independence tests to infer the causal graph [8]. Second, we claim that the offline data and the data obtained through the learned policy share the same causal structure, through which the learned structure can be applied in unseen states. Specifically, policy preference affects the relations between variables by causing quantitative changes in causal relations and spurious relations in independent relations. Thanks to the ability to distinguish between spurious correlations and causal influences, the policy preference will not result in qualitative changes in the causal structure. Consequently, the above statement holds true in offline RL.

5.1.2 Conditional independence test

Independence and conditional independence (CI) plays a central role in causal discovery [8,21,22]. Generally speaking, the CI relationship $X\perp \perp Y\mid Z$ allows us to drop $Y$ when constructing a probabilistic model for $X$ with $(Y,Z)$. There are multiple CI testing methods for various conditions, which provide the correct conclusion only given the corresponding condition. The kernel-based Conditional Independence test (KCI test) [9] is proposed for continuous variables without assuming a specific functional form between the variables as well as the data distributions.

5.1.3 Conditional variables

Besides the specific CI test method, the conclusion of conditional independence testing also depends on the conditional variable $Z$, that is, different conditional variables can lead to different conclusions. Taking the triple $(X,Y,Z)$ as an example, there are three typical structures, namely, Chain, Fork, and Collider as shown in Fig.1. Chain: There exists causation between $X$ and $Y$ but conditioning on $Z$ leads to independence. Fork: There does not exist causation between $X$ and $Y$ but not conditioning on $Z$ leads to non-independence. Collider: There does not exist causation between $X$ and $Y$ but conditioning on $Z$ leads to non-independence.

5.2 Causal structure learning

5.2.1 Applying the Independence test in RL

Based on the preliminaries, given the two target variables $X,Y$ and the condition variable $Z$, the KCI test returns a probability value $p=f_{KCI}(X,Y,Z)\in [0,1]$, which measures the probability that $X$ and $Y$ are conditionally independent given the condition $Z$. To transform a probability into a binary conclusion of whether the causation exists, we design a threshold $p^{\ast }$ that

$\begin{array}{r}Causation(X,Y)=\mathbb{I}(f_{KCI}(X,Y,Z)\le p^{\ast })\in \{0,1\},\end{array}$

where $Causation(X,Y)=1$ represents independence and $0$ represents that causation exists. Details of choosing $p^{\ast }$ can be found in Appendixes A2.

In model learning of RL, variables are composed of states and actions of the current and next timesteps and the causal structure refers to whether a variable in $t$ timestep (e.g., the $i\mathrm{t}\mathrm{h}$ dimension, $X_t^i$) causes another variable in $t+1$ timestep (e.g., the $j\mathrm{t}\mathrm{h}$ dimension, $X_{t+1}^j$). With the KCI test, we get the causal relation through the function $Causation(\cdot ,\cdot )$ for each variable pair $(X_t^i,X_{t+1}^j)$ and then form the causal structure matrix $\mathcal{G}$:

$\begin{array}{r}{\mathcal{G}}_{i,j}=Causation(X_t^i,X_{t+1}^j),\end{array}$

where ${\mathcal{G}}_{i,j}$ is the element in row $i$ and column $j$ of $\mathcal{G}$.

5.2.2 Choosing the conditional variable in RL

As stated in preliminaries, unsuitable conditional variables can reverse the conclusion of independence testing. The conditional variable set must include the intermediate variable of Chain and the common parent variable of Fork, but not the common son variable of Collider. Traditionally, the independence test traverses all possible combinations of the conditional variables and then reaches a conclusion, which is inefficient. However, in RL we can reduce the number of conditional independence tests by imposing the restriction that the future cannot cause the past. Actually, this constraint restricts the possible conditional variable sets to a tiny number. Consequently, we can have a classified discussion for every feasible collection of conditional variables. For simplicity, we eliminate two types of scenarios from the discussion:

Fig.1 The three basic structures for $(X,Y,Z)$

Full size|PPT slide

● Impossible situations. We exclude some impossible situations as Fig.2(i) (bottom left) by the temporal property of data in RL. Specifically, the direction of the causation cannot be from the variable of $t+1$ time step to that of $t$ time step because the effect cannot happen before the cause.

● Compound situations. We only discuss the basic situations and exclude the compound situations, e.g., Fig.2(j) (bottom right), which is a compound of Figs. 3(a) and 3(c). In such compound situations, the target variables $X_t^i$ and $X_{t+1}^j$ have direct causation (or it can not be a compound situation). When a causal relationship exists, it is impossible to arrive at a conclusion of “independence” in an independence test. Hence, we do not need to be concerned about coming to the conclusion that a causal relationship does not exist, thereby mistakenly excluding causal parent variables.

As seen in Fig.2, we list all conceivable circumstances involving target variables $X_t^i,X_{t+1}^j$ and condition variable $X_{t/t+1}^k$ in the environment model. With the preliminary knowledge of causal discovery, we investigate the following fundamental situations:

Top line: In Figs. 3(a) and 3(b), whether $X_t^k$ is included in the conditional variable set does not affect the conclusion of causation; In Fig. 3(c), although $X_t^k$ is an intermediate variable in a Chain and conditioning on $X_t^k$ leads to the conclusion of independence of $X_t^i$ and $X_{t+1}^j$, the causal parent set of $X_{t+1}^j$ will include $X_t^k$ when testing the causal relationship between $X_t^k$ and $X_{t+1}j$, which can offset the influence of excluding $X_t^i$. In Fig. 3(d), conditioning on $Z$ is necessary for getting the correct conclusion of causation since $X_t^k$ is the common causal parent in a $Fork$ structure.

Bottom line: In Figs. 3(e) and 3(f), whether $X_{t+1}^k$ is included in the conditional variable set does not affect the conclusion of causation; In Fig. 3(g), not conditioning on $X_{t+1}^k$ is necessary to get the correct conclusion of causation since $X_{t+1}^k$ is the common son in a $Collider$ structure; In Fig. 3(h), although $X_{t+1}^k$ is an intermediate variable in a Chain and not conditioning on $X_{t+1}^k$ leads to the conclusion of non-independence of $X_t^i$ and $X_{t+1}^j$, including $X_t^i$ in the causal parent set of $X_{t+1}^j$ will not induce any problem since $X_t^i$ does indirectly cause $X_{t+1}^j$. Based on the classified discussion above, we can conclude our principle for choosing conditional variables in RL:

Fig.2 The basic, impossible and compound situations of the causation between target variables and condition variables. In the basic situations, Top line: (a)−(d) list the situations that the condition variable $X^k$ is in the $t$ time step. Bottom line: Similarly, (e)−(h) list the situations that the condition variable $X^k$ is in the $t+1$ time step. Bottom left: (i) lists impossible situations where the direction of causation is from the future to the past. Bottom right: (j) lists compound situations where the causal structure is a compound of basic situations

Full size|PPT slide

Fig.3 The architecture of FOCUS. Given offline data, FOCUS learns a $p$ value matrix by KCI test and then gets the causal structure by choosing a $p$ threshold. After combining the learned causal structure with the neural network, FOCUS learns the policy through an offline MBRL algorithm

Full size|PPT slide

● Condition on the other variables in $t$ time step.

● Do not condition on the other variables in $t+1$ time step.

5.3 Combining learned causal structure with an offline MBRL algorithm

We combine the learned causal structure with an offline MBRL algorithm, MOPO [3], to create a causal offline MBRL algorithm as in Fig.3. The entire learning procedure can be found in Algorithms 1 and 2. Notice that our causal model learning method can be combined with any offline MBRL algorithm theoretically.

Full size|PPT slide

Full size|PPT slide

6 Experiments

In this study, we assess the performance of FOCUS on a variety of offline RL datasets to demonstrate its effectiveness. Specifically, we aim to show that (1) FOCUS enables the learning of a causal environment model in offline RL, and (2) a causal environment model surpasses a standard environment model and other related offline RL methods in policy performance.

To validate the first point, we examine the accuracy, efficiencyefficiency, and robustness of causal structure learning. In regard to the second point, we evaluate policy return and generalization ability within the context of offline MBRL.

6.1 Experimental settings

6.1.1 Baselines

We compare FOCUS with the sota offline MBRL algorithm, MOPO, MOReL, COMBO and other online RL algorithms that also learn causal structure.

● MOPO [3] is a popular and well-known offline MBRL algorithm that outperforms standard model-based RL algorithms and prior sota model-free offline RL algorithms on existing offline RL benchmarks. The central idea of MOPO is to artificially penalize rewards by the uncertainty of model predictions, hence avoiding erroneous predictions in unseen states. MOPO can be seen as the blank control with a plain environment model.

● MOReL [18] is an offline MBRL algorithm that learns a pessimistic MDP using the offline dataset and then a near-optimal policy in this P-MDP.

● COMBO [23] is also an offline algorithm, which conservatively estimates the value function by penalizing it in out-of-support states generated through model rollouts.

● LNCM (Learning Neural Causal Models from Unknown Interventions) [13] is an online algorithm, in which the causal structure learning method can be transformed to the offline setting with a simple adjustment. We take LNCM as an example to show that an online method cannot be directly converted into offline RL algorithms.

6.1.2 Environment

Toy Car Driving. Toy Car Driving is a typical RL environment where the agent can control its direction and velocity to finish various tasks including avoiding obstacles and navigating. In this paper, we use a 2D Toy Car Driving as the RL environment where the task of the car is to arrive at the destination (The visualization can be found in the Appendix). The state includes the direction $d$, the velocity $v$, the velocity on the $x$-axis $v_x$, the velocity on the $y$-axis $v_y$ and the position $(p_x,p_y)$. The action is the steering angle $a$. We design the underlying causal structure to better demonstrate how spurious relations appear and highlight their influence in model learning (The structure can be found in the Appendixes).

Example: As shown in Fig.4, when the velocity $v_{t-1}$ maintains stationary due to an imperfect sample policy, $(v_x{)}_t$ and $(v_y{)}_t$ have strong relatedness that $(v_x{)}_t^2+(v_y{)}_t^2=v_{t-1}^2$ and one can represent the other. Since we design that $(p_y{)}_{t+1}-(p_y{)}_t=(v_y{)}_t$, $(v_x{)}_t$ and $(p_y{)}_{t+1}-(p_y{)}_t$ also have strong relatedness, which leads to that $(v_x{)}_t$ becomes a spurious variable of $(p_y{)}_{t+1}$ given $(p_y{)}_t$, despite that $(v_x{)}_t$ is not the causal parent of $y_{t+1}$. By contrast, when the data is uniformly sampled with various velocities, this spuriousness will not exist.

Fig.4 The visualization of the example. The red dotted arrow presents that $(v_x{)}_t$ is a spurious variable for $(p_y{)}_{t+1}$

Full size|PPT slide

MuJoCo. The MuJoCo [24] is the most popular benchmark for evaluating performance in continuous controlling, where the variables of the state represent the positions, angles, and velocity of the agent. Each dimension in MuJoCo of the state has a specific meaning and is highly abstract, which provides the convenience of causal structure learning.

6.1.3 Offline data.

We prepare three offline data sets, Random, Medium, and Replay for the Car Driving and MuJoCo. The datasets of MuJoCo are from D4RL [25]. To provide an intuitive illustration of the data sets, we use heat maps to visualize those offline data sets in the Car Driving in Fig.5:

Fig.5 The heat map of the three offline data sets in the Car Driving. The coordinates represent two dimensions of the state. The brightness at each data point represents its probability of being visited and high brightness represents high probability. (a) Random; (b) Medium; (c) Medium-Replay

Full size|PPT slide

● In Random, data is collected by random sampling policies, and its distribution is similar to a multivariate Gaussian distribution.

● In Medium, data is collected by a fixed but not well-trained policy and its distribution is the least diverse.

● In Medium-Replay, data is a collection of trajectories sampled during training, which is the most diverse.

6.2 The validation for spurious theorems

Our theoretical analysis reveals that

● The model prediction error bound increases with increasing strength of spurious relations and increasing number of induced spurious variables.

● The policy evaluation error bound increases with increasing strength of spurious relations and increasing spurious variable density.

To validate our theoretical findings, we conduct experiments in multiple RL environments where latent causal structures are diverse, namely, the number of spurious variables, the spurious relatedness, and the spurious variable density are varying. The RL environment is built based on the Inverted Pendulum task of MuJoCo. In this environment, the spurious relatedness is changed by modifying the hyper-parameters, e.g., the fraction coefficient. The spurious variable density is changed by introducing external variables. The number of spurious variables is changed by introducing external variables while keeping the spurious variable density fixed. We estimate the model and policy prediction error bounds by taking the maximum of those errors from five repeated experiments.

The results in Fig.6 show that the model prediction error bound of the plain world model increases approximately linearly with the relatedness $\lambda$, the number and density of spurious variables. By contrast, the model prediction error bound of the causal world model does not noticeably change with the involved variables above, and it remains at a relatively low level, which supports our spurious theorem (Theorem 1). The results in Fig.7 show similar conclusions for the policy evaluation error bound, which validates our RL spurious theorem (Theorem 2).

Fig.6 Comparison of Causal World Models and Plain World Models to validate the spurious theorem. We evaluate the model prediction error on the relatedness, number and density of spurious variables in the offline dataset. (a) Comparison on relatedness λ; (b) comparison on spurious number; (c) comparison on spurious density

Full size|PPT slide

Fig.7 Comparison of Causal World Models and Plain World Models to validate the RL spurious theorem. We evaluate the policy evaluation error on the relatedness, number and density of spurious variables in the offline dataset. (a) Comparison on relatedness λ; (b) comparison on spurious number; (c) comparison on spurious density

Full size|PPT slide

6.3 Causal structure learning

We compare FOCUS with baselines on the causal structure learning with the indexes of the accuracy, efficiency, and robustness. The accuracy is evaluated by viewing the structure learning as a classification problem, where causation represents the positive example and independence represents the negative example. The efficiency is evaluated by measuring the samples for getting a stable structure. The robustness is evaluated by calculating the variance in multiple experiments. The results in Tab.1 show that FOCUS surpasses LNCM in accuracy, robustness, and efficiency in causal structure learning. Noticed that LNCM also has a low variance because it predicts the probability of existing causation between any variable pairs with around $50\mathrm{\%}$, which means that its robustness is meaningless.

Tab.1 The results on causal structure learning of our model and the baselines. Both the accuracy and the variance are calculated by five times experiments. FOCUS (-KCI) represents FOCUS with a linear independence test. FOCUS (-CONDITION) represents FOCUS with choosing all other variables as conditional variables

INDEX	FOCUS	LNCM	FOCUS(-KCI)	FOCUS(-CONDITION)
ACCURACY	0.993	0.52	0.62	0.65
ROBUSTNESS	0.001	0.025	0.173	0.212
EFFICIENCY(SAMPLES)	1×106	1×107	1×106	1×106

6.4 Policy learning

6.4.1 Policy return

We evaluate the performance of FOCUS and baselines in the two benchmarks on three typical offline data sets. The results in Tab.2 and Tab.3 show that FOCUS outperforms baselines by a significant margin in most data sets.

Tab.2 The comparison on converged policy return in the two benchmarks

ENV	CAR DRIVING				MUJOCO(INVERTED PENDULUM)
	RANDOM	MEDIUM	REPLAY		RANDOM	MEDIUM	REPLAY
FOCUS	$\mathbf{68.1}\pm \mathbf{20.9}$	$-58.9\pm 41.3$	$\mathbf{86.2}\pm \mathbf{18.2}$		$\mathbf{23.5}\pm \mathbf{17.9}$	$\mathbf{24.9}\pm \mathbf{14.1}$	$\mathbf{49.2}\pm \mathbf{19.0}$
MOPO	$-30.3\pm 49.9$	$-50.1\pm 34.2$	$46.2\pm 28.1$		$8.5\pm 6.2$	$2.5\pm 0.08$	$43.4\pm 7.7$
LNCM	$9.9\pm 42.5$	$\mathbf{-}\mathbf{5.4}\pm \mathbf{32.5}$	$11.4\pm 24.0$		$13.3\pm 0.9$	$3.1\pm 0.7$	$16.3\pm 6.4$

Tab.3 Results for D4RL datasets. For “ORIGIN” version, we take the results from their original papers. For “FOCUS” version, each number is the averaged score at the last iteration of training, averaged over 3 random seeds. We bold the highest score across all methods

DATASET TYPE	ENVIRONMENT	MOPO			MOReL			COMBO
		ORIGIN	FOCUS		ORIGIN	FOCUS		ORIGIN	FOCUS
	HALFCHEETAH	35	37		25	34		38	44
RANDOM	HOPPER	11	30		53	63		17	23
	WALKER2D	13	28		37	54		7	11
	HALFCHEETAH	42	49		42	61		54	60
MEDIUM	HOPPER	28	30		95	102		103	107
	WALKER2D	17	27		77	85		81	94
	HALFCHEETAH	53	58		40	47		55	60
MEDIUM REPLAY	HOPPER	67	71		93	90		89	85
	WALKER2D	39	44		49	54		56	63

As shown in Tab.2, on Random datasets, FOCUS has the most significant performance gains to the baselines in both benchmarks because of the accuracy of causal structure learning in FOCUS. By contrast, on Medium-Replay, the performance gains of FOCUS are least since the high data diversity of Medium-Replay leads to weak relatedness of spurious variables (corresponds to small $\lambda$), which verifies our theory. On Medium, the results in the two benchmarks are different. In Car Driving, the relatively high score of LNCM does not mean that LNCM is the best but all three fail. The failure indicates that extremely biased data makes even the causal model fail to generalize. In the Inverted Pendulum, the advantage of FOCUS indicates that causal environment models depend less on the data diversity since FOCUS can still reach high scores in such a biased dataset where the baselines fail.

In Tab.3, we showcase results from more complex experiments, including HalfCheetah, Hopper, and Walker2D, alongside additional baseline methods (MOReL and COMBO). To ensure fair comparisons, we integrate the core module of FOCUS, namely learning and applying causal environment models, into these MBRL algorithms, thereby creating various FOCUS-version algorithms. We then compare these enhanced versions (denoted as “FOCUS”) against their original counterparts (denoted as “ORIGIN”), which do not integrate causal structures. On Random and Medium datasets, FOCUS-version algorithms significantly outperform the control groups; on Medium-Replay, FOCUS still achieves the best performance in 7 out of 9 settings. The results also confirm that FOCUS can be combined with most MBRL algorithms and effectively enhance performance.

6.4.2 Generalization ability

The generalization ability of FOCUS refers to if it can learn a good policy from the data with limited data size and low data diversity. Therefore we designed datasets from $1\mathrm{\%}$ to $100\mathrm{\%}$ of the original data size and datasets with a mix of $20\mathrm{\%}$ to $80\mathrm{\%}$ other datasets, where we can compare FOCUS and baselines in datasets with different sizes and diversities. The results in Fig.8 show that the advantage of FOCUS over MOPO is much more significant in small data sizes. In the dataset of $1\mathrm{\%}$ size, the advantage of FOCUS is relatively not significant because the size is too small. The results in Fig.9 show that FOCUS can perform well with a small ratio of Medium-Replay data while the baseline performs well only with a big ratio, which indicates that FOCUS is less dependent on the diversity of data. Related experiments on more environments can be found in Appendixes A2.

Fig.8 The comparison for data size. The $X\mathrm{\%}$ in the x-axis represents that the data size is $X\mathrm{\%}$ of the original size. The ratio $Y\mathrm{\%}$ in the y-axis represents the score ratio of FOCUS over the baseline MOPO

Full size|PPT slide

Fig.9 The comparison for data diversity. The dataset is produced by mixing up Medium-Replay and Medium with different ratios. The $X\mathrm{\%}$ in the x-axis represents that the data is mixed by $(100-X)\mathrm{\%}$ of the Medium and $X\mathrm{\%}$ of the Medium-Replay

Full size|PPT slide

6.5 Ablation study

To evaluate the contribution of each component in our causal discovery method, we perform an ablation study for FOCUS. The results in Tab.1 show that the KCI test and our principle of choosing conditional variables contribute to the causal structure learning of both accuracy and robustness.

7 Conclusion

In this paper, we point out that the spurious correlations hinder the generalization ability of current offline MBRL algorithms, and that incorporating the causal structure into the model can improve generalization by removing spurious correlations. We provide theoretical support for the statement that utilizing a causal environment model reduces the generalization error bound in offline RL. We also propose a practical algorithm, FOCUS, to address the problem of learning causal structure in offline RL. The main idea of FOCUS is to leverage conditional independence tests for causal discovery, which does not need further assumptions on the causal mechanism. In FOCUS, we address the difficulties of extending the PC algorithm in offline RL, particularly to reduce the number of independence tests by leveraging sequential information. Extensive experiments on the typical benchmarks demonstrate that FOCUS performs accurate and robust causal structure learning, surpassing offline RL baselines by a significant margin.

We would like to note that: In our theoretical results (Theorem 1 and 2), we assume that the true causal structure is already known. However, in practice, we must learn it from data before applying it (Section 5), which will introduce additional theoretical errors. As it is recognized that quantifying the uncertainty in the learned causal structure from data is a difficult task, we will derive the generalization error bound with the learned causal structure as part of our future study.

Zhengmao Zhu received the BSc degree in Department of Mathematical Science in June 2018 from Zhejiang University, China. He is currently pursuing the PhD degree with the School of Artificial Intelligence, Nanjing University, China. His current research interests mainly include reinforcement learning and causal learning. His works have been accepted in the top conferences of artificial intelligence, including NeurIPS, AAAI, etc. He have served as a reviewer of NeurIPS, ICML, AAAI, etc

Honglong Tian received the BSc degree in Software Institute of Nanjing University, China in June 2022. He is currently a graduate student of Software Institute of Nanjing University Nanjing University, China. His current research interests mainly include reinforcement learning. His works have been accepted in the top conferences of artificial intelligence, including NeurIPS, AAAI, etc

Xionghui Chen received the BSc degree from Southeast University, China in 2018. Currently, he is working towards the PhD degree in the National Key Lab for Novel Software Technology, the School of Artificial Intelligence, Nanjing University, China. His research focuses on handling the challenges of reinforcement learning in real-world applications. His works have been accepted in the top conferences of artificial intelligence, including NeurIPS, AAMAS, DAI, KDD, etc. He have served as a reviewer of NeurIPS, IJCAI, KDD, DAI, etc

Kun Zhang received the BS degree in automation from the University of Science and Technology of China, China in 2001, and the PhD degree in computer science from The Chinese University of Hong Kong, China in 2005. He is currently an associate professor with the Philosophy Department and an Affiliate Faculty Member with the Machine Learning Department, Carnegie Mellon University, Pittsburgh, USA. His research interests lie in causality, machine learning, and artificial intelligence, especially in causal discovery, hidden causal representation learning, transfer learning, and general-purpose artificial intelligence

Yang Yu received the PhD degree in computer science from Nanjing University, China in 2011, and is currently a professor at the School of Artificial Intelligence, Nanjing University, China. His research interests include machine learning, mainly reinforcement learning and derivative-free optimization for learning. Prof. Yu was granted the CCF-IEEE CS Young Scientist Award in 2020, recognized as one of the AIs 10 to Watch by IEEE Intelligent Systems, and received the PAKDD Early Career Award in 2018. His team won the Champion of the 2018 OpenAI Retro Contest on transfer reinforcement learning and the 2021 ICAPS Learning to Run a Power Network Challenge with Trust. He served as AC for NeurIPS, ICML, IJCAI, AAAI, etc

References

[1] Yu F, Xian W, Chen Y, Liu F, Liao M, Madhavan V, Darrell T. BDD100K: a diverse driving video database with scalable annotation tooling. 2018, arXiv preprint arXiv: 1805.04687

[2] Gottesman O, Johansson F, Komorowski M, Faisal A, Sontag D, Doshi-Velez F, Celi L A . Guidelines for reinforcement learning in healthcare. Nature Medicine, 2019, 25( 1): 16–18

[3] Yu T, Thomas G, Yu L, Ermon S, Zou J, Levine S, Finn C, Ma T. MOPO: model-based offline policy optimization. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1185

[4] Bengio Y, Deleu T, Rahaman N, Ke N R, Lachapelle S, Bilaniuk O, Goyal A, Pal C J. A meta-transfer objective for learning to disentangle causal mechanisms. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[5] de Haan P, Jayaraman D, Levine S. Causal confusion in imitation learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 1049

[6] Tenenbaum J. Building machines that learn and think like people. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2018, 5

[7] Edmonds M, Kubricht J, Summers C, Zhu Y, Rothrock B, Zhu S C, Lu H. Human causal transfer: challenges for deep reinforcement learning. In: Proceedings of the 40th Annual Meeting of the Cognitive Science Society. 2018

[8] Spirtes P, Glymour C N, Scheines R. Causation, Prediction, and Search. 2nd ed. Cambridge: MIT Press, 2000

[9] Zhang K, Peters J, Janzing D, Schölkopf B. Kernel-based conditional independence test and application in causal discovery. In: Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence. 2011, 804−813

[10] Sun X, Janzing D, Schölkopf B, Fukumizu K. A kernel-based causal learning algorithm. In: Proceedings of the 24th International Conference on Machine Learning. 2007, 855−862

[11] Heckerman D, Meek C, Cooper G. A Bayesian approach to causal discovery. In: Holmes D E, Jain L C, eds. Innovations in Machine Learning: Theory and Applications. Berlin, Heidelberg: Springer, 2006, 1−28

[12] Margaritis D. Distribution-free learning of Bayesian network structure in continuous domains. In: Proceedings of the 20th National Conference on Artificial Intelligence. 2005, 825−830

[13] Ke N R, Bilaniuk O, Goyal A, Bauer S, Larochelle H, Schölkopf B, Mozer M C, Pal C, Bengio Y. Learning neural causal models from unknown interventions. 2019, arXiv preprint arXiv: 1910.01075

[14] Wang Z, Xiao X, Xu Z, Zhu Y, Stone P. Causal dynamics learning for task-independent state abstraction. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 23151−23180

[15] Bellman R . A Markovian decision process. Journal of Mathematics and Mechanics, 1957, 6( 5): 679–684

[16] Kurutach T, Clavera I, Duan Y, Tamar A, Abbeel P. Model-ensemble trust-region policy optimization. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[17] Williams G, Wagener N, Goldfain B, Drews P, Rehg J M, Boots B, Theodorou E A. Information theoretic MPC for model-based reinforcement learning. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). 2017, 1714−1721

[18] Kidambi R, Rajeswaran A, Netrapalli P, Joachims T. MOReL: model-based offline reinforcement learning. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1830

[19] Akkaya I, Andrychowicz M, Chociej M, Litwin M, McGrew B, Petron A, Paino A, Plappert M, Powell G, Ribas R, Schneider J, Tezak N, Tworek J, Welinder P, Weng L, Yuan Q, Zaremba W, Zhang L. Solving Rubik’s cube with a robot hand. 2019, arXiv preprint arXiv: 1910.07113

[20] Hoerl A E, Kennard R W . Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 2000, 42( 1): 80–86

[21] Pearl J. Causality: Models, Reasoning, and Inference. Cambridge: Cambridge University Press, 2000

[22] Koller D, Friedman N. Probabilistic Graphical Models: Principles and Techniques. Cambridge: MIT Press, 2009

[23] Yu T, Kumar A, Rafailov R, Rajeswaran A, Levine S, Finn C. COMBO: conservative offline model-based policy optimization. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 2218

[24] Todorov E, Erez T, Tassa Y. MuJoCo: a physics engine for model-based control. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. 2012, 5026−5033

[25] Fu J, Kumar A, Nachum O, Tucker G, Levine S. D4RL: datasets for deep data-driven reinforcement learning. 2020, arXiv preprint arXiv: 2004.07219

[26] Xu T, Li Z, Yu Y. Error bounds of imitating policies and environments. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1320

Competing interests

The authors declare that they have no competing interests or financial conflicts to disclose.

RIGHTS & PERMISSIONS

2025 Higher Education Press