Risk-averse Policy Updates for Predictive Digital Twins

A digital twin (DT) is formally defined as “a set of virtual information constructs that mimics the structure, context, and behavior of a natural, engineered, or social system (or system-of-systems), is dynamically updated with data from its physical twin, has a predictive capability, and informs decisions that realize value. The bidirectional interaction between the virtual and the physical is central to the digital twin” [2]. A DT’s personalization distinguishes it from traditional modeling and simulation, in that it is linked with and represents the particular attributes of the underlying physical asset or process. This representation evolves dynamically, continually incorporating data from the physical world and refining the DT’s predictive capabilities accordingly. When creating a DT, one must treat different possible scenarios and rare events in a probabilistic way to ensure the model’s robustness.

DTs are currently the subject of widespread attention in the realm of computational science and engineering. As such, many industries are investing in virtual replicas of products and processes, leveraging data and numerical models, and improving decision-making strategies in areas like operations, planning, control, and predictive maintenance. The DT paradigm represents an opportunity for applied mathematicians, engineers, and computer scientists to integrate mathematical foundations, uncertainty quantification, domain knowledge, and software development into new products and services with societal and economic impact. Here, we discuss the use of graphical models, parametric Markov decision processes, and dynamic belief updates to build a risk-averse structural DT of an aerial vehicle.

The Physical-to-digital Information Cycle

DTs require a continuous information flow that automatically feeds data from the physical asset into the digital space and issues optimal actions back to the physical space. To model interactions between the physical and digital assets, we use a probabilistic graphical model (PGM) formulation that was initially developed for a structural DT of an unmanned aerial vehicle (UAV) [1]. The PGM tracks the variables of interest and their associated uncertainties, thus enabling continual structural monitoring of the UAV in a data-driven fashion. Its formulation is versatile, permitting the integration of physics-based and empirical models and enabling future state predictions with quantified uncertainty.

We model several key sources of uncertainty in the PGM as random variables: the underlying unknown physical state \(S\), the observational data from sensors \(O\), and the inferred digital state \(D\). Additionally, a reward function \(R\) helps us compute the optimal set of actions \(U\) for the duration of a mission; the model determines these actions according to the optimal policy that maximizes \(R\). The edges within the PGM represent relationships between the connected variables via conditional probability distributions. 

Figure 1 depicts the flow of information through the DT. We begin with data from strain sensors on the UAV’s wing and estimate the digital state, which represents the UAV’s structural health. The PGM then unrolls its structure for a new time step and propagates the uncertainties. The unrolling is realized at a given time by duplicating the nodes and edges of the graph for the next time step. Before issuing the optimal control input, we estimate the transition probability and recompute the optimal policy. Users can easily access the future state predictions by leveraging the subgraph that describes the digital space.

The conditional probability distributions that define the PGM are usually set a priori based on historical data or blueprints. We propose a numerical method to dynamically update the state transition probabilities with a Bayesian approach [4], which could increase the interconnectivity between the graphical model and the physical asset. Overall, better estimates translate into better policies and a more effective DT.

Learning State Transition Probabilities in the Operational Phase

We refine our understanding of the transition probability by observing a process that can be described by a yes or no question, which we model as a Bernoulli process. In this context, the question is “Did my state change?” or “Did the structural health of the UAV worsen?” The data that stems from answering the question (i.e., observing the Bernoulli process) update the posterior distribution that describes our current belief about the transition probability. In the big data limit, we eventually converge to the true underlying transition. From a mathematical point of view, we model the transition with a beta-distributed random variable and leverage the fact that the beta distribution is the conjugate prior to the Bernoulli distribution.

The probability of transitioning to a state \(S_t\)—given the physical state at the previous time step \(s_{t-1}\) and the action \(u_{t-1}\)—takes the form

\[p(S_t | S_{t-1} = s_{t-1}, U_{t-1} = u_{t-1}),\]

where the subscripts denote the time step and the capital letters denote random variables. If we consider a state that describes the structural health of the wing, the UAV’s future integrity depends on the type of maneuver that the controller issues; aggressive maneuvers with higher bank angles exert more stress on the structure than gentle maneuvers. Upper triangular band matrices describe the transition probabilities between our model’s discretized states, as we assume that we can never recover from damage during a mission. 

A Test Mission with a Risk-averse Policy

Consider a test mission with 40 time steps and an initial state that has 10 percent damage in both monitored regions (indicated by red patches in Figure 2a). The digital state can track the underlying unknown physical state, although a slight discrepancy occurs when we estimate the damage in the smaller of the two regions.

A common choice for the best estimate of the transition probability is the maximum a posteriori, which does not account for rare events and their associated risks. This shortcoming is particularly problematic for engineering applications, where we must remain aware of any risks of failure. However, we can use measures of risk to account for tail events and enhance the DT’s resilience. A common choice is the conditional value at risk (CVaR), which provides a measure of the average loss magnitude beyond the value-at-risk level [3]. Figure 2b depicts the prior and posterior distributions at the end of the test mission. The posterior mode is close to the ground truth, while the CVaR at the upper quartile is 50 percent higher. To obtain the associated optimal risk-averse policy, we solve a parametric Markov decision process that depends on the posterior estimate of the transition probabilities. The decision-making process is thus refined at every time step after the incorporation of new data from the physical asset. 

Predictive Digital Twins

The PGM probabilistically enables refined future state predictions at every time step. The initial digital state estimate in Figure 3 illustrates the probability of occupying one of the possible states in the next 35 or 70 time steps. More accurate predictions allow for better preventive maintenance and planning and lower operating costs.

Outlook 

In summation, our work integrates uncertainty quantification, risk awareness, adaptive planning for autonomous systems, and predictive capabilities. The use of parametric Markov decision processes allows us to dynamically update the underlying assumptions in the creation of a PGM for DTs. We also confirmed that the decision-making agent deploys a robust policy while dynamically refining its belief based on new sensor data. Dynamic updates to the policies are a crucial characteristic for every DT, because they increase the personalization and provides better decision-making capabilities while simultaneously accounting for risk and rare events. The DT can hence monitor the physical asset in a more realistic way, and the graphical model improves the DT’s predictive accuracy. Ultimately, our efforts to advance DT personalization represent a stepping stone towards enhanced predictive maintenance and monitoring. 

Marco Tezzele delivered a minisymposium presentation on this research at the 2023 SIAM Conference on Computational Science and Engineering (CSE23), which took place in Amsterdam, the Netherlands, last year. He received funding to attend CSE23 through a SIAM Early Career Travel Award. To learn more about Early Career Travel Awards and submit an application, visit the online page.

 Acknowledgments: We thank Michael Kapteyn for the illustrations of the physical asset.

References
[1] Kapteyn, M.G., Pretorius, J.V.R., & Willcox, K.E. (2021). A probabilistic graphical model foundation for enabling predictive digital twins at scale. Nat. Comput. Sci., 1(5), 337-347.
[2] National Academies of Sciences, Engineering, and Medicine. (2024). Foundational research gaps and future directions for digital twins. Washington, D.C.: The National Academies Press.
[3] Rockafellar, R.T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. J. Risk, 2(3), 21-41.
[4] Tezzele, M., Carr, S., Topcu, U., & Willcox, K.E. (2024). Adaptive planning for risk-aware predictive digital twins. Preprint, arXiv:2407.20490.