processes, specifically, with the Ornstein–Uhlenbeck (OU) process. The OU process models parameters to have local correlation with some noise but a global reversion to the mean. An example of an OU process of feedstock composition is shown in Figure 2. Mathematically, an MDP is defined by a tuple ⟨S, A, T, R, γ⟩, where S is the state space, A is the action space, T is the transition function, R is the reward function, and γ is the discount factor. The MDP framework is a broad approach to sequential decision-making in stochastic environments that forms the basis for most reinforcement learning. In all MDPs, an agent makes decisions (referred to as actions) at discrete timesteps, which influence how the system transitions from one state to the next. The current state of the system and action taken at a given timestep result in a reward that the agent receives, which represents the optimization objective. The way that actions are chosen based on the current state or observations of the system is called a policy. An intelligent agent decides a policy by learning from interacting with the system over time (Xu et al., 2025; Kochenderfer et al., 2022). The state is defined as the set of mines, piles, and processes in the network, and the following important attributes of each that evolve over time. For each mine, these are: the current layer, how many tons of material have been extracted from each layer, and the current chemical composition measured at the surface of each layer. For each pile, these are: the current inventory in tons and its chemical composition. For each process, these are: the current output in tons and its chemical composition. “Current” means on that day of the simulation, and the state contains the day as well to track this. The action is defined as the flow assignments: how many tons of material are sent from each mine to each pile, each pile to each process, and each process to each client on a given day. The transition function describes how the state of the mines, piles, and processes evolves into the next state given the action that was taken. Along with the that are generated by The OU processes describe the probability distribution that produce the feedstock composition and process factors, and thereby provide the transition uncertainty for how the current state and action (i.e., flow assignment) result in the next state in a non-deterministic fashion. Finally, the reward represents the optimization objective, and as stated earlier, is defined as the profit (revenue generated from selling to clients minus mine and processing operating costs). The discount factor describes how much potential future rewards should be discounted relative to the present. Since we want to optimize for long-term profit, we set the discount factor high, to 0.99. 3.4 Optimization-under-Uncertainty Approach We employ a Monte Carlo tree search (MCTS) algorithm with double progressive widening (DPW) to solve the MDP. MCTS finds optimal policies by performing rollout simulations into the future, simulating possible future states based on the transition function. DPW is a technique used to explore continuous state and action spaces without being overwhelmed by the infinite possible scenarios. By simulating future scenarios over the probability distribution of possibilities, this approach inherently encodes uncertainties and probabilities in its decision-making, which it can perform over long time horizons. As an additional comparison, we use a slightly modified version of MCTS we call greedyguided MCTS (ggMCTS), which seeds the tree search with a prior of that day’s greedy action (i.e., single-day optimum computed from the MILP optimizer).
RkJQdWJsaXNoZXIy MTM0Mzk2