Papers
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao · ICLR 2023, 2023
The problem: reasoning and acting were studied separately
When the paper was written, large language models (LLMs) had already shown good performance in reasoning, for example with chain-of-thought (CoT), and in interactive decision making, for example by generating action plans. But these two capabilities had been studied as separate topics. The authors point out the limit of each side. CoT is, in their words, a static black box: the model uses only its internal representations, without grounding in the external world, and so cannot react to new information or update what it knows. This leads to hallucination of facts and to error propagation along the reasoning chain. Work that uses language models to act in interactive environments, in turn, tends to predict actions from prior language knowledge, without using the model to reason abstractly about high-level goals or to maintain a working memory that supports acting. The motivation comes from an observation about people: between two concrete actions, such as when cooking, we talk to ourselves to track progress, adjust the plan in the face of an exception and notice when external information is missing. The paper asks whether combining reasoning and acting in language models brings systematic benefits over using only one of the two.
The core idea: thoughts as actions that do not change the environment
The paper's formulation is simple. An agent receives observations from an environment and chooses actions based on a context that accumulates everything that has happened so far. ReAct expands the agent's action space to also include free-form language. An action in this language space is called a thought (or reasoning trace): it does not change the external environment and, therefore, produces no observation in return. Instead, the thought is added to the context to support later reasoning and actions. The authors list varied uses of thoughts: decomposing the goal into a plan, injecting commonsense knowledge, extracting what matters from an observation, tracking progress and changing plans, and handling exceptions. In practice, the work uses a frozen large model, PaLM-540B, and guides it only with a few examples in the prompt (few-shot). Each example is a human-written trajectory with thoughts, actions and observations. In tasks where reasoning is central, thought and action alternate over several thought-action-observation steps. In tasks with many actions, thoughts appear sparsely, and the model itself decides where to insert them.
Using tools: the Wikipedia API in HotpotQA and FEVER
For knowledge-intensive tasks, the authors use two datasets: HotpotQA, with questions that require reasoning over two or more Wikipedia passages, and FEVER, for fact verification, where each claim is labeled as supported, refuted or not enough information. The model receives only the question or the claim and must rely on its internal knowledge or look up what is missing. The tool is a simple Wikipedia API with three actions. The action search[entity] returns the first 5 sentences of the page, or suggests similar entities if the page does not exist. The action lookup[text] returns the next sentence on the page that contains the text, simulating a Ctrl+F. The action finish[answer] ends the task. The authors note that this API is much weaker than modern retrievers and say that the purpose of the design is to simulate how people interact with Wikipedia and to force the model to decide what to look up through explicit reasoning. The prompts used 6 manual examples on HotpotQA and 3 on FEVER. The resulting loop is: the thought decides what to look up, the action queries Wikipedia, the observation brings the result, and the next thought interprets it, which may lead to rewording the search before the final answer.
Results and failure modes: factuality versus flexibility
With PaLM-540B, ReAct outperforms Act (actions only, no thoughts) on both tasks: 27.4 vs 25.7 exact match on HotpotQA and 60.9 vs 58.9 accuracy on FEVER. Against CoT, the result is mixed: ReAct wins on FEVER (60.9 vs 56.3) and falls slightly behind on HotpotQA (27.4 vs 29.4). The authors manually analyzed 200 HotpotQA trajectories, 50 from each combination of method and success or failure. CoT suffers more from hallucination: 14% false positives vs 6% for ReAct, and hallucination accounts for 56% of CoT's failures, vs 0% of ReAct's failure samples. In exchange, ReAct's rigid structure reduces reasoning flexibility, and it has more reasoning errors (47% of failures vs 16%). A typical pattern is repeating the same thoughts and actions without getting out of the loop. In addition, 23% of ReAct's failures come from searches that return no useful information, which misleads the model. For that reason the authors propose combining the methods: falling back to CoT with self-consistency (CoT-SC) if ReAct does not answer within 7 steps on HotpotQA or 5 on FEVER, or falling back to ReAct when the CoT-SC majority is weak. The best combinations reached the performance of CoT-SC with 21 samples using only 3 to 5.
Decision making, limits, and future work
ReAct was also tested in two interactive environments. ALFWorld is a text game that simulates a household, in which the agent accomplishes goals by navigating and manipulating objects. WebShop is a shopping website with 1.18 million real products and user instructions. On ALFWorld, evaluated on 134 unseen games, the best ReAct attempt reached a 71% success rate, against 45% for the best Act and 37% for the best BUTLER, an agent trained by imitation. Even the worst ReAct attempt (48%) beat the best of the other two. A variant that only reacts to environment feedback (ReAct-IM) reached 53%, and the authors attribute the difference to the lack of goal decomposition and commonsense reasoning. On WebShop, ReAct had a 40.0% success rate, improving by 10 absolute points over imitation methods and imitation plus reinforcement learning methods, but expert humans reached 59.6%. As limitations, the authors note that tasks with large action spaces require more demonstrations, which may exceed the input limit of in-context learning. In an initial test, fine-tuning smaller models (PaLM-8B and 62B) with 3,000 correct trajectories made ReAct the best method on HotpotQA. The paper also cautions that connecting an LLM to actions in the world brings risks, and for that reason restricted the experiments to Wikipedia and WebShop.
See the ideas at work
Practice the designs that use this paper in the System Design school.