A. 交互式多模态感知

和人类一样，机器人可以以被动或交互的方式感知环境[12]。被动感知是指视觉或听觉监控等方式，它允许机器人快速获取信息而无需干预周围环境。然而，被动方式有其局限性，其中最突出的是在面对认知不确定性 [6]时无能为力，认知不确定性是由于缺乏知识而导致的。

认知不确定性不可避免地来自各种来源，例如来自人类指令的歧义、低分辨率感知（例如卷积的图像尺寸减小）或模态不足。其中许多只能通过交互式感知来减少，其中机器人主动询问环境以提高准确性甚至发现潜在信息。例如，当被要求提供钢螺丝而不是具有相似颜色和形状但由铝制成的螺丝时，助手机器人可能需要使用被动视觉来定位可能的候选者，并进一步以交互方式使用称重或磁性模块进行确认。

Despite the promising advantages, interactive perception is less common than the passive manner because it entails increased complexity [14]. Efforts are needed to design a mediating system to handle various sensory data and to adapt to changes in the conditions of both the robot and the environment, such as a new robotic modular being available or the involvement of novel objects.

B. Chatting With the Environment

LLMs have been showing incredible potential in areas besides robotics [1], [7], [15], [16]. Human knowledge that resides in LLMs can help a robot abstract and select only suitable features, e.g. relevant to the region of interest or informative modalities, to simplify the learning process. Moreover, in terms of generalizability, the knowledge of LLMs allows a behavioral agent to adapt efficiently to novel concepts and environmental structures. For instance, when being asked to use one adjective for each to describe how a sponge and a brick feel, ChatGPT¹ will respond with “soft” and “hard” respectively. This is helpful for a robot with a haptics sensing module to distinguish between these two novel, never-seen objects.

LLMs are usually generative models that predict tokens to come, but with certain designs, e.g. conversational prompts, LLMs are capable of generating chat-fashion texts. This allows their integration with a robot to not only plan with respect to a robot's built-in ability [27], [1] but also respond according to environmental feedback.

However, they cannot directly process application-specified raw multimodal data. We resort to modular perceptions for each modality that are separately trained before being plugged into the LLM backbone. Each module semantically translates the resulting multimodal sensations into natural language that can be understood by LLMs and processed in a unified manner.

Our contributions are threefold. Firstly, we establish a manipulation scenario with multimodal sensory data and language descriptions. Secondly, we propose Matcha² (Multimodal environment chatting agent), where an LLM is prompted to work in a chatting fashion, thus having continuous access to environmental feedback for contextual reasoning and planning. Finally, we show that LLMs can be utilized to perform interactive multimodal perception and behavior explanation. Accordingly, an interactive robot can make reasonable and robust decisions by resorting to LLMs to examine objects and clarify their properties that are essential to completing the task (see Fig. 1).

Multimodal Learning and Robotic Information Gathering

Research in multimodality in robotics nowadays attracts growing attention [2] because of its success in, for example, audio-visual learning [28], [26], [30] and language-visual learning [20], [21]. It is beneficial and sometimes essential for a robot to learn from multimodality because one modality could carry some distinct information, e.g. tones in speech, that cannot be deduced from another. [13].

Capable robots require managing one or several sensors to maximize the information needed for disambiguation [4] regarding a specific goal. This problem is known as active information acquisition [3], [24] or, particularly in robotics, robotic information gathering [19], where robots have to properly select perceiving actions to reduce ambiguity or uncertainty. Besides handcrafted rules, some information advantage measures, e.g. entropy or information gain, are usually employed to maximize [3]. However, the combination of multimodal data is usually challenging. There are studies on fusing multimodal data according to their uncertainties, but this may face numerical instability and is difficult to transfer from one application to another [25]. Instead of directly fusing the multisensory data in a numerical space, we propose to use multimodal modules to translate them into natural language expressions that an LLM can easily digest.

Large Language Models in Robotic Planning

Very recent works use LLMs to decompose high-level instructions into actionable low-level commands for zero-shot planning. They use LLMs as a planner to autoregressively select actions that are appropriate with respect to the instruction according to application-based prompts [27], the semantic similarity between mapped pairs [10], or the contextual language score grounded on realistic robot affordances [1]. Other approaches ground LLM knowledge in human interaction [7] or many other various fields where domain knowledge is distinct and modular frameworks can be composed via language as the intermediate representation [18], [16], [27].

However, these works design a robot to form a planning strategy with built-in knowledge, rather than interact with the surroundings and make decisions based on actively collected information from the environment. There is no feedback loop for their LLMs to perceive the environmental cues, such that only “blind” decisions are made in the robotic unrolling process. In contrast, our interactive architecture allows LLMs to access the environment state from multiple modalities for adaptive planning.

A. Architecture

We propose Matcha (multimodal environment chatting agent) which is able to interactively perceive (“chat” with) the environment through multimodal perception when the information from passive visual perception is insufficient for completing an instructed task. The epistemic actions are executed autoregressively until the agent is confident enough about the information sufficiency in that situation.

Fig. 2 provides an overview of the architecture of Matcha. It is a modular framework of three parts: an LLM backbone, multimodal perception modules and a low-level command execution policy. They connect to each other with language as the intermediate representation for information exchange.

To be specific, given a high-level instruction, especially the one that Matcha cannot directly perform with the command policy alone, the LLM backbone will reason the situations and select the most contextually admissible perceiving command to gather information. After the execution of the policy module, the resulting environmental response is processed by a correspondingly evoked multimodal perception module into semantic descriptions, e.g. “clinking sound” by an auditory module after the “knock on” action. Finally, the executed command itself as well as the environmental state description are fed back to the LLM for future planning.

The LLM is used in a few-shot manner without any need for fine-tuning, being independent of other components. Policy and perception modules can be separately designed and plugged into the framework whenever needed. Intrinsically linked by natural language, this framework is flexible and can scale and adapt easily to possible robotic upgrades or diverse robotic scenarios.

Fig. 2:

Overview of Matcha. The framework contains an LLM, multimodal perception modules, and a language-conditioned policy. These components communicate with each other with natural language as the intermediate representation. Three types of language information are involved in composing the prompt: I is a language instruction from the user, C is a language command produced by the LLM, and F is semantic feedback from multimodal perceptions. Dotted lines indicate possibly evoking paths.

Show All

B. Multimodal Perception and Execution Policy

We select a commonly feasible suit of modalities and a language-conditioned policy as an example implementation of our framework. Other varieties for specific scenarios can also be easily integrated due to the flexibility of modularity of the framework. Detailed experimental implementations will be introduced in Sec IV.

C. Prompt Engineering

An issue of grounding LLMs on robotic scenarios is that some of the suggestions generated by LLMs are not executable for a specific robot [1], [10], which stems from the fact that LLMs are pre-trained with extremely large open-domain corpora, while the robot is constrained by its physical capability and application scenarios, e.g. a tabletop robot is not able to perform “walk” action.

In this work, the LLM is applied for few-shot planning [16], [27], in which all the executable commands are defined together with several task examples as the initial “chat” history. See Tab. I for the leading prompt which enables the LLM to ground on the specific scenario and follow the contextual patterns for commanding the execution policy.

Table I: The snippet of the 5-shot prompt setting. The other four exemplars are omitted here due to the content limit.

We found that only language models that are large enough can follow the patterns in the prompt strictly, i.e. only generate commands that have been defined in strictly case-sensitive letters and with the same amount of allowed parameters for each, while small ones can hardly obey this constraint and generate unexpected commands, which brings extra demands for tuning. As the action planning is performed by LLMs constrained by a given prompt, the proposed framework demonstrates high flexibility and generalizability upon the possible incorporation of novel actions or perception modules into the system.

A. Experimental Setup

We evaluate the proposed framework in an object-picking task: a robot is instructed to pick up an object that is referred to by a latent property - material - which is, however, not visually distinguishable under our settings. Tasks are intentionally designed such that information from a single modality could be insufficient to determine object properties, while other perception sources can provide compensations to reduce or eliminate this ambiguity. For example, glass and metal surfaces could exhibit similar hard and smooth properties upon contact, in which case differences in impact sound can aid in further differentiation. Tab. II lists variational multimodal descriptions of the materials. These properties are wrapped as natural language sentences before being fed back to the LLM.

Experiments are done in CoppeliaSim³ simulations with the NICOL robot [11], where several blocks in various colors, materials, weights, and surface textures are randomly selected and placed on the table next to a brown container (see Fig. 1). The ViLD [9] model is meant to be easily generalized to describe complex scenes despite the simplicity of the object setting here. After detection, the objects are represented universally by their name, which serves as a parameter for the action function to identify. Objects with the same color will be distinguished as “on the left/right” given the simplicity of avoiding more than two duplicated colors for the same shape. The desktop robot is equipped with two Open-Manipulator-Pro arms ⁴, but only its right arm is activated to operate. It is capable of executing actions in [“knock on”, “touch”, “weigh”, “pick up”], with a parameter to indicate the targeted object. The first three actions correspond to the interactive perception of impact sound, haptics, and weight respectively, and the last action finalizes the task by picking and transporting an object into the box. Each instruction is guaranteed to be achievable with the capability of the robot.

Due to the lack of support for physics-driven sound and deformable object simulation in Coppeliasim, we have implemented reasonable alternatives. For the haptics of objects, we simplify haptic perception by assigning variational descriptions regarding its material, e.g. fibrous objects are usually perceived as “soft” and a plastic object can be either “soft” or “hard”. Note that advanced implementations can also be achieved using a neural network as is used in the sound perception module when haptics data for deformable objects is available. For the impact sound, we split the YCB-impact-sound dataset [8] into training and testing sets and augment them with tricks such as shifting, random cropping, and adding noise. The training set is used to train our auditory classification neural networks, while the audios in the testing part are randomly loaded as an alternative to run-time impact sound simulation for the materials mentioned,

Sound can be informative, though not perfect, for determining materials [8]. Besides showing the mediating ability of multiple modalities by the LLM, we further investigate its reasoning ability by employing indistinct descriptions instead of exact material labels.

• Distinct description: the sound module describes sound feedback by the corresponding material name and its certainty from the classification model, e.g. “It is probably glass” or “It could be plastic with a 47% chance, or ceramic with a 35% chance”. The distinct description setting is more task-oriented, and it examines the robot's ability to mediate multiple sensory data for disambiguation.

• Indistinct description: we listed some commonly used indistinct sound descriptions in human communications in Tab. II, e.g. using “dull” to describe the sound from a plastic block and “tinkling” to describe the sound for both ceramic and glass objects. This setting is more task-agnostic and thus has the potential for generalization. Moreover, it compels the LLM to infer “professional” material terminology from ambiguous yet multimodal descriptions.

The online OpenAI text-davinci-003 API⁵ is applied as the LLM backend because it demonstrates robust instruction-following ability and outstanding reasoning performance. ⁶ We also evaluate with a weaker but far less expensive LLM text-ada-001, a GPT-3 model which is usually fast and capable of simple tasks, under the same setting as comparison.

Table II: Property descriptions of different materials.

B. Results

We test the proposed framework Matcha in 50 randomly generated scenarios for each setting and report the success rate.

We report that the impact sound classification model pretrained with the selected materials achieves an accuracy of 93.33%. When using distinct descriptions, suppose we are making hard-coded rules to utilize the sound module to identify the targeted material, the robot can randomly knock on an object among three, and classify the material until the one that is classified as the target. In theory, the success rate computes as 13p+23p2|p=93.33%=89.18%, where p is the modular accuracy. Usually, other modalities, in this case, are not as distinct as sound, and it could be non-ideal for humans to craft such fusion rules for a possible slight improvement. Therefore, the theoretical success rate with only the sound module will be used as our baseline for analysis. Note that this is a reasonable rule that humans will follow, thus it can also be regarded as the upper bound for Matcha if it worked with only impact sound.

Unsurprisingly, Matcha achieves a relatively higher success rate of 90.57% compared to the ideal theory baseline, as it utilizes compensatory information from other modalities in addition to sound. When using the indistinct description of impact sound, Matcha is still able to achieve a success rate of 56.67%, which is larger than a chance success rate of 33.33% achieved by randomly picking one from the three. This result is remarkable as it performs few-shot deduction with only indistinct adjectives available. By analyzing the failure cases, we found that the similar descriptions of glass and ceramic in terms of impact sound, haptics, and weight make it challenging to distinguish one material from the other. This is not an issue with distinct descriptions, where the sound classification network directly predicts the material, preventing a significant loss of information in the translation from sensory data to language. While the system displays remarkable reasoning ability, the results still highlight the significance of a well-designed modular description that is not overly abstract in facilitating interactive perceptions.

We observe that only strong LLMs (GPT-3 text-davinci-003 in our experiments) can perform the task following the primary prompt as we defined, while weak ones possibly generate commands that are not executable and show degraded performance (see Tab. III). For example, GPT-3 text-ada-001⁵, a smaller model in the GPT-3 model series, may generate commands like “robot.knock.on(metal block)” or “robot. weight (yellow block, blue block)”, which is not feasible for the policy of the robot.

Table III: Effect of different LLMs on success rate.

C. Case Studies

We provide case studies to showcase the interactive perception ability of Matcha in multimodal environments. Following the convention of this paper, texts containing information from humans, LLMs, and the environment are indicated in red, green, and blue, respectively. Fig. 3 gives a completion that the agent can deduce the latent material property from multimodal yet indistinct descriptions. During the planning process, the agent decidedly terminates exploration of the “red block” and instead engages in interacting with the “green block” exhaustively for a comprehensive examination. Common sense and, moreover, established knowledge in the LLM enables Matcha to efficiently interact with the environment. Matcha's proficient behaviors provide evidence of effective reasoning, as it aligns with the subsequent explanation provided by the LLM, namely, that fiber can often be considered “flexible” rather than “cold and smooth”. The example depicted in Fig. 4 presents a fascinating observation: the impact sound of the “orange block” suggests it is more likely to be plastic than metal, but Matcha accurately distinguishes it from plastics after engaging in the interactive perception of weight and haptics. This showcases the potential of multimodal perception to improve classification accuracy. Fig. 5 provides a failure case with indistinct descriptions where the robot picks up a ceramic block when asked to pick up the one made of glass. The underlying cause of this failure is the sensing similarity between glass and ceramic, which creates difficulty in resolving epistemic uncertainty.

Fig. 3:

A successful example in which the robot deduces “fiber” material with indistinct descriptions of impact sound.

Show All

D. Discussion

Weak LLMs, e.g. ones without fine-tuning on instruction alignment [17], may not have sufficient capability for precise planning, and thus may require carefully engineered prompts or other grounding techniques. On the other hand, strong LLMs exhibit impressive in-context learning abilities. These observations highlight the potential of leveraging knowledge within strong LLMs, as it enables the successful execution of tasks that were previously deemed infeasible. LLMs can derive significant advantages from utilizing common knowledge, being robust to various instructions regardless of their changes in synonym, linguistic structure or even semantic meanings out of the scope that the robot is initially designed within, e.g. an instruction variation from “the metal block” to “a block that may be suitable for cracking a nut”, in which the robot has to establish a meaningful connection between the object's multimodal perceptions and the required utility.

Fig. 4:

A successful example with a distinct description of impact sound. This example shows that by leveraging multimodal perception, LLM rectifies the misclassification that may occur when relying solely on sound modules.

Show All

Fig. 5:

An example in which the agent fails to distinguish glass and ceramic in the setup of using indistinct descriptions of impact sound.

Show All

Nevertheless, the reasoning trace may not always align with human expectations. There are cases that LLMs may prematurely draw conclusions due to their limited logical reasoning ability, particularly when faced with a task that requires reasoning from a long list of facts.