Embodied Web Agents:

Bridging Physical-Digital Realms for Integrated Agent Intelligence

Yining Hong*, Rui Sun*, Bingxuan Li, Xingchen Yao, Maxine Wu, Alexander Chien, Da Yin, Ying Nian Wu, Zhecan James Wang, Kai-Wei Chang

University of California, Los Angeles
^*Indicates Equal Contribution

arXiv Benchmark Web Environments Code Datasets

Illustrative examples of our Embodied Web Agents conceptual paradigm, tasks and environments.

Blue boxes and arrows indicate web interaction / switching to the web respectively. Orange boxes and arrows indicate acting in / switching to the embodied environment. We omit most intermediate actions due to the large number of interaction steps.

Abstract

AI agents today are mostly siloed — they either retrieve and reason over vast amount of digital information and knowledge obtained online; or interact with the physical world through embodied perception, planning and action — but rarely both. This separation limits their ability to solve tasks that require integrated physical and digital intelligence, such as cooking from online recipes, navigating with dynamic map data, or interpreting real-world landmarks using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI agents that fluidly bridge embodiment and web-scale reasoning. To operationalize this concept, we first develop the EMBODIED WEB AGENTS task environments, a unified simulation platform that tightly integrates realistic 3D indoor and outdoor environments with functional web interfaces. Building upon this platform, we construct and release the EMBODIED WEB AGENTS Benchmark, which encompasses a diverse suite of tasks including cooking, navigation, shopping, tourism, and geolocation — all requiring coordinated reasoning across physical and digital realms for systematic assessment of cross-domain intelligence. Experimental results reveal significant performance gaps between state-of-the-art AI systems and human capabilities, establishing both challenges and opportunities at the intersection of embodied cognition and web-scale knowledge access.

Embodied Web Agents

We introduce Embodied Web Agents as a new conceptual paradigm of AI systems that unify physical embodiment with web-scale knowledge access — capable of perceiving and acting in the real world while reasoning over dynamic, unstructured information from the web.

Task Environments

To operationalize this concept, we first develop the Embodied Web Agents task environments, a unified simulation platform that integrates realistic 3D environments with interactive web interfaces. This platform combines (1) indoor settings from AI2-THOR, (2) outdoor navigation in Google Earth, and (3) web interfaces including Wikipedia, online stores, recipe websites, map services \textit{etc.}, enabling agents to interact seamlessly with both physical and digital spaces.

Benchmark

We construct the Embodied Web Agents Benchmark, which encompasses approximately 1.5k tasks across multiple domains, including: (1) cooking tasks where agents match physical ingredients with online recipes; (2) navigation combining online maps with physical wayfinding; (3) shopping requiring coordination between in-store actions and online options; (4) tourism connecting physical landmarks with web information; and (5) geolocation determining position through embodied exploration and online research. Together, these tasks systematically test an agent's ability to bridge embodied perception, action, and web-based reasoning across varied contexts.

Example

An Exemplar Pipeline of completing a task in our Embodied Web Agents dataset. Blue boxes indicate web interaction. Orange boxes indicate embodied interaction. Boxes with gradient colors indicate switching from one environment to the other.

Experiments and Results

[1/3] Outdoor Experiment

For outdoor planning, we use GPT-4o-mini alongside Gemini 2.0 Flash, Qwen-VL-Plus, and InternVL2.5-latest to evaluate performance across navigation, shopping, and traveling tasks. For web observation, we follow the setting of VisualWebArena.

We observe that:

1) GPT-4o-mini consistently leads across all metrics, with the highest accuracy in navigation (34.72%), shopping (25.46%), and traveling (30.91%), though still well below human performance. Gemini follows closely behind, while Qwen and Intern lag behind.
2) Web-only accuracy exceeds embodied-only accuracy for all outdoor tasks, suggesting models handle digital information more effectively than physical navigation.
3) Generally, completion rates are satisfactory, while overall accuracies are very low across all tasks. This indicates models can execute parts of complex tasks but struggle with consistent cross-domain reasoning over longer sequences.
4) From task perspective, shopping and traveling involve richer interactions between the embodied environment and the web than navigation, and each task spans longer steps. As a result, the overall accuracy for shopping and traveling is noticeably lower than for navigation. This highlights the difficulty of cross-environment tasks, particularly those that are lengthy and involve multiple steps, for current models.

[2/3] Cooking Experiment

For indoor cooking, we implement two distinct approaches: vision-based and text-based.

A substantial performance gap exists between AI models and humans, with the best model (text-based GPT-4o) achieving only 6.4% overall accuracy compared to humans' 77.08%.

Text-based models using structured scene graphs consistently outperform their vision-based counterparts using first-person views, suggesting current models struggle to ground visual observations effectively in cooking contexts.

GPT-4o and Gemini-2.0-Flash demonstrate substantially stronger performance than Qwen-VL-Plus/Qwen-PLUS and InternVL/InternLM across both modalities.

[3/3] Geolocation Experiment

For geolocation tasks, we benchmark against FairLocator, a study analyzing VLM performance on GeoGuessr using Google Street View images.

As shown in Table, our agent, capable of active exploration and web information access, significantly outperforms the passive baseline, particularly in identifying finer-grained locations like cities and streets.

This substantial improvement underscores the potential of integrating embodied and web domains to enhance performance across numerous real-world tasks, warranting further investigation.

Error Analysis

This figure presents a detailed breakdown of error types and their percentages that contribute to task failures in cooking tasks when using GPT-4o. Our analysis reveals that the primary challenges in embodied web agents lie not in isolated capabilities, but in their integration. The most prevalent failure pattern involves agents becoming trapped in single-domain cycles. While embodied errors (14.6%) and web errors (8.0%) occur, cross-domain errors (66.6%) overwhelmingly dominate the failure landscape — confirming that the critical bottleneck emerges at the intersection where physical and digital domains meet.

BibTeX

@misc{hong2025embodiedwebagentsbridging,
          title={Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence}, 
          author={Yining Hong and Rui Sun and Bingxuan Li and Xingcheng Yao and Maxine Wu and Alexander Chien and Da Yin and Ying Nian Wu and Zhecan James Wang and Kai-Wei Chang},
          year={2025},
          eprint={2506.15677},
          archivePrefix={arXiv},
          primaryClass={cs.AI},
          url={https://arxiv.org/abs/2506.15677}
        }