Showing Posts From

Llm agents

Navigating the Browser Jungle with LAMs The web has long been a sandbox for human curiosity, but the next frontier is letting machines roam it autonomously. Large Action Models (LAMs) extend the language‑model paradigm from pure text generation to action generation: they emit structured commands—clicks, form fills, JavaScript snippets—that a browser runtime can execute directly. In contrast to classic headless‑browser bots that follow hard‑coded scripts, LAMs reason about the state of the Document Object Model (DOM), plan multi‑step navigation, and adapt on the fly when pages diverge from expectations. Why does this matter now? The 2024 “Can Language Models Play the Wikipedia Game?” study demonstrated that state‑of‑the‑art LLMs (GPT‑4o, DeepSeek‑R1, QwQ:32B) can navigate massive hyperlink graphs with success rates comparable to humans, yet they still stumble on edge‑case link structures and multilingual quirks. Those same navigation capabilities translate directly to browser environments: a LAM must decide which hyperlink or button best advances its goal, just as a human player does in the Wikipedia Game. Simultaneously, the “Towards Physically Grounded Articulated 3D Assets” paper introduced multi‑agent pipelines that close the loop between generation and physical simulation. The underlying lesson is that feedback loops—where an agent’s output is validated by a simulator—are essential for robust autonomy. In the browser world the “simulator” is the live page, and the feedback is the DOM after each action. In this article, I will dissect the technical stack that makes LAM‑driven browser automation possible, illustrate it with production‑grade code, compare the leading frameworks, and outline safety practices that keep autonomous agents from becoming runaway click‑farms. By the end, you’ll have a reproducible blueprint to spin up a fleet of LAM agents that can scrape, fill, and even test web applications without a human in the loop.From Links to Logic: How LAMs Interpret the DOM A LAM’s first job is to understand the current page. Unlike static crawlers that rely on TF‑IDF or simple XPath queries, LAMs ingest the raw HTML, render it (often via a headless Chrome instance), and feed a structured snapshot of the DOM into a transformer. The snapshot includes:Tag hierarchy (parent‑child relationships) Visible text extracted via the Accessibility Tree Computed CSS properties (e.g., display: none to filter hidden elements) Event listeners attached to nodes (click, submit, change)The model then produces a probability distribution over possible actions, each expressed as a JSON‑compatible “action schema”: { "action_type": "click", "selector": "#login-button", "reasoning": "Button labeled 'Login' is the only visible CTA." }Embedding the DOM as a sequence of tokens is non‑trivial. Recent work (e.g., the OpenAI “WebGPT” approach) uses a flattened DOM linearization where each node becomes a token like <div id="main">. This representation preserves hierarchical cues while staying within the token limits of even 70‑B models.The real breakthrough comes from retrieval‑augmented prompting: the LAM queries an external knowledge base (e.g., a citation graph from OpenAlex) to resolve domain‑specific terminology before deciding which link to follow. In the Wikipedia Game experiments, models that combined sentence embeddings with external retrieval outperformed pure‑LLM baselines, echoing the same pattern for browser tasks where domain‑specific forms (tax filing, medical portals) require contextual grounding.Agentic Prompting and Tool Use: The Browser as a First‑Class API Traditional LLM agents treat browsers as opaque tools: “open URL X, then click the button labeled Y.” LAMs elevate the browser to a first‑class API by exposing a rich set of primitives:Primitive Signature Examplenavigate(url: str) Opens a new page navigate("https://news.ycombinator.com")click(selector: str) Clicks an element click("#morelink")type(selector: str, text: str) Sends keystrokes type("#search", "LAMs")evaluate(js: str) Runs arbitrary JavaScript evaluate("document.title")screenshot(path: str) Captures the viewport screenshot("/tmp/page.png")These primitives are wrapped in a JSON‑RPC layer that the LAM calls over a local WebSocket. The agent can also request tool extensions, such as a headless PDF renderer or a CAPTCHA solver, and the LAM will incorporate the tool’s output into its reasoning chain. The prompting template that yields the best performance mirrors the “chain‑of‑thought” style that won the 2023 LAMBADA competition: You are a Large Action Model controlling a browser. The current page title is "{{title}}". Your goal: "{{goal}}". Consider the DOM, any visible text, and available actions. Respond with a JSON action.Because the model receives the entire action schema in its context, it learns to reason about why a particular selector is optimal—a behavior observed in the Wikipedia Game where the best models articulated their link‑choice rationale before clicking.Safety Nets: Sandboxing, State Persistence, and Rollback Strategies Unsupervised browsing is a high‑risk activity. A LAM could inadvertently trigger malicious downloads, submit harmful data, or cause denial‑of‑service on the target site. The community has converged on three safety pillars:Containerized Sandboxing – Each LAM instance runs inside a lightweight Docker container with a restricted network namespace. Outbound traffic is filtered through a proxy that blocks known malware hosts and limits request rates.State Versioning – After every action, the browser’s snapshot (HTML, cookies, localStorage) is serialized to a Redis‑backed version store. If the LAM deviates from its goal (e.g., lands on a login page when it should stay on a product page), a supervisory controller can rollback to the previous safe state and re‑prompt the model with corrective instructions.Policy‑Driven Action Filters – Before the LAM’s action is executed, a deterministic policy engine validates the selector against a whitelist (e.g., only elements with role="button" are clickable). This prevents accidental clicks on hidden tracking pixels or malicious ads.Below is a minimal Python watchdog that enforces these rules for a Selenium‑driven LAM session: import json, redis, selenium.webdriver as webdriverr = redis.StrictRedis(host="redis", port=6379, db=0)def save_snapshot(driver, step): state = { "url": driver.current_url, "html": driver.page_source, "cookies": driver.get_cookies() } r.set(f"lam:snapshot:{step}", json.dumps(state))def is_allowed_action(action): # Simple whitelist: only click buttons or links with visible text if action["action_type"] not in {"click", "type", "navigate"}: return False if action["action_type"] == "click": return bool(action["selector"]) return Truedef execute_action(driver, action): if not is_allowed_action(action): raise PermissionError("Disallowed action attempted") if action["action_type"] == "navigate": driver.get(action["selector"]) elif action["action_type"] == "click": driver.find_element_by_css_selector(action["selector"]).click() elif action["action_type"] == "type": el = driver.find_element_by_css_selector(action["selector"]) el.clear() el.send_keys(action["text"]) save_snapshot(driver, action.get("step", 0))By coupling this watchdog with a policy‑as‑code file (see next section), teams can audit every click the LAM makes, satisfying both security auditors and ethical AI guidelines.Benchmarking Autonomy: Lessons from the Wikipedia Game and 3D Asset Pipelines The two arXiv studies provide a natural yardstick for LAM performance:Wikipedia Game – Success is measured by reaching the target article within a fixed number of clicks. Human baselines hover around 78 % success; the best LLMs (GPT‑4o, DeepSeek‑R1) achieve ~73 % after fine‑tuning. The key metric is click efficiency—the ratio of clicks taken to the optimal shortest path.Physically Grounded 3D Assets – Here, the benchmark evaluates physical plausibility (collision‑free motion) and functional fidelity. The Artisan pipeline uses multi‑agent feedback loops where a physics engine validates each generated articulation before acceptance.Translating these concepts to browsers, we define two complementary benchmarks:Benchmark Goal Success MetricLink‑Reach Navigate from a start URL to a target page using only visible UI elements. % of runs reaching target within N actions.Form‑Fill Accuracy Complete a multi‑step web form (e.g., tax filing) without validation errors. % of fields correctly populated, measured against a ground‑truth schema.Stability Score Maintain a consistent DOM state across repeated runs (idempotence). Collision‑rate analogue: number of unexpected redirects or error pages per 100 runs.In early experiments, LAMs that incorporated retrieval‑augmented prompting (pulling relevant API docs from OpenAI’s embeddings) reduced the average click count by 12 % on the Link‑Reach benchmark. Adding a physics‑style feedback loop—where the watchdog checks for DOM “collisions” (e.g., duplicate element IDs)—improved stability from 85 % to 94 %. These findings reinforce a broader principle: iterative validation—whether via a physics engine or a DOM watchdog—dramatically lifts autonomous performance.Scaling Up: Distributed LAM Orchestration with Docker & Kubernetes When moving from a single proof‑of‑concept to production, you need a robust orchestration layer. The pattern that has emerged in the last year is to treat each LAM instance as a microservice exposing a JSON‑RPC endpoint, and to manage them with Kubernetes. Below is a concise docker‑compose.yml that spins up three components: version: "3.9" services: redis: image: redis:7-alpine ports: ["6379:6379"] restart: unless-stopped lam-worker: image: ghcr.io/yourorg/lam-browser:latest environment: - REDIS_HOST=redis - SELENIUM_HOST=selenium depends_on: [redis, selenium] deploy: replicas: 3 resources: limits: memory: "2G" cpus: "1.0" restart: unless-stopped selenium: image: selenium/standalone-chrome:latest shm_size: "2g" ports: ["4444:4444"] environment: - SE_NODE_MAX_SESSIONS=5 - SE_SESSION_REQUEST_TIMEOUT=300 restart: unless-stoppedIn a Kubernetes setting, each lam-worker would be a Deployment with a Horizontal Pod Autoscaler (HPA) keyed to the length of the Redis task queue. A CustomResourceDefinition (CRD) called LamTask can encapsulate the goal description, start URL, and timeout. The controller watches for new LamTask objects, dispatches them to an available worker, and records the final state back into a PostgreSQL store for analytics. By decoupling the browser runtime (Selenium) from the model inference (served via an OpenAI‑compatible endpoint), you can horizontally scale each tier independently—critical for high‑throughput scraping or large‑scale UI testing pipelines.Putting It All Together: A Minimal End‑to‑End Pipeline Below is a compact Python orchestration script that ties the pieces: it fetches a task from Redis, sends it to the LAM RPC endpoint, validates the returned action with the watchdog, and loops until the goal is met or a timeout occurs. import json, time, redis, requestsREDIS = redis.StrictRedis(host="redis", port=6379, db=0) LAM_ENDPOINT = "http://lam-worker:8000/rpc"def fetch_task(): raw = REDIS.lpop("lam:task_queue") return json.loads(raw) if raw else Nonedef call_lam(payload): resp = requests.post(LAM_ENDPOINT, json=payload, timeout=30) resp.raise_for_status() return resp.json()def main(): while True: task = fetch_task() if not task: time.sleep(2) continue state = {"url": task["start_url"], "goal": task["goal"], "step": 0} while state["step"] < task.get("max_steps", 20): lam_input = { "dom_snapshot": state.get("dom"), "goal": task["goal"], "history": state.get("history", []) } action = call_lam(lam_input) try: execute_action(driver, action) # from watchdog example except PermissionError as e: print("Policy violation:", e) break state["history"].append(action) state["step"]### Related Insights- [The Sex Detective Book Review and Summary](/blog/the-sex-detective-anna-natasha) - [**Judge's List** Book Review and Summary](/blog/judge-s-list) - [Love on the Brain Book Review and Summary](/blog/love-on-the-brain)

We use cookies (including third-party cookies) to provide you with the best possible experience and to serve personalized ads. By clicking "Accept", you consent to the use of all cookies. Visit our Privacy Policy for more information.