Showing Posts From

Large action models

Navigating the Browser Jungle with LAMs The web has long been a sandbox for human curiosity, but the next frontier is letting machines roam it autonomously. Large Action Models (LAMs) extend the language‑model paradigm from pure text generation to action generation: they emit structured commands—clicks, form fills, JavaScript snippets—that a browser runtime can execute directly. In contrast to classic headless‑browser bots that follow hard‑coded scripts, LAMs reason about the state of the Document Object Model (DOM), plan multi‑step navigation, and adapt on the fly when pages diverge from expectations. Why does this matter now? The 2024 “Can Language Models Play the Wikipedia Game?” study demonstrated that state‑of‑the‑art LLMs (GPT‑4o, DeepSeek‑R1, QwQ:32B) can navigate massive hyperlink graphs with success rates comparable to humans, yet they still stumble on edge‑case link structures and multilingual quirks. Those same navigation capabilities translate directly to browser environments: a LAM must decide which hyperlink or button best advances its goal, just as a human player does in the Wikipedia Game. Simultaneously, the “Towards Physically Grounded Articulated 3D Assets” paper introduced multi‑agent pipelines that close the loop between generation and physical simulation. The underlying lesson is that feedback loops—where an agent’s output is validated by a simulator—are essential for robust autonomy. In the browser world the “simulator” is the live page, and the feedback is the DOM after each action. In this article, I will dissect the technical stack that makes LAM‑driven browser automation possible, illustrate it with production‑grade code, compare the leading frameworks, and outline safety practices that keep autonomous agents from becoming runaway click‑farms. By the end, you’ll have a reproducible blueprint to spin up a fleet of LAM agents that can scrape, fill, and even test web applications without a human in the loop.From Links to Logic: How LAMs Interpret the DOM A LAM’s first job is to understand the current page. Unlike static crawlers that rely on TF‑IDF or simple XPath queries, LAMs ingest the raw HTML, render it (often via a headless Chrome instance), and feed a structured snapshot of the DOM into a transformer. The snapshot includes:Tag hierarchy (parent‑child relationships) Visible text extracted via the Accessibility Tree Computed CSS properties (e.g., display: none to filter hidden elements) Event listeners attached to nodes (click, submit, change)The model then produces a probability distribution over possible actions, each expressed as a JSON‑compatible “action schema”: { "action_type": "click", "selector": "#login-button", "reasoning": "Button labeled 'Login' is the only visible CTA." }Embedding the DOM as a sequence of tokens is non‑trivial. Recent work (e.g., the OpenAI “WebGPT” approach) uses a flattened DOM linearization where each node becomes a token like <div id="main">. This representation preserves hierarchical cues while staying within the token limits of even 70‑B models.The real breakthrough comes from retrieval‑augmented prompting: the LAM queries an external knowledge base (e.g., a citation graph from OpenAlex) to resolve domain‑specific terminology before deciding which link to follow. In the Wikipedia Game experiments, models that combined sentence embeddings with external retrieval outperformed pure‑LLM baselines, echoing the same pattern for browser tasks where domain‑specific forms (tax filing, medical portals) require contextual grounding.Agentic Prompting and Tool Use: The Browser as a First‑Class API Traditional LLM agents treat browsers as opaque tools: “open URL X, then click the button labeled Y.” LAMs elevate the browser to a first‑class API by exposing a rich set of primitives:Primitive Signature Examplenavigate(url: str) Opens a new page navigate("https://news.ycombinator.com")click(selector: str) Clicks an element click("#morelink")type(selector: str, text: str) Sends keystrokes type("#search", "LAMs")evaluate(js: str) Runs arbitrary JavaScript evaluate("document.title")screenshot(path: str) Captures the viewport screenshot("/tmp/page.png")These primitives are wrapped in a JSON‑RPC layer that the LAM calls over a local WebSocket. The agent can also request tool extensions, such as a headless PDF renderer or a CAPTCHA solver, and the LAM will incorporate the tool’s output into its reasoning chain. The prompting template that yields the best performance mirrors the “chain‑of‑thought” style that won the 2023 LAMBADA competition: You are a Large Action Model controlling a browser. The current page title is "{{title}}". Your goal: "{{goal}}". Consider the DOM, any visible text, and available actions. Respond with a JSON action.Because the model receives the entire action schema in its context, it learns to reason about why a particular selector is optimal—a behavior observed in the Wikipedia Game where the best models articulated their link‑choice rationale before clicking.Safety Nets: Sandboxing, State Persistence, and Rollback Strategies Unsupervised browsing is a high‑risk activity. A LAM could inadvertently trigger malicious downloads, submit harmful data, or cause denial‑of‑service on the target site. The community has converged on three safety pillars:Containerized Sandboxing – Each LAM instance runs inside a lightweight Docker container with a restricted network namespace. Outbound traffic is filtered through a proxy that blocks known malware hosts and limits request rates.State Versioning – After every action, the browser’s snapshot (HTML, cookies, localStorage) is serialized to a Redis‑backed version store. If the LAM deviates from its goal (e.g., lands on a login page when it should stay on a product page), a supervisory controller can rollback to the previous safe state and re‑prompt the model with corrective instructions.Policy‑Driven Action Filters – Before the LAM’s action is executed, a deterministic policy engine validates the selector against a whitelist (e.g., only elements with role="button" are clickable). This prevents accidental clicks on hidden tracking pixels or malicious ads.Below is a minimal Python watchdog that enforces these rules for a Selenium‑driven LAM session: import json, redis, selenium.webdriver as webdriverr = redis.StrictRedis(host="redis", port=6379, db=0)def save_snapshot(driver, step): state = { "url": driver.current_url, "html": driver.page_source, "cookies": driver.get_cookies() } r.set(f"lam:snapshot:{step}", json.dumps(state))def is_allowed_action(action): # Simple whitelist: only click buttons or links with visible text if action["action_type"] not in {"click", "type", "navigate"}: return False if action["action_type"] == "click": return bool(action["selector"]) return Truedef execute_action(driver, action): if not is_allowed_action(action): raise PermissionError("Disallowed action attempted") if action["action_type"] == "navigate": driver.get(action["selector"]) elif action["action_type"] == "click": driver.find_element_by_css_selector(action["selector"]).click() elif action["action_type"] == "type": el = driver.find_element_by_css_selector(action["selector"]) el.clear() el.send_keys(action["text"]) save_snapshot(driver, action.get("step", 0))By coupling this watchdog with a policy‑as‑code file (see next section), teams can audit every click the LAM makes, satisfying both security auditors and ethical AI guidelines.Benchmarking Autonomy: Lessons from the Wikipedia Game and 3D Asset Pipelines The two arXiv studies provide a natural yardstick for LAM performance:Wikipedia Game – Success is measured by reaching the target article within a fixed number of clicks. Human baselines hover around 78 % success; the best LLMs (GPT‑4o, DeepSeek‑R1) achieve ~73 % after fine‑tuning. The key metric is click efficiency—the ratio of clicks taken to the optimal shortest path.Physically Grounded 3D Assets – Here, the benchmark evaluates physical plausibility (collision‑free motion) and functional fidelity. The Artisan pipeline uses multi‑agent feedback loops where a physics engine validates each generated articulation before acceptance.Translating these concepts to browsers, we define two complementary benchmarks:Benchmark Goal Success MetricLink‑Reach Navigate from a start URL to a target page using only visible UI elements. % of runs reaching target within N actions.Form‑Fill Accuracy Complete a multi‑step web form (e.g., tax filing) without validation errors. % of fields correctly populated, measured against a ground‑truth schema.Stability Score Maintain a consistent DOM state across repeated runs (idempotence). Collision‑rate analogue: number of unexpected redirects or error pages per 100 runs.In early experiments, LAMs that incorporated retrieval‑augmented prompting (pulling relevant API docs from OpenAI’s embeddings) reduced the average click count by 12 % on the Link‑Reach benchmark. Adding a physics‑style feedback loop—where the watchdog checks for DOM “collisions” (e.g., duplicate element IDs)—improved stability from 85 % to 94 %. These findings reinforce a broader principle: iterative validation—whether via a physics engine or a DOM watchdog—dramatically lifts autonomous performance.Scaling Up: Distributed LAM Orchestration with Docker & Kubernetes When moving from a single proof‑of‑concept to production, you need a robust orchestration layer. The pattern that has emerged in the last year is to treat each LAM instance as a microservice exposing a JSON‑RPC endpoint, and to manage them with Kubernetes. Below is a concise docker‑compose.yml that spins up three components: version: "3.9" services: redis: image: redis:7-alpine ports: ["6379:6379"] restart: unless-stopped lam-worker: image: ghcr.io/yourorg/lam-browser:latest environment: - REDIS_HOST=redis - SELENIUM_HOST=selenium depends_on: [redis, selenium] deploy: replicas: 3 resources: limits: memory: "2G" cpus: "1.0" restart: unless-stopped selenium: image: selenium/standalone-chrome:latest shm_size: "2g" ports: ["4444:4444"] environment: - SE_NODE_MAX_SESSIONS=5 - SE_SESSION_REQUEST_TIMEOUT=300 restart: unless-stoppedIn a Kubernetes setting, each lam-worker would be a Deployment with a Horizontal Pod Autoscaler (HPA) keyed to the length of the Redis task queue. A CustomResourceDefinition (CRD) called LamTask can encapsulate the goal description, start URL, and timeout. The controller watches for new LamTask objects, dispatches them to an available worker, and records the final state back into a PostgreSQL store for analytics. By decoupling the browser runtime (Selenium) from the model inference (served via an OpenAI‑compatible endpoint), you can horizontally scale each tier independently—critical for high‑throughput scraping or large‑scale UI testing pipelines.Putting It All Together: A Minimal End‑to‑End Pipeline Below is a compact Python orchestration script that ties the pieces: it fetches a task from Redis, sends it to the LAM RPC endpoint, validates the returned action with the watchdog, and loops until the goal is met or a timeout occurs. import json, time, redis, requestsREDIS = redis.StrictRedis(host="redis", port=6379, db=0) LAM_ENDPOINT = "http://lam-worker:8000/rpc"def fetch_task(): raw = REDIS.lpop("lam:task_queue") return json.loads(raw) if raw else Nonedef call_lam(payload): resp = requests.post(LAM_ENDPOINT, json=payload, timeout=30) resp.raise_for_status() return resp.json()def main(): while True: task = fetch_task() if not task: time.sleep(2) continue state = {"url": task["start_url"], "goal": task["goal"], "step": 0} while state["step"] < task.get("max_steps", 20): lam_input = { "dom_snapshot": state.get("dom"), "goal": task["goal"], "history": state.get("history", []) } action = call_lam(lam_input) try: execute_action(driver, action) # from watchdog example except PermissionError as e: print("Policy violation:", e) break state["history"].append(action) state["step"]### Related Insights- [The Sex Detective Book Review and Summary](/blog/the-sex-detective-anna-natasha) - [**Judge's List** Book Review and Summary](/blog/judge-s-list) - [Love on the Brain Book Review and Summary](/blog/love-on-the-brain)

The Paradigm Shift: When Models Learn to Act The last decade taught us that scaling language models yields ever‑more fluent text generators. Yet the most valuable commodity in modern software is not prose but action—the ability to mutate state, invoke APIs, and close the loop between intent and effect. Large Action Models (LAMs) embody this transition: they consume high‑level instructions, plan multi‑step procedures, and execute them on heterogeneous environments ranging from pixel‑level image editors to multi‑embodiment robot fleets. Two recent arXiv contributions illustrate the frontier. Rubric‑CEPR demonstrates a self‑evolving image editor that verifies its own output through a rubric‑augmented Contrastive Edit‑Preservation Reward, improving ImgEdit scores from 4.36 to 4.60 (+5.5 %) and delivering a 24.9 % gain on object isolation without any human‑annotated pairs. DreamTrue, on the other hand, builds a cross‑embodiment robot world model that learns from counterfactual post‑training, slashing human‑assessed interaction defects from 48.12 % to 6.25 % and securing first place in the AgiBot World Challenge 2026. Together they signal a new contract: models that self‑audit and imagine unseen futures, turning passive prediction into reliable agency. The implications ripple across developer tooling, autonomous manufacturing, and the very definition of a user interface.Rubric‑Driven Self‑Distillation in Visual Editing Rubric‑CEPR replaces the traditional supervised fine‑tuning loop with a critic‑in‑the‑loop that never leaves the model’s own latent space. A Planner samples structured edit instructions from raw images; the Editor produces a candidate set; a frozen Critic evaluates each candidate against three non‑compensatory rubrics—edit realization, removal of the prior state, and content preservation—using features already exposed by the editor backbone. Only candidates passing all gates survive to a lightweight adapter distillation step. This architecture yields two practical advantages for system builders. First, it eliminates the costly human‑in‑the‑loop annotation pipeline that has bottlenecked diffusion‑based editors. Second, the rubric gates act as hard constraints, preventing the subtle “plausible failure” modes where an edit looks realistic but leaves the target object untouched. Empirically, the method transfers to GEdit‑Bench and Complex‑Edit, and a parallel run on Step1X‑Edit adds a further +7.8 % on ImgEdit. For engineering teams, the takeaway is clear: a verifiable self‑critique loop can be dropped into any conditional diffusion backbone with only a few adapter layers, turning a static model into a continuously improving agent. Counterfactual World Models for Embodied Action DreamTrue tackles the orthogonal problem of action fidelity in video prediction for robotics. Training on logged trajectories suffers from two biases: calibration noise that misaligns action conditioning, and a surplus of successful episodes that teaches the model to hallucinate success. DreamTrue’s remedy is twofold. Offline geometric calibration projects recorded joint commands into image‑space conditions, aligning them with the visual stream. Counterfactual post‑training then rewrites action trajectories—injecting slips, collisions, and missed grasps—and rolls out the resulting futures with the world model. Because ground‑truth futures for these synthetic branches do not exist, the authors collect a modest human‑annotated defect dataset (robot, object, interaction flaws) and train an embodied video reward model. Reinforcement‑learning post‑training guided by this reward pushes the predictor toward physically plausible outcomes. The result: state‑of‑the‑art action following on AgiBot and a defect rate reduction from 48.12 % to 6.25 %. For practitioners deploying fleets of heterogeneous manipulators, DreamTrue offers a blueprint: calibrate once, augment aggressively, and close the loop with a learned visual critic.Architectural Patterns Emerging from LAM Research Both papers converge on a planner‑executor‑critic triad that mirrors classic control theory but lives entirely in latent neural space. The planner proposes discrete symbolic or semi‑symbolic goals; the executor (editor or world model) generates continuous candidates; the critic evaluates them against a decomposed rubric or reward. This pattern is now visible in several open‑source projects that have trended on GitHub: AutoGPT‑style agents that spin up a “self‑critique” loop before committing code changes, LangChain’s SelfCritiqueChain that validates tool outputs, and the OpenInterpreter sandbox that rolls back unsafe filesystem mutations. The common infrastructure pieces are:Frozen critic – a lightweight head (often a CLIP‑style encoder) that can score without gradient updates. Non‑compensatory gating – hard thresholds on each rubric dimension, preventing a high score in one dimension from masking failure in another. Adapter‑level distillation – only a few million parameters are updated per self‑improvement cycle, keeping compute budgets modest.These primitives enable a continuous deployment mindset: every inference request can optionally trigger a self‑verification pass, and the resulting high‑confidence samples are harvested for nightly adapter fine‑tunes. The operational cost is comparable to a single forward pass plus a cheap scoring head, making LAMs viable for latency‑sensitive products such as real‑time design assistants or tele‑operated warehouse robots. # Minimal self‑distillation loop inspired by Rubric‑CEPR import torch from torch import nn from transformers import AutoModelForCausalLM, AutoTokenizerclass RubricCritic(nn.Module): """Frozen critic scoring edit realization, removal, preservation.""" def __init__(self, encoder): super().__init__() self.encoder = encoder.eval() # frozen backbone self.head = nn.Linear(encoder.config.hidden_size, 3) # three rubrics @torch.no_grad() def score(self, latent): return torch.sigmoid(self.head(self.encoder(latent).last_hidden_state[:,0]))def self_distill(editor, critic, planner, data_loader, steps=5): adapter = nn.Linear(editor.config.hidden_size, editor.config.hidden_size).to(editor.device) opt = torch.optim.AdamW(adapter.parameters(), lr=1e-4) for _ in range(steps): for batch in data_loader: instr = planner(batch['image']) cand = editor.generate(instr, num_return_sequences=4) scores = critic.score(cand.latents) # (B,4,3) gate = (scores > 0.8).all(dim=-1) # non‑compensatory best = cand[gate].argmax(dim=1) # highest aggregate loss = nn.functional.mse_loss(adapter(best.latents), best.latents.detach()) loss.backward()### Related Insights- [All the Light We Cannot See Book Review and Summary](/blog/all-the-light-we-cannot-see) - [The Maze Runner Book Review and Summary](/blog/the-maze-runner) - [The Rosetta Stone: Breaking the Silence of Ancient Egypt](/blog/mark-dinn-rosetta-stone)

We use cookies (including third-party cookies) to provide you with the best possible experience and to serve personalized ads. By clicking "Accept", you consent to the use of all cookies. Visit our Privacy Policy for more information.