LightNode Logo

How to Control a Phone with AI: A Complete AutoGLM Agent + Mobile Execution Guide

LightNode
By LightNode ·

“Hand-poking a phone” does not mean physically tapping a screen.
In engineering terms, it means allowing an AI Agent to understand a mobile UI and execute actions such as tapping, swiping, and typing on its own.

AutoGLM is one of the very few open-source AI Agent frameworks designed specifically for mobile control scenarios.
Instead of relying on fixed scripts or hard-coded rules, it enables AI to see the screen, reason about the current state, and act like a human user.

This tutorial is written for an official tech documentation page.
It focuses on real implementation details, reproducibility, and deployment best practices.

What Does “AI Hand-Poking a Phone” Mean?

In one sentence:

Hand-poking a phone = AI Agent + real mobile execution

The key difference from traditional automation is that:

  • You do not write fixed scripts
  • You do not rely on static coordinates
  • The AI first understands the current screen, then decides what to do next

AutoGLM’s core philosophy is simple:

Let AI operate phones like a human, not like a rule engine.

How AutoGLM Works Internally

AutoGLM runs as a continuous perception–decision–action loop:

Task Goal
   ↓
AI Agent (Understanding & Reasoning)
   ↓
Screen Capture
   ↓
State Evaluation
   ↓
Action Generation
   ↓
ADB Execution on Android
   ↓
UI Changes → Loop Continues

The loop stops only when the task is completed or a predefined limit is reached.

System Architecture Overview

A production-ready AutoGLM mobile agent typically includes:

Vision + Reasoning Model
        ↓
AutoGLM Agent (Decision Core)
        ↓
Screen Capture Module
        ↓
Action Instructions (Tap / Swipe / Input)
        ↓
ADB Execution Layer
        ↓
Android Device (Emulator or Real Phone)

In this setup:

  • The Agent is the brain

  • ADB is the hand

  • The phone is the real execution environment

Environment Requirements

Hardware / Infrastructure

  • A VPS or local Linux/macOS machine

  • Android Emulator or physical Android device

  • Stable internet connection

💡 In real-world deployments, running AutoGLM on a LightNode VPS is highly recommended, as it provides a stable, always-on environment well suited for long-running AI Agent tasks.

In practice, the VPS + Android Emulator combination offers:

24/7 availability

Higher stability

Easy horizontal scaling

Software Dependencies

  • Python 3.10+

  • Git

  • Android Debug Bridge (ADB)

  • Android Emulator (Android Studio Emulator recommended)

  • GPU is optional (CPU-only mode works)

Step 1: Prepare the Android Device and ADB

Install ADB

sudo apt update
sudo apt install adb

Verify the connection:

adb devices

Expected output:

emulator-5554    device

This confirms that AutoGLM can interact with the device.

Configure the Android Emulator

Recommended settings:

  • Fixed screen resolution (e.g., 1080×1920)

  • Auto-rotation disabled

  • Developer mode enabled

These settings significantly improve Agent stability.

Step 2: Install AutoGLM

Clone the official repository:

git clone https://github.com/zai-org/Open-AutoGLM.git
cd Open-AutoGLM

Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Step 3: Configure the AutoGLM Agent

AutoGLM uses configuration files to control Agent behavior.

Example configuration:

task:
  goal: "Open the app and navigate to the main page"
  max_steps: 30

device:
  type: android
  adb_path: adb

agent:
  model: vision-language
  temperature: 0.2

Key points:

  • goal must be clear and verifiable

  • max_steps prevents infinite loops

  • Lower temperature improves decision consistency

How AutoGLM Understands the Screen

Step 1: Capture the Screen

adb exec-out screencap -p > screen.png

This screenshot serves as the Agent’s visual input.

Step 2: Vision + Decision Prompt

A typical structured prompt looks like this:

You are an AI Agent controlling an Android phone.

Analyze the current screen and decide the next best action
to achieve the task goal.

Respond strictly in the following JSON format:
{
  "action": "tap | swipe | input | wait",
  "x": number,
  "y": number,
  "text": "",
  "reason": ""
}

⚠️ Strict output formatting is critical for reliable execution.

Step 4: Real Mobile Execution (The Core of “Hand-Poking”)

AutoGLM translates AI decisions into ADB commands.

Tap

adb shell input tap 520 1680

Swipe

adb shell input swipe 500 1600 500 400

Text Input

adb shell input text "hello"

This is the moment where AI decisions become real physical actions on the phone.

The Core AutoGLM Agent Loop

During execution, the Agent continuously runs:

Screenshot
→ Screen Understanding
→ Action Decision
→ Execution
→ State Validation
→ Next Iteration

When issues occur (pop-ups, delays, unchanged screens), the Agent may:

  • Wait

  • Retry

  • Choose an alternative action

This adaptability is what makes Agents more robust than scripts.

Minimal Execution Logic (Conceptual)

while not task_finished:
    screenshot()
    action = agent.decide(screen)
    execute(action)
    validate_state()

The Agent stops when:

  • The task is completed

  • The maximum step count is reached

  • A critical error occurs

Common Use Cases

  • Mobile app testing and QA

  • AI-powered phone assistants

  • Internal workflow automation

  • Operating systems without APIs

  • Embodied AI research

Production Best Practices

  • Always enforce a maximum step limit

  • Log screenshots and actions

  • Normalize screen resolution

  • Validate workflows on emulators before real devices

  • Follow legal, privacy, and platform policies

FAQ

How is AutoGLM different from traditional UI automation tools?

Traditional tools rely on fixed rules and coordinates, while AutoGLM relies on visual understanding and reasoning.

Can AutoGLM run on a VPS?

Yes. Running AutoGLM on a VPS with Android emulators is a common and stable setup.

Is a GPU required?

No. CPU-only execution works well; GPUs mainly improve speed.

Can AutoGLM control real phones?

Yes. Any Android device accessible via ADB can be controlled.

Is AutoGLM suitable for commercial use?

Yes, as long as the use case complies with laws, privacy regulations, and platform policies.

Conclusion

AutoGLM moves AI beyond conversation and into real-world mobile execution.

By combining:

  • AI Agents

  • Visual understanding

  • Real phone control

you can deploy AI systems that operate directly inside mobile apps — even when no APIs exist.

This represents a major step forward for embodied AI in production environments.

Back to the list