Based on The Intelligent OS: Making AI agents more helpful for Android apps(February 25, 2026). This report provides a deep-dive source code analysis of the two core AOSP frameworks behind Android’s AI agent capabilities.

Table of Contents

I. The Big Picture

1. The Paradigm Shift
2. Android’s Dual-Path Agent Architecture

II. AppFunctions – Structured Intelligence

1. AppFunctions Overview
2. AppFunctions Core API
3. AppFunctions Server-Side Implementation
4. AppFunctions Discovery System
5. AppFunctions Access Control
6. AppFunctions Feature Flags

III. ComputerControl – Universal UI Automation

1. ComputerControl Overview
2. ComputerControl Architecture
3. Session Lifecycle
4. Input Handling
5. Display Mirroring and Live View
6. Stability Tracking
7. IME and Text Input Integration
8. Automated Package Tracking
9. ComputerControl Security Model
10. ComputerControl Feature Flags

IV. Comparative Analysis

1. AppFunctions vs ComputerControl
2. Security Model Comparison
3. The Incremental Adoption Strategy

V. Reference

1. Key Source File Reference

I. The Big Picture

1. The Paradigm Shift

The Android Developers Blog frames a fundamental transformation in mobile computing:

“User expectations for AI on their devices are fundamentally shifting how they interact with their apps.”

The traditional model – users manually opening apps, navigating UIs, and completing tasks step by step – is evolving toward task delegation. Users tell an AI agent what they want (“order my usual pizza”, “coordinate a rideshare with coworkers”), and the agent handles the multi-step workflow across multiple apps.

This shift reframes success metrics from “app opens” to “task completion.” Android’s response is to build native OS-level support for AI agents through two complementary frameworks:

• AppFunctions – Structured, API-based integration where apps explicitly expose capabilities
• ComputerControl – Universal UI automation that works with any app, unmodified

flowchart TD
    subgraph "Traditional Model"
        U1[User] --> A1[Open App A]
        A1 --> A2[Navigate UI]
        A2 --> A3[Perform Action]
        A3 --> A4[Open App B]
        A4 --> A5[Navigate UI]
        A5 --> A6[Perform Action]
    end

    subgraph "Intelligent OS Model"
        U2[User] --> AI[AI Agent<br/>e.g. Gemini]
        AI --> AF[AppFunctions<br/>Structured Path]
        AI --> CC[ComputerControl<br/>UI Automation Path]
        AF --> T1[Task Completed<br/>in App A]
        CC --> T2[Task Completed<br/>in App B]
    end

2. Android’s Dual-Path Agent Architecture

Android provides AI agents with two independent, complementary pathways to interact with apps. They share no code-level integration (zero cross-references in the codebase) but serve the same higher-level goal.

graph TB
    Agent["AI Agent<br/>(e.g. Gemini)"]

    subgraph AppFunctions["AppFunctions (Structured Path)"]
        AF_API["AppFunctionManager<br/>executeAppFunction()"]
        AF_SVC["AppFunctionManagerService<br/>(System Server)"]
        AF_APP["Target App's<br/>AppFunctionService"]
        AF_API --> AF_SVC
        AF_SVC --> AF_APP
    end

    subgraph ComputerControl["ComputerControl (UI Automation Path)"]
        CC_API["ComputerControlExtensions<br/>requestSession()"]
        CC_SVC["VirtualDeviceManagerService<br/>(System Server)"]
        CC_VD["VirtualDevice<br/>+ Virtual Display<br/>+ Input Devices"]
        CC_API --> CC_SVC
        CC_SVC --> CC_VD
    end

    Agent --> AppFunctions
    Agent --> ComputerControl

    style AppFunctions fill:#e8f5e9
    style ComputerControl fill:#e3f2fd

The blog describes this as:

• AppFunctions: “Allows apps to expose data and functionality directly to AI agents and assistants” – self-describing functions that agents discover and execute, like an on-device MCP.
• ComputerControl (the blog’s “UI automation framework”): “The platform doing the heavy lifting, so developers can get agentic reach with zero code” – generic task execution on any installed app with user transparency built in.

II. AppFunctions – Structured Intelligence

3. AppFunctions Overview

The blog introduces AppFunctions with a real-world example: on the Galaxy S26, a user asks Gemini “Show me pictures of my cat from Samsung Gallery.” The AI identifies the appropriate AppFunction, retrieves photos, and returns results within the Gemini interface – without the user ever opening the Gallery app.

AppFunctions (android.app.appfunctions) enables apps to expose discrete, self-describing functions that AI agents can discover via AppSearch and execute programmatically. It is architecturally similar to WebMCP (Model Context Protocol) but runs entirely on-device.

sequenceDiagram
    participant User
    participant Agent as AI Agent (Gemini)
    participant AFM as AppFunctionManager
    participant Server as System Server<br/>(AppFunctionManagerServiceImpl)
    participant Target as Target App<br/>(AppFunctionService)

    User->>Agent: "Show me cat photos from Gallery"
    Agent->>Agent: Identify function via AppSearch
    Agent->>AFM: executeAppFunction(request)
    AFM->>Server: IAppFunctionManager.executeAppFunction()
    Server->>Server: Validate caller, permissions,<br/>access grants, enterprise policy,<br/>function enabled state
    Server->>Target: Bind & call IAppFunctionService.executeAppFunction()
    Target->>Target: onExecuteFunction()
    Target-->>Server: ExecuteAppFunctionResponse<br/>(+ optional URI grants)
    Server-->>AFM: Response callback
    AFM-->>Agent: Result (GenericDocument)
    Agent-->>User: Display cat photos

4. AppFunctions Core API

Entry Point: AppFunctionManager

The main client-facing API, registered as Context.APP_FUNCTION_SERVICE:

How Apps Expose Functions: AppFunctionService

Apps extend this abstract service to expose their functions:

classDiagram
    class AppFunctionService {
        <<abstract>>
        +onExecuteFunction(request, callingPackage, signingInfo, signal, receiver)*
        +createBinder(context, onExecuteFunction)$ Binder
    }

    class ExecuteAppFunctionRequest {
        +getTargetPackageName() String
        +getFunctionIdentifier() String
        +getParameters() GenericDocument
        +getExtras() Bundle
        +getAttribution() AppFunctionAttribution
    }

    class ExecuteAppFunctionResponse {
        +getResultDocument() GenericDocument
        +getExtras() Bundle
        +getUriGrants() List~AppFunctionUriGrant~
    }

    class AppFunctionException {
        +ERROR_DENIED = 1000
        +ERROR_INVALID_ARGUMENT = 1001
        +ERROR_DISABLED = 1002
        +ERROR_FUNCTION_NOT_FOUND = 1003
        +ERROR_SYSTEM_ERROR = 2000
        +ERROR_CANCELLED = 2001
        +ERROR_ENTERPRISE_POLICY_DISALLOWED = 2002
        +ERROR_APP_UNKNOWN_ERROR = 3000
    }

    class AppFunctionAttribution {
        +getInteractionType() int
        +getThreadId() String
        +getInteractionUri() Uri
    }

    AppFunctionService ..> ExecuteAppFunctionRequest : receives
    AppFunctionService ..> ExecuteAppFunctionResponse : returns
    AppFunctionService ..> AppFunctionException : throws
    ExecuteAppFunctionRequest --> AppFunctionAttribution : contains

Manifest declaration (required by target apps):

<service android:name=".MyAppFunctionService"
         android:permission="android.permission.BIND_APP_FUNCTION_SERVICE">
    <intent-filter>
        <action android:name="android.app.appfunctions.AppFunctionService"/>
    </intent-filter>
</service>

AIDL Interfaces

5. AppFunctions Server-Side Implementation

Core Service Architecture

graph TB
    subgraph SystemServer["System Server"]
        AFS["AppFunctionManagerService<br/>(SystemService wrapper)"]
        AFSI["AppFunctionManagerServiceImpl<br/>(~1220 lines, extends IAppFunctionManager.Stub)"]
        CV["CallerValidatorImpl"]
        RSC["RemoteServiceCallerImpl"]
        MSA["MetadataSyncAdapter"]
        AALS["AppFunctionAgentAllowlistStorage"]
        AH["MultiUserAppFunctionAccessHistory"]
        PM["AppFunctionPackageMonitor"]

        AFS --> AFSI
        AFSI --> CV
        AFSI --> RSC
        AFSI --> MSA
        AFSI --> AALS
        AFSI --> AH
        AFSI --> PM
    end

    subgraph Permission["Permission Service"]
        AAS["AppFunctionAccessService"]
        AAP["AppIdAppFunctionAccessPolicy"]
        APG["AppFunctionAccessPregrant"]

        AAS --> AAP
        AAS --> APG
    end

    AFSI --> AAS

Execution Pipeline

The executeAppFunction() method in AppFunctionManagerServiceImpl follows this pipeline:

flowchart TD
    A[executeAppFunction called] --> B[Validate caller identity<br/>UID matches claimed package]
    B --> C[Verify target user handle<br/>cross-user permissions]
    C --> D[Submit to THREAD_POOL_EXECUTOR]
    D --> E[Check enterprise policy<br/>DevicePolicyManager.getAppFunctionsPolicy]
    E --> F{Enterprise<br/>allowed?}
    F -->|No| G[ERROR_ENTERPRISE_POLICY_DISALLOWED]
    F -->|Yes| H[Verify execution permission<br/>EXECUTE_APP_FUNCTIONS + access grants]
    H --> I{Permission<br/>granted?}
    I -->|No| J[ERROR_DENIED]
    I -->|Yes| K[Check function enabled<br/>via AppSearch runtime metadata]
    K --> L{Enabled?}
    L -->|No| M[ERROR_DISABLED]
    L -->|Yes| N[Resolve AppFunctionService<br/>in target package]
    N --> O{Service<br/>found?}
    O -->|No| P[ERROR_SYSTEM_ERROR]
    O -->|Yes| Q[Grant implicit visibility<br/>target sees caller package]
    Q --> R[Bind to target service<br/>5s cancellation timeout]
    R --> S[Call IAppFunctionService.executeAppFunction]
    S --> T[Return response to caller]
    T --> U[Record access history<br/>in SQLite audit trail]

Remote Service Calling

RemoteServiceCallerImpl manages the one-off service connection:

• Binds to target’s AppFunctionService using bindServiceAsUser()
• Sets up cancellation signal with configurable timeout (default 5 seconds)
• Links to caller’s binder death for auto-cancel if caller dies
• Unbinds after completion, cancellation timeout, or service disconnect

6. AppFunctions Discovery System

Functions are discovered through a dual-metadata system stored in AppSearch:

graph LR
    subgraph AppsDB["AppSearch: apps-db (Static)"]
        SM["AppFunctionStaticMetadata<br/>- functionId<br/>- packageName<br/>- parameter schema<br/>- return type schema<br/>- enabledByDefault"]
    end

    subgraph FunctionsDB["AppSearch: appfunctions-db (Runtime)"]
        RM["AppFunctionRuntimeMetadata<br/>- functionId<br/>- packageName<br/>- enabled state override"]
    end

    SM ---|"Joined via<br/>qualifiedId"| RM

    PackageInstall["Package Install/Update"] -->|"System indexes"| SM
    StateChange["setAppFunctionEnabled()"] -->|"System writes"| RM

    Agent["AI Agent"] -->|"GlobalSearchSession<br/>query"| SM
    Agent -->|"Join query"| RM

Metadata synchronization (MetadataSyncAdapter):

1. Triggered on user unlock and AppSearch observer changes
2. Queries both static and runtime AppSearch databases
3. Computes diff (added/removed functions)
4. Creates runtime metadata for new functions, removes for deleted ones
5. Sets AppSearch schema visibility per-package using signing certificates and EXECUTE_APP_FUNCTIONS permission

Effective enabled state: Runtime override takes precedence over static enabledByDefault.

7. AppFunctions Access Control

Three-Layer Agent Allowlisting

graph TD
    subgraph Layer1["Layer 1: System Hardcoded"]
        SH["com.android.shell"]
    end

    subgraph Layer2["Layer 2: DeviceConfig (Updatable)"]
        DC["machine_learning namespace<br/>allowlisted_app_functions_agents<br/>Format: packageName:certHex"]
    end

    subgraph Layer3["Layer 3: ADB Secure Setting"]
        ADB["Settings.Secure<br/>APP_FUNCTION_ADDITIONAL_AGENT_ALLOWLIST"]
    end

    Layer1 --> Merge["Merged AgentAllowlist<br/>(ArraySet of SignedPackage)"]
    Layer2 --> Merge
    Layer3 --> Merge

    Merge --> Policy["AppIdAppFunctionAccessPolicy"]

    Storage["AppFunctionAgentAllowlistStorage<br/>/data/system/appfunctions/agent_allowlist.txt<br/>(AtomicFile, crash-safe)"]
    DC -.->|"Persisted to"| Storage

Access Flag Bitmask System

Per agent-target pair, access is tracked via bitmask flags:

Priority: USER flags override OTHER flags. If no DENIED flags present, PREGRANTED means granted.

Audit Trail

Every execution is recorded in a per-user SQLite database (appfunction_access.db):

Cleanup: Scheduled every 24 hours, retaining 7 days of history.

8. AppFunctions Feature Flags

III. ComputerControl – Universal UI Automation

9. ComputerControl Overview

The blog introduces the second pathway:

“A UI automation framework for AI agents and assistants to intelligently execute generic tasks on users’ installed apps, with user transparency and control built in.”

“The platform doing the heavy lifting, so developers can get agentic reach with zero code.”

ComputerControl (android.companion.virtual.computercontrol) is a framework-level feature that enables programmatic automation of Android applications running on a trusted virtual display. It is built on the VirtualDeviceManager (VDM) infrastructure and allows an authorized agent to:

• Launch applications on an isolated virtual display
• Inject input (taps, swipes, long presses, key events, text)
• Capture screenshots of the display content
• Mirror the display for user “live view” with interactive touch forwarding
• Monitor UI stability (detect when the display has settled)
• Hand over automated apps back to the user’s default display

The blog highlights key user-facing aspects:

• Users monitor progress via “notifications or ‘live view’“ (implemented by MirrorView + InteractiveMirrorDisplay)
• Users “can switch to manual control at any point” (implemented by handOverApplications() and MirrorView.setInteractive(true))
• Alerts precede “sensitive tasks, such as making a purchase”
• Currently in early preview on Galaxy S26 series and select Pixel 10 devices

10. ComputerControl Architecture

Layered Architecture

graph TB
    subgraph Client["Client Application (Agent)"]
        CCE["ComputerControlExtensions<br/>(Entry point / Factory)"]
        CCS_EXT["ComputerControlSession<br/>(Extension SDK)"]
        MV["MirrorView<br/>(TextureView + Overlay)"]
        IM["InteractiveMirror"]
    end

    subgraph PlatformAPI["Core Platform API"]
        VDM["VirtualDeviceManager<br/>requestComputerControlSession()"]
        CCS["ComputerControlSession"]
        CCSP["ComputerControlSessionParams"]
        IMD["InteractiveMirrorDisplay"]
    end

    subgraph SystemServer["System Server"]
        VDMS["VirtualDeviceManagerService"]
        CCSP_P["ComputerControlSessionProcessor<br/>(consent, limits, validation)"]
        CCSI["ComputerControlSessionImpl<br/>(777 lines, core binder)"]
        SC["StabilityCalculator"]
        IMDI["InteractiveMirrorDisplayImpl"]
        APR["AutomatedPackagesRepository"]
    end

    subgraph VirtualDevice["Virtual Device"]
        VD["VirtualDevice"]
        TD["Trusted VirtualDisplay<br/>(animations disabled, IME hidden)"]
        TS["VirtualTouchscreen<br/>(0xCC03)"]
        KB["VirtualKeyboard<br/>(0xCC02)"]
        DP["VirtualDpad<br/>(0xCC01)"]
    end

    CCE --> VDM
    CCS_EXT --> CCS
    MV --> IM
    IM --> IMD

    VDM -->|"Binder IPC"| VDMS
    VDMS --> CCSP_P
    CCSP_P -->|"creates"| CCSI
    CCSI --> SC
    CCSI --> IMDI
    VDMS --> APR

    CCSI --> VD
    VD --> TD
    VD --> TS
    VD --> KB
    VD --> DP

    style Client fill:#e3f2fd
    style SystemServer fill:#fff3e0
    style VirtualDevice fill:#f3e5f5

Component Responsibilities

Client-Side (Extension SDK Library)

Server-Side (System Services)

Public API Surface

Entry point: VirtualDeviceManager.requestComputerControlSession() (requires ACCESS_COMPUTER_CONTROL)

Session configuration (ComputerControlSessionParams):

Session operations (ComputerControlSession):

AIDL IPC Interfaces

11. Session Lifecycle

sequenceDiagram
    participant Agent as Agent App
    participant Server as System Server<br/>(SessionProcessor)
    participant User as User

    Agent->>Server: requestComputerControlSession(params)
    Server->>Server: Validate params<br/>(unique name, valid targets)
    Server->>Server: Check AppOps<br/>OP_COMPUTER_CONTROL

    alt Consent needed
        Server-->>Agent: onSessionPending(PendingIntent)
        Agent->>User: Launch consent dialog
        User->>User: Allow / Don't Allow / Always Allow
        alt Denied
            User-->>Server: RESULT_CANCELED
            Server-->>Agent: onSessionCreationFailed(PERMISSION_DENIED)
        else Approved
            User-->>Server: RESULT_OK
        end
    end

    Server->>Server: Check device not locked
    Server->>Server: Check session limit (< 5)
    Server->>Server: Create VirtualDevice
    Server->>Server: Create trusted VirtualDisplay
    Server->>Server: Create input devices<br/>(dpad, keyboard, touchscreen)
    Server->>Server: Disable animations
    Server->>Server: Set IME policy HIDE

    Server-->>Agent: onSessionCreated(displayId, session)

    loop Active Session
        Agent->>Server: tap / swipe / longPress
        Agent->>Server: insertText / performAction
        Agent->>Server: launchApplication
        Agent->>Server: getScreenshot
        Server-->>Agent: onSessionStable()
    end

    Agent->>Server: close()
    Server-->>Agent: onSessionClosed()

Error Conditions

Auto-Close Triggers

• Agent app process dies (via DeathRecipient / binder death)
• Explicit close() call
• System decides to terminate the session

Server-Side Session Creation

ComputerControlSessionProcessor:

• Dedicated ServiceThread at THREAD_PRIORITY_FOREGROUND (lazy-initialized)
• Maximum 5 concurrent sessions (MAXIMUM_CONCURRENT_SESSIONS)
• Session name uniqueness enforced per-package
• Consent integration via AppOpsManager.OP_COMPUTER_CONTROL

ComputerControlSessionImpl (777 lines):

• Creates VirtualDeviceParams with DEVICE_POLICY_CUSTOM for activity blocking
• Trusted virtual display with focus-stealing disabled
• Three virtual input devices: Dpad (0xCC01), Keyboard (0xCC02), Touchscreen (0xCC03)
• Activity policy: strict mode (only target packages) or legacy mode (all except PermissionController)

12. Input Handling

Input Device Architecture

graph TD
    CCSI["ComputerControlSessionImpl"]

    CCSI --> Dpad["VirtualDpad (0xCC01)<br/>DPAD_*, BACK, HOME,<br/>ENTER, TAB"]
    CCSI --> KB["VirtualKeyboard (0xCC02)<br/>Character key events"]
    CCSI --> TS["VirtualTouchscreen (0xCC03)<br/>Touch input"]

    tap["tap(x, y)"] --> TS
    swipe["swipe(from, to)"] --> TS
    longPress["longPress(x, y)"] --> TS
    sendTouch["sendTouchEvent()"] --> TS
    sendKey["sendKeyEvent()"] -->|"DPAD keycodes"| Dpad
    sendKey -->|"Other keycodes"| KB
    performAction["performAction(GO_BACK)"] --> Dpad

High-Level Gesture Implementation

Swipe interpolation: Uses Math.sin(fraction * PI/2) for natural-feeling sine easing. Dispatched via ScheduledExecutorService with 50ms delays.

Extension SDK Input Types

• TouchEvent: Immutable, builder pattern. pointerId (0-15), toolType (FINGER/PALM), action (DOWN/UP/MOVE/CANCEL), x, y, pressure (0-255), majorAxisSize. ACTION_CANCEL must pair with TOOL_TYPE_PALM.
• KeyEvent: Immutable, builder pattern. keyCode, action (DOWN/UP), eventTimeNanos.

13. Display Mirroring and Live View

The blog highlights that users can “monitor a task’s progress via notifications or ‘live view’” and “switch to manual control at any point.” This is implemented through the interactive mirror display system.

graph TB
    subgraph SessionDisplay["Session Virtual Display (Trusted)"]
        Apps["Automated Apps<br/>Running Here"]
    end

    subgraph MirrorSystem["Interactive Mirror System"]
        IMDI["InteractiveMirrorDisplayImpl"]
        MirrorDisplay["Mirror VirtualDisplay<br/>(AUTO_MIRROR flag)"]
        MirrorTS["Mirror VirtualTouchscreen"]
        ClientSurface["Client Surface"]
    end

    subgraph ClientView["Client-Side MirrorView"]
        TV["TextureView<br/>(renders mirror content)"]
        OV["Overlay View<br/>(touch indicators)"]
        MH["MirrorHelper<br/>(lifecycle management)"]
    end

    SessionDisplay -->|"AUTO_MIRROR"| MirrorDisplay
    IMDI --> MirrorDisplay
    IMDI --> MirrorTS
    MirrorDisplay --> ClientSurface
    ClientSurface --> TV

    TV --> MH
    MH -->|"Touch events<br/>(when interactive)"| MirrorTS
    OV -->|"Red circles<br/>with fade animation"| TV

Server-Side: InteractiveMirrorDisplayImpl

• Creates a mirror virtual display with AUTO_MIRROR flag linked to session display
• Associates a virtual touchscreen with the mirror display
• resize(w, h): Resizes display, destroys and recreates touchscreen (no touchscreen resize API)
• sendTouchEvent(): Forwards to mirror touchscreen

Client-Side: MirrorView

A ready-to-use FrameLayout providing the “live view” experience:

• setComputerControlSession(session) – Binds/unbinds session for mirroring
• setInteractive(boolean) – Enables touch-to-control forwarding (manual takeover)
• setShowTouches(boolean) – Shows/hides touch indicator dots
• Handles coordinate translation with letterboxing/pillarboxing

14. Stability Tracking

ComputerControl provides two stability mechanisms to help agents wait for UI to settle:

Platform-Level StabilityListener (Current)

Server-side StabilityCalculator uses timeout-based heuristics:

stateDiagram-v2
    [*] --> Waiting: Input event occurs

    Waiting --> Stable: Timeout elapsed, no new input
    Waiting --> Waiting: New input event, timer reset
    Stable --> [*]: onSessionStable() fired

    note right of Waiting
        Timeouts by event type:
        - Non-continuous (tap, text): 2000ms
        - Continuous (swipe, long press): 2500ms
        - App launch: 3000ms
        - Interaction failed: Immediate
    end note

• Registered via setStabilityListener(executor, listener)
• Only one listener per session
• Each new input event resets the timer
• Marked as temporary implementation (TODO b/428957982)

Deprecated StabilityHintCallback (Extension SDK)

• Uses AccessibilityDisplayProxy monitoring accessibility events on the virtual display
• EventIdleTracker with 500ms idle timeout
• One-shot callback pattern (must be re-registered)

15. IME and Text Input Integration

ComputerControl bypasses the normal on-screen keyboard for programmatic text entry:

sequenceDiagram
    participant Agent as Agent App
    participant Session as ComputerControlSessionImpl
    participant IMMS as InputMethodManagerService
    participant TextField as Active Text Field<br/>(in automated app)

    Note over Session: Display IME policy = HIDE<br/>(no on-screen keyboard)

    Agent->>Session: insertText("hello", false, true)

    alt computerControlTyping flag ON
        Session->>IMMS: getComputerControlInputConnection(userId, displayId)
        IMMS-->>Session: IRemoteComputerControlInputConnection
        Session->>TextField: commitText("hello") via InputConnection
        Session->>Session: Send ENTER key (commit=true)
    else Legacy path (flag OFF)
        Session->>Session: Convert text to KeyEvent[] via KeyCharacterMap
        Session->>Session: Send individual key events with 10ms delays
    end

How the InputConnection is established:

1. When an app gains input focus on a CC display, InputMethodManagerService.startInputOrWindowGainedFocus() stores the IRemoteComputerControlInputConnection in UserData.mComputerControlInputConnectionMap keyed by displayId
2. The session impl retrieves it via InputMethodManagerInternal.getComputerControlInputConnection()
3. When the client disconnects, the entry is removed

16. Automated Package Tracking

The blog mentions users can “monitor a task’s progress via notifications.” This is powered by AutomatedPackagesRepository, which allows launchers (ROLE_HOME apps) to display indicators for automated packages.

flowchart TD
    subgraph CCSession["CC Session<br/>(Virtual Display)"]
        RunningApps["Running Apps<br/>(on CC display)"]
    end

    VDMS["VirtualDeviceManagerService<br/>onRunningAppsChanged()"]
    APR["AutomatedPackagesRepository"]
    Launcher["Launcher App<br/>(ROLE_HOME)"]
    Warning["AutomatedAppLaunchWarningActivity"]

    RunningApps -->|"App changes"| VDMS
    VDMS -->|"update(deviceId, packages)"| APR
    APR -->|"onAutomatedPackagesChanged()"| Launcher

    APR -->|"Launch interception"| Warning
    Warning -->|"Wait"| Dismiss["Dismiss (don't launch)"]
    Warning -->|"Stop"| Close["Close virtual device"]

Data model:

• mAutomatedPackages: ownerPackage -> userId -> Set<packageNames> (aggregated across devices)
• mDevicePackages: deviceId -> userId -> Set<packageNames> (per-device)
• mInterceptedLaunches: Tracks packages where user already approved automated launch

17. ComputerControl Security Model

The blog emphasizes “privacy and security at their core.” ComputerControl implements multiple security layers:

Permission and Consent

flowchart TD
    Agent["Agent App"] --> Perm{"Has ACCESS_COMPUTER_CONTROL?<br/>(internal + knownSigner)"}
    Perm -->|No| Blocked["Cannot create sessions"]
    Perm -->|Yes| Consent{"AppOps OP_COMPUTER_CONTROL?"}
    Consent -->|MODE_ALLOWED| Create["Create session"]
    Consent -->|Not set| Dialog["Show consent dialog"]
    Dialog --> Allow["Allow (one-time)"]
    Dialog --> Deny["Don't Allow"]
    Dialog --> Always["Always Allow<br/>(sets MODE_ALLOWED)"]
    Allow --> Create
    Deny --> Failed["ERROR_PERMISSION_DENIED"]
    Always --> Create

Anti-Tampering Measures

UI Components (VirtualDeviceManager Package)

18. ComputerControl Feature Flags

All flags in frameworks/base/core/java/android/companion/virtual/flags/flags.aconfig:

IV. Comparative Analysis

19. AppFunctions vs ComputerControl

graph LR
    subgraph AF["AppFunctions"]
        AF1["Structured API"]
        AF2["App opt-in required"]
        AF3["Fast & reliable"]
        AF4["Type-safe data"]
        AF5["Background execution"]
        AF6["Request/response"]
    end

    subgraph CC["ComputerControl"]
        CC1["UI Automation"]
        CC2["Works with any app"]
        CC3["Visual context"]
        CC4["Pixel-based data"]
        CC5["Visible via live view"]
        CC6["Session-based"]
    end

    Agent["AI Agent"] --> AF
    Agent --> CC

    AF -.->|"Preferred when<br/>app supports it"| Result["Task Completed"]
    CC -.->|"Universal fallback"| Result

When an AI Agent Would Use Each

20. Security Model Comparison

graph TB
    subgraph AF_Security["AppFunctions Security"]
        AF_P["Permission:<br/>EXECUTE_APP_FUNCTIONS<br/>(normal + allowlist)"]
        AF_A["Agent Gating:<br/>3-layer allowlist<br/>(system + DeviceConfig + ADB)"]
        AF_C["Consent:<br/>Access request UI<br/>per agent-target pair"]
        AF_AU["Audit:<br/>SQLite access history<br/>with attribution"]
        AF_E["Enterprise:<br/>DevicePolicyManager<br/>AppFunctionsPolicy"]
        AF_S["Scope:<br/>Only declared<br/>function results"]
    end

    subgraph CC_Security["ComputerControl Security"]
        CC_P["Permission:<br/>ACCESS_COMPUTER_CONTROL<br/>(internal + knownSigner)"]
        CC_A["Agent Gating:<br/>Known signer<br/>certificates"]
        CC_C["Consent:<br/>Per-session dialog<br/>(Allow/Don't Allow/Always)"]
        CC_AU["Audit:<br/>AutomatedPackagesRepository<br/>+ launcher notifications"]
        CC_E["Enterprise:<br/>No enterprise<br/>policy integration"]
        CC_S["Scope:<br/>Full display content<br/>(screenshots + input)"]
    end

ComputerControl has a stricter permission model (internal|knownSigner vs normal with allowlist) because it has broader access – it can see and interact with any UI content, whereas AppFunctions only exposes what apps explicitly declare.

21. The Incremental Adoption Strategy

The blog outlines a roadmap: Android 17 will “broaden these capabilities to reach even more users, developers, and device manufacturers.” The dual-path architecture enables an incremental adoption strategy:

timeline
    title Android AI Agent Ecosystem Evolution
    section Phase 1 - Launch
        ComputerControl provides universal coverage : Works with any app unmodified
        AppFunctions adopted by key partners : Samsung Gallery, Calendar, Notes
        Limited rollout : Galaxy S26, select Pixel 10
    section Phase 2 - Expansion (Android 17)
        More apps adopt AppFunctions : Higher quality agent experiences
        ComputerControl coverage expands : More device manufacturers
        Developer tools and documentation : AppFunctions Jetpack library
    section Phase 3 - Maturity
        AppFunctions becomes primary path : Most apps expose structured functions
        ComputerControl remains universal fallback : Legacy apps, complex workflows
        Full ecosystem : User trust established via transparency

Key insight: The ecosystem can start with ComputerControl for broad “zero code” coverage, then progressively shift to AppFunctions as more apps integrate. This mirrors the blog’s framing:

• AppFunctions = “expose data and functionality directly” (preferred, high-quality path)
• ComputerControl = “platform doing the heavy lifting… zero code” (universal fallback)

Both are unified by a commitment to user sovereignty: AppFunctions via access management UI and audit trails, ComputerControl via consent dialogs, live view, and manual takeover.

V. Reference

22. Key Source File Reference

AppFunctions Core API (frameworks/base/core/java/android/app/appfunctions/)

AppFunctions Server (frameworks/base/services/appfunctions/java/.../appfunctions/)

AppFunctions Permission (frameworks/base/services/permission/java/.../appfunction/)

ComputerControl Extension SDK (frameworks/base/libs/computercontrol/)

ComputerControl Core API (frameworks/base/core/java/android/companion/virtual/computercontrol/)

ComputerControl Server (frameworks/base/services/companion/java/.../computercontrol/)

VirtualDevice Integration

InputMethod Integration

UI Components (frameworks/base/packages/VirtualDeviceManager/)

Feature Flags

I am not a software engineer focus on fronted area, and I wanted to replace the minima theme on my Jekyll blog with a custom one – no frontend experience required. I used Claude Code to build the entire theme in a single session.

What Was Done

Built a Complete Local Theme

Claude Code read the existing repository structure, fetched the reference design to understand the visual language, and produced a full theme from scratch:

• 4 layouts: default.html, home.html, page.html, post.html
• 4 includes: head.html (meta, fonts, dark mode flash prevention), header.html (sticky nav, theme toggle), footer.html (social links), scripts.html (theme/nav/mermaid JS)
• 4 SCSS files: variables.scss (design tokens), base.scss (reset, CSS custom properties for light/dark themes), layout.scss (all components), syntax.scss (code highlighting for both themes)

Design: Inter font, JetBrains Mono for code, purple accent (#6041ed), CSS custom properties for theme switching, responsive down to mobile.

Removed All External Theme Dependencies

Dropped the minima gem from Gemfile, removed theme: minima from _config.yml, and traced every site.minima.* reference across all templates to replace them with direct config paths.

Made Mermaid Diagrams Built-in

Before: required mermaid: true in front matter and <div class="mermaid"> HTML blocks.

After: mermaid loads automatically on every page. Standard ` ```mermaid ` fenced code blocks work out of the box – same syntax as VS Code markdown preview. JavaScript converts Jekyll’s rendered <pre><code class="language-mermaid"> elements into mermaid divs at runtime. All existing posts were refactored to the new format.

Replaced an External Plugin with a Local One

The jekyll-remote-include gem (pulled from GitHub) was replaced with a 38-line local plugin in _plugins/remote_include.rb. The local version adds proper HTTP redirect handling and error reporting. One fewer external dependency.

Homepage with Author Info and Formatted Post Excerpts

Extracted author bio and project links from the about page into a hero section on the homepage. Post excerpts render full HTML content (headings, code blocks, lists) inside a CSS height-clamp with a fade-out gradient, preserving the original article formatting.

Applied 7 Code Review Suggestions from GitHub Copilot

Gave Claude Code the PR URL. It fetched all Copilot review comments and applied them:

1. Fixed typo: giscuss to giscus
2. Mermaid dark mode: initializes with matching theme based on site preference
3. Giscus theme sync: sends postMessage to iframe on theme toggle
4. Google Analytics: gated behind jekyll.environment == 'production', measurement ID moved to _config.yml
5. Mobile nav accessibility: added aria-controls and aria-expanded state updates
6. Social links consistency: unified all templates to use site.social_links
7. Homepage excerpts: evaluated the suggestion, kept full content with CSS clamp (with rationale)

How the Session Worked

The session was iterative. Each prompt built on the previous result:

1. “Create a new theme like this” – got the base theme
2. “Article briefs should be at least 10 lines” – adjusted excerpt length
3. “Keep format/line separators in briefs” – switched to rendering full HTML with CSS height-clamp
4. “Extract about page info to the homepage” – added hero section with author bio
5. “Make mermaid support built-in” – refactored mermaid integration and all posts
6. “Clean up all minima information” – found and removed every trace
7. “Write a local plugin to replace jekyll-remote-include” – created the plugin
8. “Apply suggestions from Copilot” – processed 7 review comments

Claude Code ran bundle exec jekyll build after every change and verified the output. When it switched SCSS from @import to @use, it confirmed deprecation warnings disappeared. When it created the local plugin, it verified the about page still rendered correctly.

The Result

The blog now runs on a fully local theme:

• Dark/light mode with system preference detection and localStorage persistence
• Sticky header with responsive mobile navigation and ARIA attributes
• Built-in mermaid via standard fenced code blocks
• Giscus comments synced with site theme
• Production-only Google Analytics
• Syntax highlighting for both light and dark themes
• Zero external theme or plugin gem dependencies

All source lives in the repository. Deployable on GitHub Pages directly.

Thoughts

The Claude Code’s performance was very impressive, and generated theme is enough for me. At least, I don’t require work from another people who have front-end experience and wait their jobs. Now, I can use Claude Codew doing it by myself.

1. Introduction

Robolectric is the de-facto standard for running Android unit tests on the JVM without a device or emulator. Historically, Robolectric used hand-written Java “shadow” classes to approximate the behavior of Android framework native code. Starting around Android O (API 26), Robolectric introduced Native Runtime — a mechanism to load real AOSP native libraries (compiled for host platforms) and delegate Android framework native method calls to them, yielding pixel-accurate graphics, real SQLite behavior, and authentic text layout.

This report documents how the Robolectric Native Runtime works within the AOSP main source tree: its architecture, the native libraries involved, how they are compiled for host platforms, how they are integrated via JNI, and how the shadow layer delegates to them.

2. High-Level Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                        Robolectric Test (JVM)                        │
├──────────────────────────────────────────────────────────────────────┤
│  Test Code (JUnit)                                                   │
│       │                                                              │
│       ▼                                                              │
│  Android Framework Classes (android.graphics.*, android.database.*)  │
│       │  (bytecode-instrumented by Robolectric Sandbox)              │
│       ▼                                                              │
│  ┌────────────────────────────────────────────────────────────────┐  │
│  │           Shadow Layer (ShadowNative* classes)                 │  │
│  │  ShadowNativePaint, ShadowNativeBitmap, ShadowNativeCanvas,   │  │
│  │  ShadowNativeSQLiteConnection, ShadowNativeTypeface, ...      │  │
│  │       │                                                        │  │
│  │       ▼                                                        │  │
│  │  Natives Delegate Classes (*Natives.java)                      │  │
│  │  PaintNatives, BitmapNatives, SQLiteConnectionNatives, ...     │  │
│  │       │  (declare Java native methods)                         │  │
│  │       ▼                                                        │  │
│  │  DefaultNativeRuntimeLoader                                    │  │
│  │  - Detects OS/arch, extracts .so/.dylib/.dll from JAR          │  │
│  │  - Sets system properties for class registration               │  │
│  │  - Calls System.load() on the native library                   │  │
│  │  - Copies fonts, ICU data to temp directory                    │  │
│  └────────────────────────────────────────────────────────────────┘  │
│       │  JNI / System.load()                                         │
│       ▼                                                              │
│  ┌────────────────────────────────────────────────────────────────┐  │
│  │      Host Native Shared Library (libandroid_runtime.so)        │  │
│  │      ┌──────────────────────────────────────────────────────┐  │  │
│  │      │ HostRuntime.cpp (JNI_OnLoad entry point)             │  │  │
│  │      │  → register_android_core_classes()                   │  │  │
│  │      │  → register_android_graphics_classes()               │  │  │
│  │      │  → loadIcuData(), property_initialize_ro_cpu_abilist │  │  │
│  │      └──────────────────────────────────────────────────────┘  │  │
│  │      ┌──────────────────────────────────────────────────────┐  │  │
│  │      │ libhwui (statically linked for host)                 │  │  │
│  │      │  - Skia CPU pipeline (HWUI_NULL_GPU mode)            │  │  │
│  │      │  - Bitmap, Canvas, Paint, Path, Typeface, etc.       │  │  │
│  │      │  - Text: LineBreaker, MeasuredText, TextRunShaper    │  │  │
│  │      │  - RenderNode, RenderEffect, HardwareRenderer        │  │  │
│  │      └──────────────────────────────────────────────────────┘  │  │
│  │      ┌──────────────────────────────────────────────────────┐  │  │
│  │      │ Other statically linked host libraries               │  │  │
│  │      │  libskia, libandroidfw, libsqlite, libminikin,       │  │  │
│  │      │  libharfbuzz_ng, libft2, libpng, libjpeg, libwebp,   │  │  │
│  │      │  libicu, libz, libhostgraphics, ...                  │  │  │
│  │      └──────────────────────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

3. Supported Host Platforms

The native runtime supports the following host configurations (from DefaultNativeRuntimeLoader.isSupported()):

Source evidence: DefaultNativeRuntimeLoader.java lines 325–329:

return (OsUtil.isMac()
        && (Objects.equals(arch(), "aarch64") || Objects.equals(arch(), "x86_64")))
    || (OsUtil.isLinux() && Objects.equals(arch(), "x86_64"))
    || (OsUtil.isWindows() && Objects.equals(arch(), "x86_64"));

4. Native Libraries: What Gets Compiled & How

4.1 The Core Library: libandroid_runtime

The primary native shared library is libandroid_runtime, defined in frameworks/base/core/jni/Android.bp (line 512).

Key characteristics:

• host_supported: true — enabled for host builds
• Inherits from libandroid_runtime_defaults which includes both core JNI and graphics JNI code
• On host, uses a static linking strategy for most dependencies (libhwui, libskia, libsqlite, etc.) to produce a self-contained .so

Host-specific source files (frameworks/base/core/jni/Android.bp lines 440–508):

platform/host/HostRuntime.cpp      ← Main JNI_OnLoad entry point
platform/host/native_window_jni.cpp ← Stub native window for host

Host-specific static libraries (line 451–479):

libandroidfw          (Android Framework resources)
libhostgraphics       (Host stubs for graphics surfaces)
libhwui               (Hardware UI / graphics engine)
libft2                (FreeType font rendering)
libharfbuzz_ng        (Text shaping)
libminikin            (Text layout)
libsqlite             (SQLite database engine)
libskia               (2D graphics engine)
libpng, libjpeg, libwebp  (Image codecs)
libz                  (Compression)
libnativehelper_jvm   (JNI helper for host JVM)
libultrahdr           (Ultra HDR support)

Source evidence: frameworks/base/core/jni/Android.bp lines 451–479.

4.2 libhwui — The Graphics Engine

libhwui is defined in frameworks/base/libs/hwui/Android.bp (line 726) with host_supported: true.

Host mode differences:

• Compiled with -DHWUI_NULL_GPU — disables GPU acceleration, uses Skia CPU pipeline only
• Uses platform/host/ stub implementations for:
  • renderthread/RenderThread.cpp — simplified single-threaded rendering
  • renderthread/CacheManager.cpp — no GPU cache needed
  • Readback.cpp — stub readback
  • WebViewFunctorManager.cpp — no WebView support on host
• Includes libhostgraphics as substitute for real libgui/libnativewindow

Source evidence: frameworks/base/libs/hwui/Android.bp lines 699–722:

host: {
    srcs: [
        "platform/host/renderthread/CacheManager.cpp",
        "platform/host/renderthread/RenderThread.cpp",
        ...
    ],
    cflags: [
        "-DHWUI_NULL_GPU",
        "-DNULL_GPU_MAX_TEXTURE_SIZE=4096",
    ],
},

4.3 libhostgraphics — Host Surface Stubs

Defined in frameworks/base/libs/hostgraphics/Android.bp, this library provides host-compatible stubs for Android-specific display and surface APIs:

• ANativeWindow.cpp — stub native window
• Fence.cpp — stub sync fence
• HostBufferQueue.cpp — stub buffer queue
• PublicFormat.cpp — image format utilities
• ADisplay.cpp — stub display

4.4 Pre-V: librobolectric-nativeruntime (Prebuilt)

For Android SDK versions prior to V (VanillaIceCream), the native library is called librobolectric-nativeruntime and ships as prebuilt binaries in two forms:

1. In AOSP: prebuilts/misc/common/robolectric-native-prebuilt/
   • native/linux/x86_64/librobolectric-nativeruntime.so
   • native/mac/librobolectric-nativeruntime.dylib
2. In Gradle/Maven: The nativeruntime-dist-compat artifact (version 1.0.17) published to Maven Central.

Source evidence: prebuilts/misc/common/robolectric-native-prebuilt/Android.bp:

// Releases including and since VanillaIceCream ships with a libandroid_runtime.so
// equivalent to the librobolectric-nativeruntime.so included in this artifact.
name: "robolectric_nativeruntime_native_prebuilt"

4.5 V+ (Android 15+): libandroid_runtime (Built from AOSP)

Starting with Android V, Robolectric uses the same libandroid_runtime.so that AOSP builds for host. This library is packaged as a JAR via a java_genrule:

Source evidence: external/robolectric/Android.bp lines 65–75:

java_genrule {
    name: "libandroid_runtime_jar",
    host_first_srcs: [":libandroid_runtime"],
    out: ["libandroid_runtime.jar"],
    cmd: "mkdir -p ./native/linux/x86_64/ && " +
        "cp $(location :libandroid_runtime) ./native/linux/x86_64/ && " +
        "$(location soong_zip) -o $(location libandroid_runtime.jar) -D ./native",
}

The resulting JAR is then bundled into robolectric-host-android_all (line 123–157).

5. JNI Registration Mechanism

5.1 Two-Phase Registration (V+)

For Android V and above, JNI registration uses a class-name-driven dynamic lookup mechanism. The classes to register are communicated via Java system properties:

System.setProperty("core_native_classes", "android.database.CursorWindow,...");
System.setProperty("graphics_native_classes", "android.graphics.Bitmap,...");
System.setProperty("method_binding_format", "$$robo$$${method}$nativeBinding");

Phase 1 — Core Classes (from HostRuntime.cpp → register_android_core_classes()):

• Reads core_native_classes system property
• Looks up each class name in gRegJNIMap (an unordered_map<string, RegJNIRec>)
• Calls the corresponding register_*() function

Phase 2 — Graphics Classes (from LayoutlibLoader.cpp → register_android_graphics_classes()):

• Reads graphics_native_classes system property
• Uses a similar map in LayoutlibLoader.cpp

5.2 Method Name Rewriting

A critical mechanism is JNI method name rewriting. Normally, JNI maps native void nFoo() in class android.graphics.Paint to a C function Java_android_graphics_Paint_nFoo. But Robolectric’s bytecode instrumentor replaces native methods with Java stubs and creates a separate native binding method with a mangled name.

The format is: $$robo$$<method>$nativeBinding

This is set via:

System.setProperty("method_binding_format", "$$robo$$${method}$nativeBinding");

On the native side, graphics_jni_helpers.h contains jniRegisterMaybeRenamedNativeMethods() which transforms each JNI method registration to use the rewritten name format:

// For method "nInit", registers as "$$robo$$nInit$nativeBinding" instead
std::string modifiedName = jniMethodFormat;
modifiedName.replace(methodNamePos, 9, gMethods[i].name);

Source evidence: frameworks/base/libs/hwui/jni/graphics_jni_helpers.h lines 95–123.

5.3 Bytecode Instrumentation

Robolectric’s ClassInstrumentor (in the sandbox module) transforms native methods:

1. Original: native long nInit() in android.graphics.Paint
2. After instrumentation:
   • nInit() — becomes a non-native Java method that delegates to shadow/handler
   • $$robo$$nInit$nativeBinding() — a new native method for actual JNI binding

When callNativeMethodsByDefault = true on a shadow and no explicit shadow method is found, ShadowWrangler routes the call to the $nativeBinding method:

// ShadowWrangler.java line 166-168
Method method = definingClass.getDeclaredMethod(
    ShadowConstants.ROBO_PREFIX + name + "$nativeBinding", paramTypes);

Source evidence: sandbox/src/main/java/.../ClassInstrumentor.java lines 574–589, ShadowWrangler.java lines 160–174.

6. Core Native Class Registrations

6.1 Core Classes (CORE_CLASS_NATIVES)

These are non-graphics native classes registered for Android V+:

Source evidence: DefaultNativeRuntimeLoader.java lines 59–70.

6.2 Graphics Classes (GRAPHICS_CLASS_NATIVES)

These are graphics-related native classes (40+ classes):

Source evidence: DefaultNativeRuntimeLoader.java lines 73–121.

7. Shadow → Native Delegation Flow

┌──────────────────────────────────────────────────────────────────┐
│  Test code:  Paint paint = new Paint();                          │
│                                                                  │
│  1. Android Paint constructor calls native method nInit()        │
│                                                                  │
│  2. Instrumented nInit() → ClassHandler → ShadowWrangler         │
│                                                                  │
│  3. ShadowWrangler finds ShadowNativePaint                       │
│     → @Implementation nInit() method exists:                     │
│       a) Calls DefaultNativeRuntimeLoader.injectAndLoad()        │
│       b) Ensures native library is loaded                        │
│       c) Delegates to PaintNatives.nInit()                       │
│                                                                  │
│  4. PaintNatives.nInit() is a Java native method                 │
│     → JNI resolves to $$robo$$nInit$nativeBinding                │
│     → Calls C++ Paint_init() in libhwui's jni/Paint.cpp         │
│     → Creates real SkPaint object via Skia                       │
│     → Returns native pointer as jlong                            │
│                                                                  │
│  For V+ with callNativeMethodsByDefault = true:                  │
│  If no explicit shadow method exists, ShadowWrangler             │
│  routes directly to $$robo$$<method>$nativeBinding               │
└──────────────────────────────────────────────────────────────────┘

7.1 Shadow Example: ShadowNativePaint

@Implements(value = Paint.class, minSdk = O, callNativeMethodsByDefault = true)
public class ShadowNativePaint {
    @Implementation(minSdk = O, maxSdk = U.SDK_INT)
    protected static long nInit() {
        DefaultNativeRuntimeLoader.injectAndLoad();  // Ensure library loaded
        ShadowNativeTypeface.ensureInitialized();    // Dependency init
        return PaintNatives.nInit();                 // Delegate to real native
    }
    // For V+, most methods use callNativeMethodsByDefault
    // and don't need explicit shadow implementations
}

Source evidence: shadows/framework/.../ShadowNativePaint.java lines 36–50.

7.2 Shadow Example: ShadowNativeSQLiteConnection

@Implements(className = "android.database.sqlite.SQLiteConnection",
            callNativeMethodsByDefault = true)
public class ShadowNativeSQLiteConnection extends ShadowSQLiteConnection {
    @Implementation(minSdk = O_MR1, maxSdk = U.SDK_INT)
    protected static long nativeOpen(String path, int openFlags, ...) {
        DefaultNativeRuntimeLoader.injectAndLoad();
        return PerfStatsCollector.getInstance()
            .measure("androidsqlite",
                () -> SQLiteConnectionNatives.nativeOpen(path, openFlags, ...));
    }
}

Source evidence: shadows/framework/.../ShadowNativeSQLiteConnection.java lines 46–66.

8. Resource Loading & Initialization

8.1 Native Library Loading Flow

DefaultNativeRuntimeLoader.ensureLoaded()
│
├── 1. Check isSupported() — verify OS/arch
├── 2. Create TempDirectory("nativeruntime")
├── 3. maybeCopyFonts() — extract fonts from JAR to temp dir
│      └── Sets: robolectric.nativeruntime.fontdir=/tmp/.../fonts/
├── 4. maybeCopyIcuData() — extract ICU data file
│      └── Sets: icu.data.path=/tmp/.../icu/icudt68l.dat
│      └── Sets: icu.locale.default=<default locale tag>
├── 5. [V+ only] Set system properties:
│      ├── core_native_classes=android.database.CursorWindow,...
│      ├── graphics_native_classes=android.graphics.Bitmap,...
│      └── method_binding_format=$$robo$$${method}$nativeBinding
├── 6. loadLibrary() — extract .so/.dylib/.dll and System.load()
│      └── Looks up: native/<os>/<arch>/libandroid_runtime.so
├── 7. [V+ only] invokeDeferredStaticInitializers()
│      └── Calls __staticInitializer__ on classes that need
│          deferred init after JNI registration
└── 8. [V+ only] Typeface.loadPreinstalledSystemFontMap()

Source evidence: DefaultNativeRuntimeLoader.java lines 176–218.

8.2 Library Path Convention

Native libraries are stored in JAR resources at:

native/<os>/<arch>/<library_name>

Examples:

native/linux/x86_64/libandroid_runtime.so
native/mac/x86_64/libandroid_runtime.dylib
native/mac/aarch64/libandroid_runtime.dylib
native/windows/x86_64/libandroid_runtime.dll

Source evidence: DefaultNativeRuntimeLoader.java lines 332–334.

8.3 Bundled Resources

The nativeruntime module includes these runtime resources:

9. AOSP Build Integration (Soong)

9.1 Module Dependency Graph

Robolectric_all (java_library_host)
├── Robolectric_nativeruntime (java_library_host)
│   ├── robolectric_nativeruntime_native_prebuilt (pre-V prebuilt .so/.dylib)
│   ├── Robolectric_sandbox
│   ├── Robolectric_resources
│   └── Robolectric_pluginapi
├── Robolectric_shadows_framework (shadows layer)
├── Robolectric_robolectric
├── robolectric-host-android_all (java_library)
│   ├── robolectric_android-all-device-deps (framework JARs)
│   │   ├── framework-all, core-libart-for-host, services, ...
│   ├── libandroid_runtime_jar (native .so in JAR for V+)
│   │   └── :libandroid_runtime (cc_library_shared, host)
│   ├── robolectric_framework_res
│   ├── robolectric_props_jar (build.prop via robolectric.go)
│   └── icu-data_host_robolectric
└── ...

9.2 android_robolectric_test Module Type

Platform tests use the android_robolectric_test Soong module type which:

1. Adds Robolectric_all as a runtime dependency
2. Sets up the classpath with robolectric-host-android_all
3. Runs tests on the host JVM with the native library available

9.3 robolectric_build_props

A custom Soong module (soong/robolectric.go) generates a build.prop file with platform-specific properties (SDK version, codename, etc.) that Robolectric reads at runtime:

// soong/robolectric.go
"ro.build.version.sdk=" + ctx.Config().PlatformSdkVersion().String(),
"ro.build.version.release=" + ctx.Config().PlatformVersionName(),
"ro.build.version.codename=" + ctx.Config().PlatformSdkCodename(),

Source evidence: soong/robolectric.go lines 42–86.

10. JNI_OnLoad & Native Entry Point

When System.load() loads the native library, JNI_OnLoad in HostRuntime.cpp runs:

JNIEXPORT jint JNI_OnLoad(JavaVM* vm, void*) {
    JNIEnv* env = nullptr;
    vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION_1_6);

    // Check if base host runtime should be initialized
    string useBaseHostRuntime = getJavaProperty(env, "use_base_native_hostruntime", "true");
    if (useBaseHostRuntime == "true") {
        Vector<String8> args;
        HostRuntime runtime;
        runtime.onVmCreated(env);   // Store JavaVM reference
        runtime.start("HostRuntime", args, false);  // Register JNI + init
    }
    return JNI_VERSION_1_6;
}

HostRuntime::start() triggers:

1. setJniMethodFormat() — reads method_binding_format property for method renaming
2. startReg() → register_android_core_classes() + register_android_graphics_classes()
3. onStarted() → loadIcuData(), property_initialize_ro_cpu_abilist(), set locale

Source evidence: frameworks/base/core/jni/platform/host/HostRuntime.cpp lines 296–306.

11. Version Evolution (Pre-V vs V+)

Source evidence: DefaultNativeRuntimeLoader.java lines 336–343, 387–389.

12. Shadow Inventory

There are 88 ShadowNative* files in shadows/framework/src/main/java/.../shadows/, covering these major subsystems:

13. Compiled Native Library Dependency Tree

The following is a summary of major native libraries statically linked into the host libandroid_runtime.so and their roles:

libandroid_runtime.so (host shared library)
│
├── HostRuntime.cpp — JNI_OnLoad, AndroidRuntime host impl
├── android_database_SQLiteConnection.cpp — SQLite JNI bindings
├── android_database_CursorWindow.cpp — CursorWindow JNI
├── android_view_Surface.cpp — Surface JNI
├── android_animation_PropertyValuesHolder.cpp — Animation JNI
├── com_android_internal_util_VirtualRefBasePtr.cpp — RefBase JNI
│
├── libhwui (static, compiled with HWUI_NULL_GPU)
│   ├── LayoutlibLoader.cpp / jni_runtime.cpp — Graphics JNI registration
│   ├── jni/Paint.cpp, jni/Bitmap.cpp, jni/Canvas.cpp, ... — All graphics JNI
│   ├── hwui/Bitmap.cpp, hwui/Canvas.cpp, hwui/Typeface.cpp — Core graphics impl
│   ├── pipeline/skia/SkiaCpuPipeline.cpp — CPU-only rendering
│   └── platform/host/* — Host-specific stubs
│
├── libskia (static) — Google's 2D graphics engine
├── libminikin (static) — Android text layout engine
├── libharfbuzz_ng (static) — Unicode text shaping
├── libft2 (static) — FreeType font rasterization
├── libandroidfw (static) — Android framework (resources, assets)
├── libsqlite (static) — SQLite database engine
├── libpng (static) — PNG codec
├── libjpeg (static) — JPEG codec
├── libwebp-decode/encode (static) — WebP codec
├── libz (static) — zlib compression
├── libhostgraphics (static) — Host stubs for ANativeWindow, Fence, etc.
├── libicu{i18n,uc} (static) — ICU internationalization
├── libbase (static) — Android base/logging
├── libcutils (static) — Android core utilities
├── liblog (static) — Android logging
└── libutils (static) — Android utility library

14. The nativeruntime-dist-compat Maven Artifact

For external (non-AOSP) Gradle builds, Robolectric distributes prebuilt native libraries as the Maven artifact org.robolectric:nativeruntime-dist-compat (current version: 1.0.17).

This JAR contains pre-compiled native shared libraries for all supported platforms:

native/linux/x86_64/librobolectric-nativeruntime.so
native/mac/x86_64/librobolectric-nativeruntime.dylib
native/mac/aarch64/librobolectric-nativeruntime.dylib
native/windows/x86_64/robolectric-nativeruntime.dll

It is used in the Gradle build via:

// nativeruntime/build.gradle.kts
implementation(libs.robolectric.nativeruntime.dist.compat)

The CI workflow .github/workflows/graphics_tests.yml runs graphics tests on all four platform configurations (Linux x86_64, macOS x86_64, macOS arm64, Windows x86_64).

Source evidence: gradle/libs.versions.toml line 2, 186.

15. Some AOSP Changes for Native Runtime

15.1 Thematic Analysis

Phase 1: Foundation (Jan–Apr 2022)

The first wave of changes created the entire build infrastructure from scratch:

• Build target creation — I19296fc4d established the initial robolectric_native_runtime build target in frameworks/base/core/jni/Android.bp, a cc_library_shared with host_supported: true. This single CL (+164 lines) laid the foundation for the entire effort.

• Distribution pipeline — Ieb351f5b added the library to $DIST_DIR/robolectric/nativeruntime, and I5a99f9f7 similarly distributed the ICU dat file required for text processing.

• Core JNI classes — A rapid series of CLs incrementally added JNI registration for the most critical Android framework classes:
  • I9661646a: Matrix and initial graphics JNI
  • I046e1447: Bitmap and BitmapFactory — enabling image decoding
  • I6071aa4d: FontFamily with namespace handling (com.android.internal.graphics.fonts vs android.graphics.fonts for different SDK levels)
  • I369181f0: Typeface and text classes — completing text layout support
• Compatibility guard — I7aab27c7 added an SDK-level check to only register graphics JNI for SDK >= 26 (Android O), avoiding crashes from missing Matrix JNI fields in older versions.

Phase 2: libnativehelper and ICU (May–Sep 2022)

• libnativehelper adaptation (Icaef0cb0) — Changed the Buffer internals accessor class from a LayoutLib class to org.robolectric.nativeruntime.NIOAccess, and enabled a static cc_library for the host build. This was essential because libnativehelper provides core JNI helper functions used throughout the native runtime.

• ICU locale property (Ia219a6c2) — Updated the system property name used for ICU’s default language tag, ensuring proper locale handling in host text operations.

Phase 3: macOS Support (Jan 2023)

• Ic51c46c8 (“Add support for building RNG on Mac”, +75/-57) was a significant change that:
  • Updated the robolectric_native_runtime target with macOS-specific build rules
  • Excluded AnimatedImageDrawable from Mac builds (requires Linux’s epoll)
  • Included CursorWindow.cpp for Mac (SQLite cursor support)
  • Validated with ShadowNativeMatrixTest on Mac artifacts

Phase 4: Windows Platform (Dec 2023 – Mar 2024)

The Windows porting effort required changes across multiple AOSP repositories:

• Ie932ddec (“Enable host Windows build”) — Initial Windows build target. SQLite JNI was initially disabled because CursorWindow.cpp and SQLite JNI needed mmap/ashmem.

• Ie72a5f04 (“ashmem-host for Windows”, system/core) — Migrated to tmpfile/fileno for temp file operations (supported on MinGW), enabling CursorWindow on Windows.

• I138064847 (“SQLite JNI in Windows”) — Used libbase’s MappedFile for cross-platform mmap operations in CursorWindow.cpp and android_database_SQLiteConnection.cpp.

• I38b67e6f (“AnimatedImageDrawable for Mac and Windows”) — Re-enabled AnimatedImageDrawable on Mac/Windows by disabling the AnimatedEndListener mechanism (which requires native Looper, unavailable on host).

Two changes landed on AOSP main to improve portability for all host builds:

• I77b6b548 — Replaced raw mmap/munmap calls with MappedFile in CursorWindow
• I535449c2 — Same replacement in android_database_SQLiteConnection

Phase 5: ICU and Locale Improvements (Jan–Mar 2024)

• I7c0ffffe — Changed the ICU language tag property from robolectric.nativeruntime.languageTag to icu.locale.default for alignment with upstream LayoutLib.
• I894b9b79 (+84/-18) — Backported the latest LayoutLib ICU loading logic to the hostruntime-dev branch, improving internationalization support.
• I21e71152 — Used the minimal "C" numeric locale in strtof calls to prevent float parsing failures in locales where commas are decimal separators (e.g., German locale).

Phase 6: Hardware Rendering and Surface Support (May–Jul 2024)

This phase added hardware-accelerated rendering support (via CPU-only Skia pipeline):

• Ic037fea1 — Fixed Mac pixel color swap issue by using kN32_SkColorType (architecture- dependent) instead of a hardcoded color type for SkiaHostPipeline.
• I949382eb — Extended ImageReader and HardwareRenderer.syncAndDrawFrame support to Android Q and R (previously only Android S+).
• If52fe2a4 (+46/-44) — Synchronized hwui files with sc-layoutlib-native branch, enabling CanvasContext, RootRenderNode, RenderThread, animation logic, and more.
• I74a7d09e — Fixed hwui compile errors so m libhwui works with the Android toolchain (not just the host toolchain).
• Ieca45f62 — Un-ifdef’d Surface.lockCanvas and Surface.unlockCanvasAndPost (they compile fine on host).
• I22038e78 — Added android_view_ThreadedRenderer_setSurfacePtr for backwards compatibility with Android O/P.
• I1631d08d — Fixed frameinfo data conversion from the 9-length (Android R) to 12-length (Android S+) format for proper HW rendering support.
• Ic80c6aef — Added FileDescriptor-based ImageDecoder.nDecode variant.

Phase 7: Path APIs and Newer SDK Support (Aug 2024 – Jan 2026)

• Iab0c43c7 (+84) — Backported PathIterator JNI from Android U.
• Ib980365a (+74/-40) — Added nConicTo, nRConicTo, nInterpolate, nGetGenerationID, and nIsInterpolatable from Android U’s Path API.
• I217e91ae — Added a host-specific variant of PathIterator.nNext JNI.
• I96475c4b (+12) — Added unexported symbol list for Mac to prevent symbol clashes between librobolectric-nativeruntime (pre-V) and libandroid_runtime (V+) for ANativeWindow and Minikin symbols.
• I19e69cd5 (+45/-2, Jul 2025) — Added android.text.Hyphenator JNI support with a configurable hyphen data directory via system property.
• I8a533353 (+3/-1, Jan 2026) — Bound BitmapRegionDecoder JNI for region-based image decoding.

15.2 Development Branch Strategy

android12-hostruntime-dev (main development branch)
    │
    │   Most RNR-specific changes land here first
    │   Based on Android 12 codebase
    │   Contains customizations that can't go to main
    │     (host-only ifdef's, backported APIs, etc.)
    │
    ├── Changes stay here: JNI additions, host-only adaptations
    │
    └── Portable improvements → cherry-picked to main
            │
            ▼
        main (AOSP trunk)
            Only gets changes that benefit ALL host builds
            (LayoutLib + Robolectric)
            e.g., MappedFile portability improvements

The android12-hostruntime-dev branch serves as a long-running development branch where RNR-specific modifications are made independently of AOSP main. This approach is necessary because:

1. Many changes are host-only — They add #ifdef __ANDROID__ guards, host-specific code paths, or backported APIs that don’t belong in the shipping Android platform.

2. The base is Android 12 — The native runtime needs to support older SDK levels (O through R) with a stable codebase, while main continuously evolves.

3. Select improvements get upstreamed — Portable changes (like MappedFile for mmap) that benefit both Robolectric and LayoutLib are cherry-picked to AOSP main.

15.3 Cross-Repository Impact

The changes touch 6 AOSP repositories, demonstrating the breadth of modifications needed:

15.4 Key Engineering Patterns

1. Incremental JNI binding — Rather than a single massive CL, classes were added in small groups (Matrix → Bitmap → Font → Typeface), each tested independently.

2. Platform-first, then broaden — Linux was first, then Mac (Jan 2023), then Windows (Dec 2023), with each platform bringing unique challenges (epoll, mmap, symbol visibility).

3. Upstream when possible — Portable improvements (MappedFile) go to main; host-only customizations stay on android12-hostruntime-dev.

4. API backporting — Newer APIs (PathIterator from U, Hyphenator) are backported to the hostruntime-dev branch so RNR can support them across all SDK levels.

5. Shared infrastructure with LayoutLib — Many patterns (ICU loading, locale handling, JNI method format) are shared with Android Studio’s LayoutLib, reducing duplication.

16. Summary

Robolectric’s Native Runtime achieves high-fidelity Android emulation on the JVM by:

1. Compiling real AOSP C++ libraries (libhwui, libskia, libsqlite, libminikin, etc.) for host platforms (Linux/macOS/Windows) with GPU support disabled (HWUI_NULL_GPU)

2. Packaging them as a single shared library (libandroid_runtime.so) that statically links all dependencies

3. Using bytecode instrumentation to redirect native method calls through a renaming scheme ($$robo$$<method>$nativeBinding) that allows the shadow layer to intercept calls when needed while defaulting to real native execution

4. Providing 88+ shadow classes that ensure correct initialization, lazy loading of the native library, and version-specific API compatibility across Android O through V+

5. Bundling runtime resources (fonts, ICU data, build properties) that the native code requires for correct operation

This approach delivers pixel-accurate graphics rendering, real SQLite behavior, and authentic text layout in Robolectric tests, significantly improving test fidelity compared to the older shadow-only approach.

1) Overview

.apkm is a file format created by APKMirror to distribute Android applications that are built using Android App Bundle (AAB). Since Google Play generates device-specific split APK sets from AAB files at install time, third-party distributors like APKMirror cannot distribute a single monolithic APK for bundle-based apps. The .apkm format solves this by packaging the base APK together with all available split APKs into a single distributable archive.

An .apkm file is installed on-device using the APKMirror Installer application or APKMInstaller application from me, or manually via adb install-multiple.

2) File structure

An .apkm file is a standard ZIP archive with the .apkm file extension. The archive is signed by APKMirror using JAR signing (META-INF/) and contains metadata, an app icon, split APKs, and an installer URL shortcut.

2.1) Real-world example — Google Play Store 50.3.27

The following is the actual structure of com.android.vending v50.3.27 distributed by APKMirror (48.0 MB compressed, 106.3 MB uncompressed, 42 entries):

com.android.vending_50.3.27-31_[...].apkm (ZIP archive)
│
│  # Archive-level signing (JAR Signature v1)
├── META-INF/MANIFEST.MF                                  # SHA1 digests of all entries
├── META-INF/APKMIRRO.SF                                  # Signed manifest
├── META-INF/APKMIRRO.RSA                                 # APKMirror certificate (self-signed)
│
│  # Metadata
├── info.json                                              # Package metadata (required)
├── icon.png                                               # App icon, 96x96 PNG (optional)
├── APKM_installer.url                                     # Windows .url shortcut to APKMirror Installer page
│
│  # Base APK
├── base.apk                                               # 53.7 MB uncompressed
│
│  # ABI configuration splits (3 architectures)
├── split_config.arm64_v8a.apk                             # 8.6 MB
├── split_config.armeabi_v7a.apk                           # 5.4 MB
├── split_config.x86.apk                                   # 9.1 MB
│
│  # Language configuration splits (24 languages)
├── split_config.ar.apk
├── split_config.de.apk
├── split_config.en.apk
├── split_config.es.apk
├── split_config.et.apk
├── split_config.fi.apk
├── split_config.fr.apk
├── split_config.hi.apk
├── split_config.hu.apk
├── split_config.in.apk
├── split_config.it.apk
├── split_config.ja.apk
├── split_config.ko.apk
├── split_config.ms.apk
├── split_config.nl.apk
├── split_config.pl.apk
├── split_config.pt.apk
├── split_config.ru.apk
├── split_config.sv.apk
├── split_config.th.apk
├── split_config.tr.apk
├── split_config.uk.apk
├── split_config.vi.apk
├── split_config.zh.apk
│
│  # Dynamic feature module: phonesky_data_loader
├── split_phonesky_data_loader.apk                         # Module base (16 KB)
├── split_phonesky_data_loader.config.arm64_v8a.apk        # Module ABI split
├── split_phonesky_data_loader.config.armeabi_v7a.apk
├── split_phonesky_data_loader.config.x86.apk
│
│  # Dynamic feature module: phonesky_webrtc_native_lib
├── split_phonesky_webrtc_native_lib.apk                   # Module base (16 KB)
├── split_phonesky_webrtc_native_lib.config.arm64_v8a.apk  # Module ABI split (6.6 MB)
├── split_phonesky_webrtc_native_lib.config.armeabi_v7a.apk
└── split_phonesky_webrtc_native_lib.config.x86.apk

2.2) Compression

Most entries use DEFLATED compression. Only small non-APK assets (icon.png, APKM_installer.url) are STORED uncompressed. In this example, DEFLATED achieves ~47% compression on the contained APK files (106 MB → 48 MB).

2.3) Archive-level signing

The .apkm archive itself is signed using JAR Signature (v1) by APKMirror. The META-INF/ directory contains:

This allows installers to verify that the archive contents have not been tampered with after distribution by APKMirror.

2.4) Naming conventions

The naming follows the conventions used by Android’s bundletool when generating APK sets from AAB files.

3) Metadata — info.json

The .apkm archive includes an info.json file at the root of the archive. This JSON file describes the package, its available configurations, and APKMirror-specific distribution metadata.

3.1) Real-world example — Google Play Store 50.3.27

{
    "apkm_version": 5,
    "apk_title": "Google Play Store 50.3.27-31 [0] [PR] 875414499 (nodpi) (Android 12L+)",
    "app_name": "Google Play Store",
    "release_version": "50.3.27-31 [0] [PR] 875414499",
    "variant": "(universal) (nodpi) (Android 12L+)",
    "release_title": "Google Play Store 50.3.27-31 [0] [PR] 875414499 (nodpi) (Android 12L+)",
    "versioncode": "85032730",
    "pname": "com.android.vending",
    "post_date": "2026-02-27 00:48:12",
    "capabilities": [
        "auto",
        "cardboard"
    ],
    "languages": [
        "ar", "de", "en", "es", "et", "fi", "fr", "hi", "hu", "in",
        "it", "ja", "ko", "ms", "nl", "pl", "pt", "ru", "sv", "th",
        "tr", "uk", "vi", "zh"
    ],
    "arches": [
        "arm64-v8a",
        "armeabi-v7a",
        "x86"
    ],
    "dpis": [
        "nodpi"
    ],
    "min_api": "32",
    "accent_color": "4084f4",
    "apk_id": 12692714,
    "release_id": 12691902
}

3.2) Field definitions

Core fields

Configuration fields

Distribution metadata (APKMirror-specific)

4) Format versions

For version 1 archives (no info.json), the installer must parse AndroidManifest.xml from each contained APK to determine package name, version, and split configuration.

5) Split APK types

The split APKs within an .apkm file correspond to Android’s split APK mechanism introduced in Android 5.0 (API 21). The following split types may be present:

5.1) ABI splits

Contain native libraries (.so files) for a specific CPU architecture.

5.2) Screen density splits

Contain drawable resources for a specific screen density.

5.3) Language splits

Contain string resources and other locale-specific resources for a particular language or locale. In the Google Play Store example, 24 language splits are included:

ar, de, en, es, et, fi, fr, hi, hu, in, it, ja, ko, ms, nl, pl, pt, ru, sv, th, tr, uk, vi, zh

Language split sizes range from ~340 KB (split_config.et.apk) to ~860 KB (split_config.zh.apk).

5.4) Dynamic feature modules

Contain code and resources for on-demand or install-time feature modules defined in the app’s build.gradle. These follow the pattern split_<module_name>.apk and may have their own ABI configuration splits.

In the Google Play Store example, two feature modules are present:

The phonesky_webrtc_native_lib module’s ABI splits are notably large (6.6 MB for arm64-v8a) because they contain the WebRTC native library.

6) Installation

6.1) Using APKMirror Installer

The recommended method. The APKMirror Installer app:

1. Opens the .apkm file and reads info.json (or parses APKs directly for v1).
2. Selects the appropriate splits for the device (matching ABI, density, and language).
3. Installs the base APK and selected splits using the Android PackageInstaller session API (createSession → openSession → openWrite for each APK → commit).

6.2) Using APKMInstaller

APKMInstaller is an open-source, ads-free alternative installer for .apkm files. Built with Kotlin and Jetpack Compose (Material You), it provides:

1. File picker or intent-based opening — browse with the system file picker or tap an .apkm file in any file manager.
2. Package preview — displays app icon, name, package name, version, size, split count, and declared permissions before installing.
3. Step-by-step progress — animated Extract → Verify → Install flow with success/failure states.
4. Split installation — extracts all APKs to a temporary cache directory and installs them together as a single split-APK session using the Android PackageInstaller API.

Requirements: Android 8.0 (API 26)+, “Install unknown apps” permission granted.

6.3) Using adb

For manual installation, extract the archive and use adb install-multiple. You must select only the splits matching your target device:

# Extract the .apkm file (rename to .zip if your tool doesn't recognize .apkm)
unzip com.android.vending_50.3.27-31_*.apkm -d playstore/

# Install on an arm64 device with English language
adb install-multiple \
    playstore/base.apk \
    playstore/split_config.arm64_v8a.apk \
    playstore/split_config.en.apk \
    playstore/split_phonesky_data_loader.apk \
    playstore/split_phonesky_data_loader.config.arm64_v8a.apk \
    playstore/split_phonesky_webrtc_native_lib.apk \
    playstore/split_phonesky_webrtc_native_lib.config.arm64_v8a.apk

Note: When using adb install-multiple, you must only include the ABI splits that match your target device. Including mismatched ABI splits (e.g., x86 on an ARM device) will cause installation to fail. Language splits for non-installed languages are harmless but unnecessary. Feature module base splits and their matching ABI config splits should be included together.

6.4) Programmatic installation (Android)

The following code is derived from APKMInstaller’s implementation. The full process has three stages: extract, verify, and install.

Stage 1 — Extract APKs from the .apkm ZIP

// Open the .apkm file via ContentResolver (works with file pickers, intent shares, etc.)
val apkFiles = mutableListOf<File>()
var totalBytes = 0L
val maxBytes = 2L * 1024 * 1024 * 1024 // 2 GB safety cap

context.contentResolver.openInputStream(apkmUri)?.use { stream ->
    ZipInputStream(stream).use { zip ->
        var entry = zip.nextEntry
        while (entry != null) {
            if (!entry.isDirectory && entry.name.endsWith(".apk")) {
                val outFile = File(cacheDir, entry.name.substringAfterLast('/'))
                outFile.parentFile?.mkdirs()
                FileOutputStream(outFile).use { out ->
                    val written = zip.copyTo(out, bufferSize = 65_536)
                    totalBytes += written
                    require(totalBytes <= maxBytes) { "Package exceeds size limit" }
                }
                apkFiles += outFile
            }
            zip.closeEntry()
            entry = zip.nextEntry
        }
    }
}

// Sort so base.apk is always first
apkFiles.sortWith(compareBy { if (it.name == "base.apk") 0 else 1 })

Stage 2 — Verify base.apk

val baseApk = apkFiles.first()
val flags = PackageManager.GET_PERMISSIONS or PackageManager.GET_META_DATA
val packageInfo = context.packageManager
    .getPackageArchiveInfo(baseApk.absolutePath, flags)
    ?: throw IllegalArgumentException("Cannot parse base.apk")

// Set sourceDir so the system can load the app icon and label
packageInfo.applicationInfo?.let {
    it.sourceDir = baseApk.absolutePath
    it.publicSourceDir = baseApk.absolutePath
}

Stage 3 — Install via PackageInstaller session API

val installer = context.packageManager.packageInstaller

// 1. Create a session
val params = SessionParams(SessionParams.MODE_FULL_INSTALL).apply {
    setInstallReason(PackageManager.INSTALL_REASON_USER)
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
        setRequireUserAction(SessionParams.USER_ACTION_NOT_REQUIRED)
    }
}
val sessionId = installer.createSession(params)

// 2. Write each split APK into the session
installer.openSession(sessionId).use { session ->
    apkFiles.forEachIndexed { index, file ->
        session.openWrite("split_$index.apk", 0, file.length()).use { out ->
            file.inputStream().use { input -> input.copyTo(out) }
            session.fsync(out) // Ensure write is flushed to disk
        }
    }

    // 3. Commit — the system will prompt the user for confirmation
    val action = "com.example.INSTALL_RESULT_$sessionId"
    val pendingIntent = PendingIntent.getBroadcast(
        context, sessionId,
        Intent(action).setPackage(context.packageName),
        PendingIntent.FLAG_MUTABLE or PendingIntent.FLAG_UPDATE_CURRENT,
    )
    session.commit(pendingIntent.intentSender)
}

The commit triggers a system callback via the PendingIntent. The callback delivers one of:

Source: Simplified from SplitApkInstaller.kt and ApkmParser.kt in APKMInstaller.

7) Relationship to other formats

7.1) APKM vs APKS

Both formats package split APKs into a ZIP archive, but they differ in metadata format:

• APKS uses toc.pb, a Protocol Buffers file generated by bundletool, which includes detailed targeting information (SDK version ranges, device tiers, etc.).
• APKM uses info.json, a simpler JSON manifest focused on distribution metadata. It does not include granular targeting rules — split selection is handled by the installer at install time.

8) MIME type and file association

Since .apkm is a ZIP file, standard ZIP tools can open and inspect its contents. The .apkm extension serves as a hint for the APKMirror Installer to register as a handler.

9) Security considerations

• Archive-level signing: Starting from version 5, the .apkm archive itself is signed using JAR Signature v1 (META-INF/APKMIRRO.{RSA,SF} + MANIFEST.MF). The certificate is self-signed by APKMirror (O=APKMirror.com, C=US, ST=California, L=San Francisco). Installers can verify the archive integrity by checking these signatures.
• APK-level signing: Each APK within the archive is individually signed using Android’s APK signing scheme (v1/v2/v3/v4). Installers should verify APK signatures before installation.
• Package consistency: All split APKs must be signed with the same certificate as the base APK. The Android package manager enforces this during installation.
• Integrity: Users should download .apkm files only from trusted sources. APKMirror performs its own signature verification before publishing.

10) References

• APKMirror — The origin of the .apkm format.
• APKMirror Installer — Official installer for .apkm files.
• Android App Bundle documentation — Google’s documentation on the AAB format and split APKs.
• bundletool — Google’s tool for building and inspecting app bundles and APK sets.
• Split APKs — Android documentation on APK splitting.
• PackageInstaller API — Android API for installing split APK sessions.

Disclaimer: This document is an unofficial specification based on publicly observable behavior of .apkm files distributed by APKMirror and the APKMirror Installer application. APKMirror has not published a formal specification. Details may change without notice.

The analysis generated with help of GitHub Copilot CLI with multiple models.

0) Scope and evidence base

This report is based on analysis of:

• Android Studio source: ~/android-toolchains/studio-main
  • tools/adt/idea/layout-inspector/
  • tools/base/dynamic-layout-inspector/agent/appinspection/
  • tools/adt/idea/app-inspection/
• Jetpack Compose inspector source: ~/android-toolchains/androidx-main/frameworks/support/compose/ui/ui-inspection/
• Android platform/emulator source subset: ~/android-toolchains/emu-master-dev
• Running device verification:
  • emulator: emulator-5554, a local running Emulator instance made and started manually
  • Android: 16
  • API level: 36

Analysis focus:

1. legacy + modern approaches
2. embedded (streaming) + separate-window modes together
3. API-level and sequence detail
4. click/input response and highlight (View + Compose)
5. snapshot mode vs interactive mode
6. how Compose injects into Layout Inspector

Executive summary

• On modern devices (API 29+), Layout Inspector is primarily an App Inspection pipeline with two device-side inspectors: View + Compose.
• Selection/highlight is a single host-side state flow (setSelection) fed by either IDE clicks or device UserInputEvent.
• Live mode is continuous fetch (StartFetchCommand{continuous=true}); snapshot mode is cached-state analysis plus explicit snapshot capture.
• Compose integration is runtime-hook based (WrappedComposition.setContent) and exports virtual nodes with anchor-stable identity.

Evidence legend used in this report

• Verified in source: directly supported by classes/proto definitions in this workspace.
• Verified on device: confirmed via adb checks on the connected emulator.
• Inferred from implementation flow: high-confidence behavior derived from code paths spanning host and agent modules.

1) High-level architecture (old + new)

flowchart TD
    H[Studio Layout Inspector Plugin] --> HL[InspectorClientLauncher]
    HL --> HM[AppInspectionInspectorClient API29Plus]
    HL --> HO[LegacyClient pre-29 fallback]
    HM -->|transport protobuf| D[Device App Process]
    D --> DV[View Inspector Agent]
    D --> DC[Compose Inspector Agent]
    H --> UI[UI Modes Standalone and Embedded]
    DV[View Inspector Agent] --> A1[WindowInspector]
    DV --> A2[ViewDebug and HardwareRenderer]
    DV --> A3[PixelCopy and ViewTreeObserver]
    DV --> A4[View inspector APIs]
    DC[Compose Inspector Agent] --> C1[CompositionData and SlotTable]
    DC --> C2[Recomposer observation APIs]
    DC --> C3[ART hook points]

Working terminology

• Live/interactive mode: continuous streaming from device (StartFetchCommand{continuous=true}).
• Snapshot/paused mode: no continuous streaming; inspect cached state and optionally capture explicit snapshot payloads.
• Standalone mode: classic Layout Inspector tool window.
• Embedded mode: Layout Inspector integrated in Running Devices.
• On-device rendering: highlight/overlay rendering performed on the device surface via draw/intercept commands.

2) Old vs new technical approaches

2.1 Legacy approach (pre-API 29 path in Studio)

Host classes (Studio):

• LegacyClient
• LegacyTreeLoader
• LegacyTreeParser
• LegacyPropertiesProvider

Transport/data approach:

• DDM/ddmlib based
• Client.dumpViewHierarchy(...)
• Client.captureView(...)
• DebugViewDumpHandler
• shell commands (am get-config, dumpsys activity activities)

Characteristics:

• no App Inspection agent pipeline
• older hierarchy/image dump flow
• pre-Q compatibility fallback

2.2 Modern approach (API 29+)

Host classes:

• AppInspectionInspectorClient (orchestrator)
• ViewLayoutInspectorClient (View protocol)
• ComposeLayoutInspectorClient (Compose protocol)
• AppInspectionTreeLoader

On-device agents:

• View agent from Studio repo (dynamic-layout-inspector)
• Compose agent from AndroidX repo (compose/ui/ui-inspection)

Characteristics:

• command/response/event protocol via protobuf
• continuous/live mode + snapshot support
• supports on-device rendering path (overlay drawing/input)
• supports Compose tree + parameters + recomposition/state-read features

3) Android platform API mapping by mode (core)

Evidence anchors for Section 3 claims

• Verified in source:
  • modern host/client selection: studio-main/tools/adt/idea/layout-inspector/.../InspectorClientLauncher.kt
  • legacy DDM capture path: .../pipeline/legacy/LegacyTreeLoader.kt
  • view agent APIs (WindowInspector, ViewDebug, PixelCopy): studio-main/tools/base/dynamic-layout-inspector/agent/appinspection/src/main/...
  • compose protocol usage in host: .../pipeline/appinspection/compose/ComposeLayoutInspectorClient.kt
  • compose runtime extraction logic: androidx-main/frameworks/support/compose/ui/ui-inspection/src/main/java/...
• Verified on device:
  • API level and runtime checks were observed via adb (ro.build.version.sdk, dumpsys window, debug settings).
• Inferred from implementation flow:
  • cross-module timing/ordering behavior (for example, exact end-to-end event interleaving under load) is inferred from host+agent code paths rather than a single traced run.

4) Protocol surfaces used by Layout Inspector

4.1 View protocol (view_layout_inspection.proto)

Commands (host -> device)

Must-know commands (core workflow)

• StartFetchCommand { continuous }
• StopFetchCommand
• GetPropertiesCommand { root_view_id, view_id }
• CaptureSnapshotCommand { screenshot_type }
• UpdateScreenshotTypeCommand { type, scale }

Advanced/feature commands (mode-dependent)

• EnableBitmapScreenshotCommand
• EnableXrInspectionCommand
• EnableOnDeviceRenderingCommand
• DrawCommand { draw_instructions, type }
• InterceptTouchEventsCommand { intercept }
• DrawOverlayCommand { image }
• SetOverlayAlphaCommand { alpha }

Events (device -> host)

• WindowRootsEvent
• LayoutEvent
• PropertiesEvent
• ProgressEvent
• FoldEvent
• UserInputEvent (selection/hover/right-click/double-click)
• ErrorEvent

Data payload highlights

• ViewNode
• Bounds (layout + transformed render quad)
• Screenshot (SKP / BITMAP)
• PropertyGroup
• DrawInstruction

4.2 Compose protocol (compose_layout_inspection.proto)

Commands

Must-know commands (core Compose inspection)

• GetComposablesCommand
• GetParametersCommand

Advanced commands (deep diagnostics)

• GetAllParametersCommand
• GetParameterDetailsCommand
• settings commands for recomposition/state-read support

Payload highlights

• ComposableRoot
• ComposableNode
• Parameter
• references/anchors for node identity and parameter expansion

4.3 Skia parser protocol (skia.proto)

• service SkiaParserService
  • GetViewTree(...)
  • GetViewTree2(stream ...)
• used for SKP parsing + per-node images

5) Embedded mode vs separate window mode

Why the two modes can look similar

At first glance, both modes can look almost identical because they share the same inspector model, node selection logic, and highlight semantics.
The practical difference is the base image layer and coordinate pipeline:

• Embedded mode draws on top of the Running Devices stream surface (AbstractDisplayView) and applies stream-specific transforms (displayRectangle, screenScalingFactor, orientation correction).
• Standalone mode renders in LI-owned panels without depending on the Running Devices display widget tree.

Embedded mode rendering stack (streaming path)

flowchart TD
    A[Device mirroring path] --> B[Screen sharing agent]
    B --> C[Video frames and display metadata]
    C --> D[RunningDevices DeviceView and AbstractDisplayView]
    D --> E[Layout Inspector embedded renderer]
    E --> F[Transform LI bounds to stream rectangle and orientation]
    F --> G[Overlay highlight in embedded panel]

Embedded mode and virtual display details

For streamed physical devices, the screen-sharing agent path includes virtual-display style capture setup in the agent implementation:

• DisplayManager::CreateVirtualDisplay(...) path for newer feature levels.
• Fallback display token path (SurfaceControl::CreateDisplay(...)) on older paths.
• Running Devices passes frame metadata used by LI embedded renderer (for example, orientation correction and display rectangle alignment).

For emulators, the code path documents that Running Devices orientation correction is typically 0, while streamed physical-device paths may carry non-zero correction that LI must reconcile.

That is why your recollection is correct: embedded mode is tied to the streaming/mirroring stack, and that stack can use virtual display mechanisms on device side.

sequenceDiagram
    participant RD as Running Devices Studio
    participant Agent as Screen Sharing Agent
    participant DM as DisplayManager SurfaceControl
    participant LI as Embedded Layout Inspector

    RD->>Agent: Start mirroring session
    Agent->>DM: Create virtual display or create display
    DM-->>Agent: Display capture surface token
    Agent-->>RD: Encoded frames and orientation correction metadata
    RD->>LI: Render frame in AbstractDisplayView
    LI->>LI: Align LI bounds to stream rectangle and orientation

Evidence

• Verified in source:
  • LI embedded integration over Running Devices display components: layout-inspector/.../runningdevices/LayoutInspectorManager.kt
  • stream-space transform inputs in embedded renderer: StudioRendererPanel.kt (displayRectangleProvider, screenScaleProvider, orientation correction)
  • stream metadata consumed by Running Devices display view: streaming/device/DeviceView.kt
  • virtual display creation in screen-sharing agent: streaming/screen-sharing-agent/.../display_streamer.cc, .../accessors/display_manager.cc
• Inferred from implementation flow:
  • exact visual differences depend on whether embedded rendering is Studio-side overlay or on-device overlay path, plus device/emulator stream behavior.

6) Click/input response and highlight path (View + Compose)

6.1 Click in Studio panel -> selected node -> highlighted node

flowchart TD
    C1[User click in Studio panel] --> C2[Transform panel coordinates to model coordinates]
    C2 --> C3[Hit test against bounds and hit rects]
    C3 --> C4[Resolve top node by depth and heuristics]
    C4 --> C5[Update selected node in inspector model]
    C5 --> C6[Selection listeners fire]
    C6 --> C7[Render model updates selected/hover state]
    C7 --> C8[Border or overlay highlight appears]

Important implementation behavior

• hit testing is not just first-contains; it resolves ordering/depth
• Compose-aware prioritization exists (draw-modifier-related heuristics)
• selection event fans out to tree/properties/preview/overlay components

6.2 View node vs Compose node selection

View node:

• maps directly to actual View hierarchy entries
• property retrieval via view inspector property caches/providers

Compose node:

• virtual node (not 1:1 Android View)
• resolved through Compose tree data produced by compose agent
• anchored by compose identity/anchor mapping; can still be selected/highlighted in Studio

6.3 On-device click interception path (embedded on-device rendering mode)

sequenceDiagram
    participant Host as Studio Host
    participant Agent as Device Inspector Agent
    participant Overlay as Device Overlay

    Host->>Agent: Enable touch interception command
    Agent->>Overlay: Enable touch interception
    Overlay-->>Agent: User touch event
    Agent-->>Host: User input event with coordinates
    Host->>Host: Resolve node from coordinates and root
    Host->>Host: Update InspectorModel selection
    Host->>Agent: DrawCommand and overlay update

With touch interception enabled, device taps are emitted as UserInputEvent and resolved through the same host selection pipeline used for IDE clicks.

7) How highlight drawing is performed

There are two primary rendering patterns:

1. Studio-side highlight rendering
   • paint borders/hover outlines in Studio renderer panels
   • colors/style differ by selected/hovered/recomposing states
2. On-device highlight rendering
   • host computes DrawInstruction(bounds,color,label,stroke)
   • sends DrawCommand
   • device overlay draws highlight directly on device frame

So highlighting can be host-rendered, device-rendered, or hybrid depending on mode/settings.

8) Interactive/live mode vs snapshot mode

8.1 Live (interactive) mode

• host starts fetch with StartFetchCommand { continuous=true }
• agent continuously emits LayoutEvent/PropertiesEvent and related events
• model updates in near real-time
• suitable for interaction, immediate response to UI changes

8.2 Snapshot/paused mode

• host issues StopFetchCommand
• model uses latest cached data
• no continuous live stream updates
• user can inspect properties/tree from captured state

8.3 Refresh in paused mode

• one-shot refresh path (refresh() flow)
• pulls a single update round without switching to full continuous streaming

8.4 Snapshot capture to disk

• CaptureSnapshotCommand used to collect complete snapshot payloads
• persisted using snapshot.proto structures (Metadata, Snapshot, compose info, fold info, etc.)
• can include view tree + properties + compose payloads + screenshot data

Snapshot format highlights (snapshot.proto)

• Metadata: api level, process name, source/version, dpi/font scale, etc.
• Snapshot.view_snapshot: view-capture payload
• compose sections for composables + parameters
• optional fold/device state data

9) Compose injection into Layout Inspector (deep dive)

9.1 Registration/discovery

• Compose inspector provides ComposeLayoutInspectorFactory
• discovered via ServiceLoader/inspection framework using inspector ID

9.2 Injection/hook strategy

Compose inspection initializes through the following runtime path:

flowchart TD
    I1[Compose inspector starts] --> I2[Enable inspection support]
    I2 --> I3[Install ART hook on WrappedComposition setContent]
    I3 --> I4[On setContent, attach slot-table tracking to owner view]

Implementation concepts:

• artTooling.registerEntryHook(...) on composition entry points
• owner view gets slot-table marker/tag storage
• storage uses weak references for lifecycle safety (WeakHashMap<CompositionData,...> pattern)

Evidence:

• Verified in source: androidx-main/.../ComposeLayoutInspector.kt and .../framework/ViewExtensions.kt
• Inferred from implementation flow: the exact ordering between hook install and already-alive compositions can vary by app lifecycle timing.

9.3 Hot reload/bootstrap support

Compose inspector can trigger runtime pathways to ensure existing compositions become visible to ins

``pection:

• calls around HotReloader state save/restore paths
• ensures slot-table/composition data is populated for already-running content

Evidence:

• Verified in source: androidx-main/.../ComposeLayoutInspector.kt (HotReloader access paths)

9.4 Building virtual Compose tree

flowchart TD
    V1[CompositionData groups] --> V2[CompositionBuilder conversion]
    V2 --> V3[InspectorNode virtual tree]
    V3 --> V4[Anchor-based stable IDs]
    V4 --> V5[Return tree to Studio via compose protocol]

Because Compose nodes are virtual, this anchor-based mapping is critical for:

• click resolution
• parameter lookup
• recomposition counters
• stable identity across updates

Evidence:

• Verified in source: androidx-main/.../inspector/CompositionBuilder.kt, .../inspector/InspectorNode.kt, .../LayoutInspectorTree.kt

10) Sequence diagrams (API/data flow detail)

10.1 Attach and client choice

sequenceDiagram
    participant U as User
    participant L as InspectorClientLauncher
    participant A as AppInspectionInspectorClient
    participant G as Device Agents
    participant C as LegacyClient

    U->>L: Select process
    alt API 29 plus
        L->>A: Create modern client
        A->>G: Connect and initialize View and Compose inspectors
        A->>A: Start mode per settings
    else API below 29
        L->>C: Create legacy client
        C->>C: Attach via ddmlib path
    end

10.2 Live interactive cycle (modern)

sequenceDiagram
    participant S as Studio
    participant D as Device Agent

    S->>D: StartFetchCommand continuous mode
    loop Continuous updates
        D-->>S: WindowRootsEvent when roots change
        D-->>S: LayoutEvent
        D-->>S: PropertiesEvent
        S->>S: Merge view and compose trees
        S->>S: Refresh render properties and selection overlays
    end

10.3 Snapshot save cycle

sequenceDiagram
    participant U as User
    participant S as Studio
    participant D as Device Agent
    participant F as Snapshot File

    U->>S: Save Snapshot
    S->>D: CaptureSnapshotCommand
    D-->>S: Snapshot payload data
    S->>S: Serialize metadata and snapshot proto
    S->>F: Write snapshot to disk

10.4 Studio click to highlight cycle

Covered in detail in Section 6.1; the same path is reused here in sequence context.

10.5 On-device click to Studio selection cycle

Covered in detail in Section 6.3; this section keeps the end-to-end runtime timeline without repeating the same diagram.

10.6 Compose injection lifecycle

sequenceDiagram
    participant Host as Studio Host
    participant Agent as Compose Inspector Agent
    participant Runtime as Compose Runtime

    Host->>Agent: Launch compose inspector
    Agent->>Runtime: Install setContent hook
    Runtime-->>Agent: Composition setup callbacks
    Agent->>Runtime: Attach slot-table tracking
    Agent->>Runtime: Observe composition data
    Agent-->>Host: Send virtual tree + parameters

11) Android API-level and feature matrix

These API levels are practical lower bounds observed in current implementation paths; exact behavior can vary by platform patch level, Compose runtime version, and fallback code path (legacy vs app inspection).

Evidence:

• Verified in source: host-side fallback/client selection and API-gated behavior in layout inspector pipeline classes.
• Inferred from implementation flow: practical runtime behavior on OEM images may differ from AOSP/emulator defaults.

12) Emulator verification notes (Android 16 / API 36)

Observed on running emulator:

• ro.build.version.sdk = 36
• dumpsys window windows returns active windows (as expected)
• debug_view_attributes setting currently null on this image

Interpretation:

• platform has modern APIs expected by App Inspection path
• no contradiction found vs modern LI architecture

Evidence:

• Verified on device: adb outputs captured during this analysis session.

13) Requested topics summarized

Appendix A) Practical extension points

If you plan to extend Layout Inspector behavior, these are the lowest-risk integration points:

1. Host-side analysis/enrichment
   • augment tree post-processing in Studio plugin side
   • add extra diagnostics panels from existing event stream
2. Overlay behavior extension (embedded mode)
   • use existing draw command concepts (DrawInstruction) for custom highlights
   • avoid forking platform APIs when possible
3. Compose diagnostics extension
   • build on anchor-based node identity and compose parameter/state-read data
   • keep version guards for runtime API differences
4. Snapshot pipeline extension
   • treat snapshot file as canonical, append your own metadata section externally
   • maintain backward compatibility with current proto contracts

Appendix B) Key source locations (quick index)

• Studio host pipeline:
  • tools/adt/idea/layout-inspector/src/com/android/tools/idea/layoutinspector/pipeline/
• Legacy path:
  • .../pipeline/legacy/LegacyClient.kt
  • .../pipeline/legacy/LegacyTreeLoader.kt
• Modern appinspection path:
  • .../pipeline/appinspection/AppInspectionInspectorClient.kt
  • .../pipeline/appinspection/view/ViewLayoutInspectorClient.kt
  • .../pipeline/appinspection/compose/ComposeLayoutInspectorClient.kt
• Embedded mode / running devices:
  • .../layoutinspector/runningdevices/
• View agent (device side):
  • tools/base/dynamic-layout-inspector/agent/appinspection/...
• Compose agent (device side):
  • androidx-main/frameworks/support/compose/ui/ui-inspection/...
• Protocols:
  • view_layout_inspection.proto
  • compose_layout_inspection.proto
  • skia.proto
  • snapshot.proto

16) Final synthesis

Layout Inspector is a dual-agent system built around explicit protocol boundaries:

• legacy DDM fallback still exists for older targets;
• the App Inspection path is the default on modern devices;
• embedded and standalone UIs share core model logic but differ in rendering/integration;
• View and Compose are unified in the host model while coming from different device-side pipelines;
• click/highlight and snapshot/live behavior are clear state-machine flows.

For extension work, the most stable points are host-side model enrichment, protocol-compatible overlays, and snapshot post-processing that preserves existing proto contracts.

Recommended next actions:

1. Prototype custom highlight behavior in embedded mode first (Section 5 + Section 6) because it already supports bidirectional input/render flow.
2. Add snapshot-time enrichment metadata outside existing proto messages (Section 8 + Appendix A) to keep backward compatibility.
3. Gate Compose-specific features by runtime capability checks (Section 9 + Section 11) before enabling advanced diagnostics.

Purpose

I want to build and run Vulkan applications on Windows 11 that runs in VirtualBox. I have tried to upgrade VirtualBox to 7.x, a version with GPU improvements, but it didn’t enable the supporting for Vulkan. So I decide to use swiftshader.

Build

git clone https://github.com/google/swiftshader.git --recursive
cd build
cmake .. -DSWIFTSHADER_ENABLE_VULKAN_DEBUGGER=0 -G "Visual Studio 17 2022"
cmake --build . --config Release

You can change the generate “Visual Studio 17 2022” based on your environment. I think “Visual Studio 16 2019” also can work.

We need to set SWIFTSHADER_ENABLE_VULKAN_DEBUGGER to 0 explicitly to avoid that system shows “Wait to debugger” dialog when running Vulkan applications.

After building, we can get related configuration files and dll files in the directory swiftshader/build/Windows.

Configure the swiftshader as the default Vulkan driver

We need to add the full path of vk_swiftshader_icd.json like C:\Users\test\swiftshader\build\Windows\vk_swiftshader_icd.json to the system environment variable called VK_DRIVER_FILES. THe VK_DRIVER_FILES will tell the the Vulkan loader the driver icd configuration file of swiftshader. And the Vulkan loader can find necessary swiftshader driver files based on this icd configuration json file.

Run Vulkan application

We can run Vulkan application with a CPU based Vulkan driver now.

Usage

From Android P/SDK 28, Android supports developers to use AppComponentFactory to delegate the default Service/BroadcastReceiver/ClassLoader/Activity/Application initialization. For example, we can use the following example to initialize the BroadcastReceiver with custom constructor:

AndroidManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools">

    <application
        android:appComponentFactory=".CustomAppCompFactory"
        ...
        tools:replace="android:appComponentFactory"
        tools:targetApi="31">
        ...
        <receiver
            android:name=".CustomConstructorReceiver"
            android:exported="true">
            <intent-filter>
                <action android:name="com.robolectric.CUSTOM_CONSTRUCTOR" />
            </intent-filter>
        </receiver>
    </application>

</manifest>

CustomAppCompFactory.java:

public class CustomAppCompFactory extends AppComponentFactory {
    @NonNull
    @Override
    public BroadcastReceiver instantiateReceiver(@NonNull ClassLoader cl, @NonNull String className, @Nullable Intent intent) throws ClassNotFoundException, IllegalAccessException, InstantiationException {
        if (className.contains("CustomConstructorReceiver")) {
            return new CustomConstructorReceiver(100); 
        }
        return super.instantiateReceiver(cl, className, intent);
    }
    ...
}

CustomConstructorReceiver.java:

public class CustomConstructorReceiver extends BroadcastReceiver {
    private int value;

    public CustomConstructorReceiver(int value) {
        this.value = value;
    }

    @Override
    public void onReceive(Context context, Intent intent) {
        ...
    }
}

The AppComponentFactory is powerful, and this article will analyze how AOSP supports AppComponentFactory from Android P/SDK 28. The real supporting analysis can help me to implement similar supporting for Robolectric.

Analysis

AOSP part

Parsing AndroidManifest.xml

The ParsingPackageUtils.java parses android:appComponentFactory value from AndroidManifest.xml at its parseBaseApplication method:

String factory = sa.getNonResourceString(
    R.styleable.AndroidManifestApplication_appComponentFactory);
if (factory != null) {
    String appComponentFactory = ParsingUtils.buildClassName(pkgName, factory);
    if (appComponentFactory == null) {
        return input.error("Empty class name in package " + pkgName);
    }

    pkg.setAppComponentFactory(appComponentFactory);
}

And the parsed andorid:appComponentFactory string will be passed to ParsingPackageImpl’s appComponentFactory field:

@Nullable
@DataClass.ParcelWith(ForInternedString.class)
private String appComponentFactory;

public ApplicationInfo toAppInfoWithoutStateWithoutFlags() {
    ApplicationInfo appInfo = new ApplicationInfo();
    ...
    appInfo.appComponentFactory = appComponentFactory;
    ...
}

@Override
public ParsingPackageImpl setAppComponentFactory(@Nullable String appComponentFactory) {
    this.appComponentFactory = appComponentFactory;
    return this;
}

The ParsingPackageImpl will pass this appComponentFactory string to ApplicationInfo#appComponentFactory. This process will pass the android:appComponentFactory string in AndroidManifest.xml to ApplicationInfo#appComponentFactory, from PackageManagerService part to application’s part.

Loading AppComponentFactory instance

When AOSP passes android:appComponentFactory in AndroidManifest.xml to ApplicationInfo#appComponentFactory, the next step is to load real custom AppComponentFactory instance from this string. We can use exception in test code to get real initialization process:

java.lang.RuntimeException: utzocz customAppComponentFactory
    at com.demo.myapplication.CustomAppCompFactory.<init>(CustomAppCompFactory.java:14)
    at java.lang.Class.newInstance(Native Method)
    at android.app.LoadedApk.createAppFactory(LoadedApk.java:273)
    at android.app.LoadedApk.createOrUpdateClassLoaderLocked(LoadedApk.java:1039)
    at android.app.LoadedApk.getClassLoader(LoadedApk.java:1126)
    at android.app.LoadedApk.getResources(LoadedApk.java:1374)
    at android.app.ContextImpl.createAppContext(ContextImpl.java:3090)
    at android.app.ContextImpl.createAppContext(ContextImpl.java:3082)
    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6650)

When ActivityThread initializes Application and its Context, it will create/update Application’s ClassLoader. Because AppComponentFactory is able to provide custom ClassLoader, so this process needs to update AppComponentFactory instance in LoadedApk:

if (mBaseClassLoader != null) {
    mDefaultClassLoader = mBaseClassLoader;
} else {
    mDefaultClassLoader = ClassLoader.getSystemClassLoader();
}
mAppComponentFactory = createAppFactory(mApplicationInfo, mDefaultClassLoader);

Using AppComponentFactory to initialize BroadcastReceiver

Although AppComponentFactory can provide different critical components of Android, this part only analyzes that how AppComponentFactory is used to initialize BroadcastReceiver. And it’s very simple. The ActivityThread#handleReceiver uses AppComponentFactory#instantiateReceiver to do this task when it needs to provide a BroadcastReceiver instance:

private void handleReceiver(ReceiverData data) {
    ...
    receiver = packageInfo.getAppFactory()
        .instantiateReceiver(cl, data.info.name, data.intent);
    ...

Application part

AppComponentFactory initializes BroadcastReceiver finally

public class CustomAppCompFactory extends AppComponentFactory {
    @NonNull
    @Override
    public BroadcastReceiver instantiateReceiver(@NonNull ClassLoader cl, @NonNull String className, @Nullable Intent intent) throws ClassNotFoundException, IllegalAccessException, InstantiationException {
        if (className.contains("CustomConstructorReceiver")) {
            return new CustomConstructorReceiver(100); 
        }
        return super.instantiateReceiver(cl, className, intent);
    }
    ...
}

In CustomAppCompFactory, we can initialize different custom receivers with different constructors with different input parameters depends on passed Intent and className.

Robolectric is the industry-standard local testing framework for Android. With Robolectric, your tests run in a simulated Android environment inside a JVM, without the overhead and flakiness of an emulator. At Android testing Fundamentals tutorial, Google gives a name for Robolectric: simulator. I have used it and contributed to it very much, and think it is very useful tool for Android app’s local testing. This article will explain the reason why we should consider Robolectric, and some practices that I want to recommend when using Robolectric, e.g. sharedTest pattern.

Test Pyramid and Test Scope

Before we discuss Robolectric details, we can discuss test pyramid and scope firstly.

In old Android testing tutorial, Android testing team uses test pyramid variant diagram to clarify test types in Android area:

And in new Android testing tutorial, Android testing team introduces test scope to express similar concepts but with test’s scope consideration:

Tests also vary depending on size, or degree of isolation:

1. Unit tests or small tests only verify a very small portion of the app, such as a method or class.
2. End-to-end tests or big tests verify larger parts of the app at the same time, such as a whole screen or user flow.
3. Medium tests are in between and check the integration between two or more units.

From unit test to e2e test, the fidelity, execution time, tested item scope, and the difficulty of maintenance and debugging grow progressive. When we write tests, we should balance between fidelity and development speed. We could write more tests at unit test and integration test level with some fidelity lost to improve development speed, and write small but critical e2e/ui tests to ensure fidelity finally with lower running frequency. The recommend percents of test scope from unit test to e2e test are: 70%, 20%, 10%.

At different test scope, there are some tools we can choose:

1. unit test: Mockito/MockK/PowerMock/Robolectric.
2. integration test: Robolectric/AndroidX test.
3. e2e test: Robolectric/AndroidX test.

For unit test, there are many awesome mock tools for Android: Mockito, PowerMock, and MockK. But if you want to use fake style tool for Android unit test, Robolectric is a good choice, as Roboletric’s shadow is one type of Android’s fake implementation. New Android testing tutorial prefers to fake style instead of mock style for Android testing, and Robolectric is trying to improve its performance to reduce overload when using Robolectric to write unit test, e.g. instrumented android-all jars. Robolectric also supports those mock tools on many occasions when Robolectric’s shadow doesn’t meet your requirement.

For integration test, we can use Robolectric as simulator and run those tests on JVM, and AndroidX test library to run those tests on Emulator or real devices. Robolectric also supports AndroidX test library, and developers can run tests written with AndroidX test library on Robolectric 4.x+. In this scope, we can run those tests on CI more easily when using Robolectric, because we only need JVM.

For e2e test, Robolectric also can work sometimes with better development speed and easily CI integration when writing UI related tests. And Robolectric also has advantage when testing app with system’s core services/settings, because it has various shadow APIs to control those services/settings.

Beside of test pyramid and test scope, I prefer to use local test to identify the occasion that I select Robolectric as the first choice for Android testing. If I want to run tests on local development machine or CI machine, I prefer to use Robolectric to write tests, including unit test, integration test and e2e test or instrumentation test.

Why Robolectric?

We have discussed some reasons of selecting Robolectric to write tests. We can summarize these advantages together(thanks hoisie for summarizing these advantages):

Performance

Robolectric tests run on the JVM. This avoids all of the overhead with Emulators, such as startup time, APK dexing + packaging + copying + installing.

Flakiness

Tests on Emulators have more concurrent threads, leading to nondeterminism and flakiness.

APIs

Robolectric offers lots of powerful and extensible testing APIs (shadow APIs) not available in Emulators.

Those advantages come when comparing Robolectric with Emulator for instrumentation test. And many folks also think Robolectric is useful for instrumentation test. But what about unit test? We can use mock tools for unit test totally if related logic doesn’t have too much dependencies on Android’s Context or other system APIs. If not, we can prefer Robolectric to reduce our work to mock those system APIs, and write unit test more conveniently. If test method involves a lot of modules, including hidden Android system modules, I will group it to integration test although this test method only has three lines of test code, and prefer Robolectric if I want to run it locally.

How to integrate Robolectric?

Integrating Robolectric is very simple: enabling unitTests.includeAndroidResources to use Robolectric’s maintained resource mechanism, and adding Robolectric and related recommended dependencies(AndroidX test, Google Truth and JUnit4).

android {
    testOptions {
        unitTests.includeAndroidResources = true
    }
}

dependencies {
    testImplementation 'junit:junit:4.13.2'
    testImplementation 'androidx.test:monitor:1.4.0'
    testImplementation 'androidx.test:runner:1.4.0'
    testImplementation 'androidx.test:rules:1.4.0'
    testImplementation 'androidx.test.ext:junit:1.1.3'
    testImplementation 'androidx.test.ext:junit-ktx:1.1.3'
    testImplementation 'androidx.test.ext:truth:1.4.0'
    testImplementation 'androidx.test:core:1.4.0'
    testImplementation 'com.google.truth:truth:1.1.3'
    testImplementation 'org.robolectric:robolectric:4.7.3'
}

Now, we can write test with Robolectric:

@RunWith(RobolectricTestRunner::class)
class MainActivityRobolectricTest {
   @Test
   fun `click hint button and hint view should update content with text Hint`() {
       ActivityScenario.launch(MainActivity::class.java).use { scenario ->
           scenario.onActivity { activity: MainActivity ->
               val button = activity.findViewById<Button>(R.id.btn_show_hint)
               button.performClick()
               val tvHint = activity.findViewById<TextView>(R.id.tv_hint)
               assertThat(tvHint.text).isEqualTo("Hint")
           }
       }
   }
}

With six lines of test code, we can test the response of one button’s clicking logic, and run it on JVM with command ./gradlew test. This test sample also leverages AndroidX test APIs to test UI related logic. We can change test runner to AndroidJUnit4 and run this test on real Emulator. We will discuss it at later sharedTest pattern part.

Core features

Robolectric is very simple to integration, it also has many core features that useful to simplifier tests.

Real resources

The supporting of real resources is one of my favorite core feature of Robolectric. We can test logic with resources very easily with enabling unitTests.includeAndroidResources for project:

android {
    testOptions {
        unitTests.includeAndroidResources = true
    }
}

After that, we can access resources directly in our tests with Robolectric:

val button = activity.findViewById<Button>(R.id.btn_show_hint)
button.performClick()
val tvHint = activity.findViewById<TextView>(R.id.tv_hint)
assertThat(tvHint.text).isEqualTo(MainActivity.HINT_HINT)

It’s not recommended to use legacy resources supporting, because it’s not maintained and supported with high priority by Robolectric team now.

Configure SDK

Robolectric supports Android SDK from 16-31 now if we use Robolectric 4.7.3(the latest version that recommended to use). We can use configurable SDK to control test’s testing range. Although almost of all APIs of Android SDK are stable, but there are some changes between different SDKs. For example, Activity#onMultiWindowModeChanged(boolean) is added from SDK 24, and deprecated from SDK 26 with added replacement onMultiWindowModeChanged (boolean isInMultiWindowMode, Configuration newConfig). Many apps need support many Android versions, and have compatible behaviors on different Android versions. If you have this need, Robolectric’s configurable SDK can help a lot.

The following test is used to test Activity#onMultiWindowModeChanged(boolean) from SDK N to N_MR1:

@Config(minSdk = N, maxSdk = N_MR1)
@Test
fun `old multi window mode changed and hint view should update content with text old-multi-window`() {
   rule.scenario.use { scenario: ActivityScenario<MainActivity> ->
       scenario.onActivity { activity: MainActivity ->
           // Deprecated from SDK 26
           activity.onMultiWindowModeChanged(true)
           val tvHint = activity.findViewById<TextView>(R.id.tv_hint)
           assertThat(tvHint.text).isEqualTo(MainActivity.HINT_OLD_MULTI_WINDOW)
       }
   }
}

And the next test is used to test onMultiWindowModeChanged (boolean isInMultiWindowMode, Configuration newConfig) from SDK O:

@Config(minSdk = O)
@Test
fun `multi window mode changed and hint view should update content with text multi-window`() {
   rule.scenario.use { scenario: ActivityScenario<MainActivity> ->
       scenario.onActivity { activity: MainActivity ->
           activity.onMultiWindowModeChanged(true, activity.resources.configuration)
           val tvHint = activity.findViewById<TextView>(R.id.tv_hint)
           assertThat(tvHint.text).isEqualTo(MainActivity.HINT_MULTI_WINDOW)
       }
   }
}

We can run ./gradlew test now to test onMultiWindowChanged callbacks for different Android versions.

Configure qualifiers

With Robolectric, we can configure resource qualifier for test class or test method.

// https://developer.android.com/guide/topics/resources/providing-resources.html#QualifierRules
@Config(qualifiers = "port")
@Test
fun `orientation hint view should show normal for portrait layout`() {
   rule.scenario.use { scenario: ActivityScenario<MainActivity> ->
       scenario.onActivity { activity: MainActivity ->
           val tvHintOrientation = activity.findViewById<TextView>(R.id.tv_hint_orientation);
           assertThat(tvHintOrientation.text).isEqualTo(
               activity.resources.getString(R.string.hint_orientation_normal)
           )
       }
   }
}

// https://developer.android.com/guide/topics/resources/providing-resources.html#QualifierRules
@Config(qualifiers = "land")
@Test
fun `orientation hint view should show landscape for landscape layout`() {
   rule.scenario.use { scenario: ActivityScenario<MainActivity> ->
       scenario.onActivity { activity: MainActivity ->
           val tvHintOrientation = activity.findViewById<TextView>(R.id.tv_hint_orientation);
           assertThat(tvHintOrientation.text).isEqualTo(
               activity.resources.getString(R.string.hint_orientation_landscape)
           )
       }
   }
}

For example, we can use @Config to configure display’s landscape or port state, and test different screen state for cross-device apps. It’s recommend use it with real resources enabling. We can follow Android’s qualifier rules to configure qualifier rules based on real need.

Configure display

Another core feature used by many projects, e.g. Flutter, is configuring display state when testing.

private void setExpectedDisplayRotation(int rotation) {
 ShadowDisplay display =
     Shadows.shadowOf(
         ((WindowManager)
                 RuntimeEnvironment.systemContext.getSystemService(Context.WINDOW_SERVICE))
             .getDefaultDisplay());
 display.setRotation(rotation);
}

Above example is copied from Flutter test code, and is used to change display’s rotation when testing. It’s very useful to test logic related to display rotation and rotation changing. It’s not convenient to configure display rotation when testing with Emulator.

APIs

IMO, Robolectric is a fake implementation of Android frameworks, and provides massive APIs for developer to configure frameworks’s state and get framework’s state. For example, there is a BroadcastReceiver implementation class to receive special action, and start a foreground service:

public class ShowHideTaskbarReceiver extends BroadcastReceiver {
   @Override
   public void onReceive(Context context, Intent intent) {
       if (intent == null || !ACTION_SHOW_HIDE_TASKBAR.equals(intent.getAction())) {
           return;
       }
       // some checks
       Intent notificationIntent = new Intent(context, NotificationService.class);
       // some checks
       U.startForegroundService(context, notificationIntent);
   }
}

We can use following ShadowSettings to enable canDrawOverlay to pass this receiver’s checking, and use ShadowApplication#peekNextStartedService to get last started service after passing special action to this receiver’s onReceive method:

@Test
fun `show hidden Taskbar when receiving ACTION_SHOW_HIDE_TASKBAR`() {
   val intent = Intent(Constants.ACTION_SHOW_HIDE_TASKBAR)
   // Seen things of Shadow.
   ShadowSettings.setCanDrawOverlays(true)
   Shadows.shadowOf(application).clearStartedServices()
   // onReceive will start/notify notification service to show Taskbar.
   // Robolectric will start service for it.
   // hidden thing of Shadow.
   showHideTaskbarReceiver.onReceive(context, intent)
   // Robolectric will store started service component for testing.
   // Another seen thing of Shadow.
   val startedServiceIntent = Shadows.shadowOf(application).peekNextStartedService()
   Assert.assertNotNull(startedServiceIntent)
   Assert.assertEquals(notificationIntent.component, startedServiceIntent.component)
}

Robolectric simulates a “real” service starting logic, stores started service to internal fields, and exposure those state with shadow APIs. We can visit Robolectric’s online javadoc to check supported shadow APIs, e.g. Robolectric’s 4.7 javadoc.

Multi build system support

Beside of Gradle, robolectric-bazel is used to support Bazel. If you are using Bazel for your Android project(I know there are many companies are using Bazel for it), you can use Bazel’s rules_jvm_external to integrate Robolectric:

http_archive(
    name = "robolectric",
    urls = ["https://github.com/robolectric/robolectric-bazel/archive/4.7.3.tar.gz"],
    strip_prefix = "robolectric-bazel-4.7.3",
)
load("@robolectric//bazel:robolectric.bzl", "robolectric_repositories")
robolectric_repositories()
http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-4.1",
    sha256 = "f36441aa876c4f6427bfb2d1f2d723b48e9d930b62662bf723ddfb8fc80f0140",
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/4.1.zip",
)
load("@rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
    artifacts = [
        "org.robolectric:robolectric:4.7.3",
    ],
    repositories = [
        "https://maven.google.com",
        "https://repo1.maven.org/maven2",
    ],
)

android_local_test(
    name = "greeter_activity_test",
    srcs = ["GreeterTest.java"],
    manifest = "TestManifest.xml",
    test_class = "com.example.bazel.GreeterTest",
    deps = [
        ":greeter_activity",
        "@maven//:org_robolectric_robolectric",
        "@robolectric//bazel:android-all",
    ],
)

There is an official example of local_test with robolectric-bazel from rules_jvm_external.

M1 support

M1 is very popular, Robolectric also knows it. From Robolectric 4.7, it started to support M1 with native SQLite mechanism with massive performance improvement. Many users, including me have run Robolectric on their M1 development machine. If you are using a M1 machine, what about giving a try for Robolectric?

sharedTest pattern

Google has introduced an interesting project called: Project Nitrogen

It aspires to achieve the goal of Write Once, Run Everywhere Tests on Android. Robolectirc has supported AndroidX Test from 4.0, and we can use a wide-used pattern called sharedTest pattern to re-structure our tests that can run on Robolectric and Emulator.

For example, there is a test class with AndroidX test library, and can run Robolectric and Emulator:

@RunWith(AndroidJUnit4::class)
class MainActivityCommonTest {
   @Test
   fun clickHintButton_hintViewShouldShowTextHint() {
       ActivityScenario.launch(MainActivity::class.java).use { scenario ->
           scenario.onActivity { activity: MainActivity ->
               val button = activity.findViewById<Button>(R.id.btn_show_hint)
               button.performClick()
               val tvHint = activity.findViewById<TextView>(R.id.tv_hint)
               assertThat(tvHint.text).isEqualTo("Hint")
           }
       }
   }
}

The AndroidJUnit4 can select proper runner based on running environment, e.g. RobolectricTestRunner when running on Robolectric and AndroidJUnit4ClassRunner when running on Emulator.

We can use following structure to our tests:

├── androidTest
│   └── java
│       └── demo
│           └── ExampleInstrumentedTest.kt
├── sharedTest
│   └── java
│       └── demo
│           └── MainActivityCommonTest.kt
└── test
    └── java
        └── demo
            └── MainActivityRobolectricTest.kt

And configuring test source with following config:

sourceSets {
   String sharedTestDir = 'src/sharedTest/'
   String sharedTestSourceDir = sharedTestDir + 'java'
   String sharedTestResourceDir = sharedTestDir + 'resources'
   test.resources.srcDirs += sharedTestResourceDir
   test.java.srcDirs += sharedTestSourceDir
   androidTest.resources.srcDirs += sharedTestResourceDir
   androidTest.java.srcDirs += sharedTestSourceDir
}

I also recommend to use ATD + GMD to run tests on Emulator quickly:

import com.android.build.api.dsl.ManagedVirtualDevice

testOptions {
   unitTests.includeAndroidResources = true
   devices {
       // ./gradlew -Pandroid.sdk.channel=3 nexusOneApi30DebugAndroidTest
       nexusOneApi30(ManagedVirtualDevice) {
           device = "Nexus One"
           apiLevel = 30
           systemImageSource = "aosp-atd"
           abi = "x86"
       }
   }
}

Now, we can use following commands to run tests on both Robolectric and Emulator:

./gradlew test
./gradlew -Pandroid.sdk.channel=3 nexusOneApi30DebugAndroidTest

It’s a long-term goal, and there are massive things left to do, but IMO it deserves a try. Actually, Android Studio doesn’t support sharedTest pattern’s sharedTest directory, we can star https://issuetracker.google.com/issues/132426298 to raise awareness of shardTest pattern supporting to Android Studio team.

Open-source projects use Robolectric

There are many open-source projects use Robolectric for their local tests, and we can check it with link https://github.com/search?q=org.robolectric%3Arobolectric.

1. code: 819K
2. commits: 13K+
3. Issues: 1K+

Before adopting/using Robolectric, we can check how some top open-source projects how to use Robolectric to write tests.

Flutter

https://github.com/flutter/engine/tree/main/shell/platform/android/test/io/flutter is Flutter’s local test directory, and we can show an example of Flutter to use Robolectric to test navigation bar’s location for different SDKs and display rotations.

This is Flutter’s logic to calculate navigation bar’s location:

if (orientation == Configuration.ORIENTATION_LANDSCAPE) {
 if (rotation == Surface.ROTATION_90) {
   return ZeroSides.RIGHT;
 } else if (rotation == Surface.ROTATION_270) {
   // In android API >= 23, the nav bar always appears on the "bottom" (USB) side.
   return Build.VERSION.SDK_INT >= 23 ? ZeroSides.LEFT : ZeroSides.RIGHT;
 }
 // Ambiguous orientation due to landscape left/right default. Zero both sides.
 else if (rotation == Surface.ROTATION_0 || rotation == Surface.ROTATION_180) {
   return ZeroSides.BOTH;
 }
}

The result depends on display rotation and SDK version. And Flutter uses following two tests to test all behaviors with configuring display and configuring SDK:

@Test
@Config(minSdk = 20, maxSdk = 22)
public void systemInsetHandlesFullscreenNavbarRightBelowSDK23() {
 RuntimeEnvironment.setQualifiers("+land");
 FlutterView flutterView = spy(new FlutterView(RuntimeEnvironment.systemContext));
 setExpectedDisplayRotation(Surface.ROTATION_270);
 // ...
 flutterView.onApplyWindowInsets(windowInsets);
 // ...
 validateViewportMetricPadding(viewportMetricsCaptor, 100, 0, 0, 0);
}

@Test
@Config(minSdk = 23, maxSdk = 29)
public void systemInsetHandlesFullscreenNavbarRight() {
 RuntimeEnvironment.setQualifiers("+land");
 FlutterView flutterView = spy(new FlutterView(RuntimeEnvironment.systemContext));
 setExpectedDisplayRotation(Surface.ROTATION_90);
 // ...
 flutterView.onApplyWindowInsets(windowInsets);
 // ...
 validateViewportMetricPadding(viewportMetricsCaptor, 100, 0, 0, 0);
}

Actually, we can improve those tests by using @Config to set land qualifier instead of RuntimeEnvironment.setQualifiers. Robolectric works with mockito, and Flutter also uses mockito with Robolectric to simply test logic.

AOSP

We can check projects that use Robolectric with link https://cs.android.com/search?q=android_robolectric_test&sq=&ss=android. And we will show CarSettings’ some tests as an example for AOSP usage.

CarSettings’ ScreenshotContextPreferenceControllerTest uses real resource supporting to response Preference’s state change:

@Test
public void refreshUi_screenshotDisabled_preferenceUnchecked() {
   mTwoStatePreference.setChecked(true);

   Settings.Secure.putInt(mContext.getContentResolver(),
           Settings.Secure.ASSIST_SCREENSHOT_ENABLED, 0);
   mController.refreshUi();

   assertThat(mTwoStatePreference.isChecked()).isFalse();
}

CarSettings’ ErrorDialogTest also uses real source supporting to response button click action:

@Test
public void testOkDismissesDialog() {
   ErrorDialog dialog = ErrorDialog.show(mTestFragment, R.string.delete_user_error_title);

   assertThat(isDialogShown()).isTrue(); // Dialog is shown.

   // Invoke cancel.
   DialogTestUtils.clickPositiveButton(dialog);

   assertThat(isDialogShown()).isFalse(); // Dialog is dismissed.
}

Chromium

We can visit https://source.chromium.org/search?q=robolectric_all_java&sq=&ss=chromium to check Chromium’s usage of Robolectric.

ShadowColorUtils leverages Robolectric’s shadow mechanism to shadow its ColorUtils methods to provide a fake implementation for its isNightMode. And WebContentsDarkModeControllerUnitTest adds it to shadow list, and ShadowColorUtils#isNightMode is called when normal code calls ColorUtils#isNightMode. The following tests use ShadowColorUtils to control night mode value, and test related logic.

/** Shadow class for {@link org.chromium.ui.util.ColorUtils} */
@Implements(ColorUtils.class)
public class ShadowColorUtils {
   public static boolean sInNightMode;

   @Implementation
   public static boolean inNightMode(Context context) {
       return sInNightMode;
   }
}

@RunWith(BaseRobolectricTestRunner.class)
@Config(manifest = Config.NONE, shadows = {ShadowRecordHistogram.class, ShadowColorUtils.class})
public class WebContentsDarkModeControllerUnitTest {
    //...
    @Test
    public void testFeatureEnabled() {
        ShadowColorUtils.sInNightMode = true;
        mIsGlobalSettingsEnabled = true;
        Assert.assertTrue(
                "Feature should be enabled, if both global settings and night mode enabled.",
                WebContentsDarkModeController.isFeatureEnabled(mMockContext, mMockProfile));
    }

    @Test
    public void testFeatureEnabled_LightMode() {
        ShadowColorUtils.sInNightMode = false;
        mIsGlobalSettingsEnabled = true;
        Assert.assertFalse("Feature should be disabled when not in night mode.",
                WebContentsDarkModeController.isFeatureEnabled(mMockContext, mMockProfile));
    }

    @Test
    public void testFeatureEnabled_NoUserSettings() {
        ShadowColorUtils.sInNightMode = true;
        mIsGlobalSettingsEnabled = false;
        Assert.assertFalse("Feature should be disabled when global settings disabled.",
                WebContentsDarkModeController.isFeatureEnabled(mMockContext, mMockProfile));
    }
}

Summary

Robolectric is not a perfect solution for local test, but it has massive features and fake implementation of Android frameworks to make running Android tests on JVM come true. The Robolectric team is also trying to improve Robolectric’s performance and enrich Robolectric’s functionality to provide great experience for developers. If you don’t use Robolectric ever before, and above features and examples can attract you, what about using Robolectric to write local tests? Looking forward to receive your feedback about Robolectric.

Robolectric has an official Twitter account, you can follow it if you want to receive latest news about Robolectric.

NativeActivity is added to Android from API 9, and used for games and apps that write almost of all logic with native code. The NativeActivity is used to pass basic Android app’s lifecycle to native code, and help them to manage its logic with Android app lifecycle aware. There are also some glue code from NDK for NativeActivity and real native logic to pass Android app’s lifecycle. This article will show the pipeline of passing Android app’s lifecycle from NativeActivity to native code.

Code base

AOSP android-12.0.0_r21

App sample

This article uses official NativeActivity sample for analyzing. If you are not familiar with NativeActivity, you can clone this project and run it with emulator to experience NativeActivity.

What does NativeActivity in pure Java world do?

frameworks/base/core/java/android/app/NativeActivity.java

Receive and pass lifecycle to native

The NativeActivity is an implementation of Activity, and used to receive Android app’s lifecycle:

public class NativeActivity extends Activity implements SurfaceHolder.Callback2,
        InputQueue.Callback, OnGlobalLayoutListener {
    //...
    @Override
    protected void onSaveInstanceState(Bundle outState) {
        super.onSaveInstanceState(outState);
        byte[] state = onSaveInstanceStateNative(mNativeHandle);
        if (state != null) {
            outState.putByteArray(KEY_NATIVE_SAVED_STATE, state);
        }
    }

    @Override
    protected void onStart() {
        super.onStart();
        onStartNative(mNativeHandle);
    }
    //...
    @Override
    public void onConfigurationChanged(Configuration newConfig) {
        super.onConfigurationChanged(newConfig);
        if (!mDestroyed) {
            onConfigurationChangedNative(mNativeHandle);
        }
    }

    @Override
    public void onLowMemory() {
        super.onLowMemory();
        if (!mDestroyed) {
            onLowMemoryNative(mNativeHandle);
        }
    }
    //...
}

The NativeActivity uses native methods, such as onLowMemoryNative, onConfigurationChangeNative etc, to pass lifecycle state to native. Those native methods are bound to JNI methods, and we will take look at those JNI methods at later part.

Load app’s native code

App may implements almost logic at native, so NativeActivity defines a contract for native library, and will load assigned native library at NativeActivity#onCreate:

// ...
/**
 * Optional meta-that can be in the manifest for this component, specifying
 * the name of the native shared library to load.  If not specified,
 * "main" is used.
 */
public static final String META_DATA_LIB_NAME = "android.app.lib_name";
    
/**
 * Optional meta-that can be in the manifest for this component, specifying
 * the name of the main entry point for this native activity in the
 * {@link #META_DATA_LIB_NAME} native code.  If not specified,
 * "ANativeActivity_onCreate" is used.
 */
public static final String META_DATA_FUNC_NAME = "android.app.func_name";
    
private static final String KEY_NATIVE_SAVED_STATE = "android:native_state";
// ...
@Override
protected void onCreate(Bundle savedInstanceState) {
    String libname = "main";
    String funcname = "ANativeActivity_onCreate";
    ActivityInfo ai;
    
    try {
        ai = getPackageManager().getActivityInfo(
                getIntent().getComponent(), PackageManager.GET_META_DATA);
        if (ai.metaData != null) {
            String ln = ai.metaData.getString(META_DATA_LIB_NAME);
            if (ln != null) libname = ln;
            ln = ai.metaData.getString(META_DATA_FUNC_NAME);
            if (ln != null) funcname = ln;
        }
    } catch (PackageManager.NameNotFoundException e) {
        throw new RuntimeException("Error getting activity info", e);
    }

    BaseDexClassLoader classLoader = (BaseDexClassLoader) getClassLoader();
    String path = classLoader.findLibrary(libname);

    if (path == null) {
        throw new IllegalArgumentException("Unable to find native library " + libname +
                                        " using classloader: " + classLoader.toString());
    }

    byte[] nativeSavedState = savedInstanceState != null
        ? savedInstanceState.getByteArray(KEY_NATIVE_SAVED_STATE) : null;

    mNativeHandle = loadNativeCode(path, funcname, Looper.myQueue(),
        getAbsolutePath(getFilesDir()), getAbsolutePath(getObbDir()),
        getAbsolutePath(getExternalFilesDir(null)),
        Build.VERSION.SDK_INT, getAssets(), nativeSavedState,
        classLoader, classLoader.getLdLibraryPath());

    if (mNativeHandle == 0) {
        throw new UnsatisfiedLinkError(
                "Unable to load native library \"" + path + "\": " + getDlError());
    }
    // ...
}

The NativeActivity#onCreate will read android.app.lib_name and android.app.func_name as so file name and native created method name from app’s meta data defined in AndroidManifest.xml. There is an example from official NativeActivity sample:

<!-- Tell NativeActivity the name of our .so -->
<meta-data android:name="android.app.lib_name"
        android:value="native-activity" />

After reading so file name and native created method name(maybe not existed), NativeActivity#onCreate uses BaseDexClassLoader#findLibrary to search so file’s full path. The so file will be found at extracted apk directory, if extractNativeLibs is enabled(default value is true); otherwise it will be found at apk file. We don’t discuss the occasion that system apps use so file added at /system/lib*, vendor/lib* or other supported so search paths.

If so file path is found, the NativeActivity#onCreate will call native method loadNativeCode to load so file to load native code.

Meets NativeActivity’s JNI

frameworks/base/core/jni/android_app_NativeActivity.cpp

Bind methods between Java and native

In previous part, we have saw NativeActivity in pure Java world logic to load native library and pass lifecycle with native methods, such as loadNativeCode. Those methods are defined in NativeActivity’s JNI part:

static const JNINativeMethod g_methods[] = {
    { "loadNativeCode",
        "(Ljava/lang/String;Ljava/lang/String;Landroid/os/MessageQueue;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILandroid/content/res/AssetManager;[BLjava/lang/ClassLoader;Ljava/lang/String;)J",
        (void*)loadNativeCode_native },
    { "getDlError", "()Ljava/lang/String;", (void*) getDlError_native },
    { "unloadNativeCode", "(J)V", (void*)unloadNativeCode_native },
    { "onStartNative", "(J)V", (void*)onStart_native },
    { "onResumeNative", "(J)V", (void*)onResume_native },
    // ...
};

static const char* const kNativeActivityPathName = "android/app/NativeActivity";

int register_android_app_NativeActivity(JNIEnv* env)
{
    //ALOGD("register_android_app_NativeActivity");
    jclass clazz = FindClassOrDie(env, kNativeActivityPathName);

    gNativeActivityClassInfo.finish = GetMethodIDOrDie(env, clazz, "finish", "()V");
    gNativeActivityClassInfo.setWindowFlags = GetMethodIDOrDie(env, clazz, "setWindowFlags",
                                                               "(II)V");
    gNativeActivityClassInfo.setWindowFormat = GetMethodIDOrDie(env, clazz, "setWindowFormat",
                                                                "(I)V");
    gNativeActivityClassInfo.showIme = GetMethodIDOrDie(env, clazz, "showIme", "(I)V");
    gNativeActivityClassInfo.hideIme = GetMethodIDOrDie(env, clazz, "hideIme", "(I)V");

    return RegisterMethodsOrDie(env, kNativeActivityPathName, g_methods, NELEM(g_methods));
}

The g_methods defines the bound relationship between native methods in NativeActivity.java and android_app_NativeActivity.cpp. For example, the method loadNativeCode_native in android_app_NativeActivity.cpp is the real implementation of loadNativeCode in NativeActivity.java.

Load native library code

The loadNativeCode_native uses OpenNativeLibrary in art/libnativeloader/native_loader.h to load native library found by BaseDexClassLoader in NativeActivity.java:

ScopedUtfChars pathStr(env, path);
std::unique_ptr<NativeCode> code;
bool needs_native_bridge = false;

char* nativeloader_error_msg = nullptr;
void* handle = OpenNativeLibrary(env,
                                sdkVersion,
                                pathStr.c_str(),
                                classLoader,
                                nullptr,
                                libraryPath,
                                &needs_native_bridge,
                                &nativeloader_error_msg);

Maybe you can notify the variable needs_native_bridge in loadNativeCode_native. needs_native_bridge is used for native bridge to determine whether need to load bridge libraries for ABI compatibility, for example loading x86_64 arch libraries for ARM arch libraries on Android-x86 platform.

Establish MessageQueue and Looper

Another work of loadNativeCode_native is to establish message queue with looper mechanism:

code->messageQueue = android_os_MessageQueue_getMessageQueue(env, messageQueue);
if (code->messageQueue == NULL) {
    g_error_msg = "Unable to retrieve native MessageQueue";
    ALOGW("%s", g_error_msg.c_str());
    return 0;
}

int msgpipe[2];
if (pipe(msgpipe)) {
    g_error_msg = android::base::StringPrintf("could not create pipe: %s", strerror(errno));
    ALOGW("%s", g_error_msg.c_str());
    return 0;
}
code->mainWorkRead = msgpipe[0];
code->mainWorkWrite = msgpipe[1];
int result = fcntl(code->mainWorkRead, F_SETFL, O_NONBLOCK);
SLOGW_IF(result != 0, "Could not make main work read pipe "
        "non-blocking: %s", strerror(errno));
result = fcntl(code->mainWorkWrite, F_SETFL, O_NONBLOCK);
SLOGW_IF(result != 0, "Could not make main work write pipe "
        "non-blocking: %s", strerror(errno));
code->messageQueue->getLooper()->addFd(
        code->mainWorkRead, 0, ALOOPER_EVENT_INPUT, mainWorkCallback, code.get());

It creates a pair pipe, one for writing, and one for reading. code->mainWorkWrite is used by native methods need passing values, such as android_NativeActivity_setWindowFlags in android_app_NativeActivity.cpp:

void android_NativeActivity_setWindowFlags(
        ANativeActivity* activity, int32_t values, int32_t mask) {
    NativeCode* code = static_cast<NativeCode*>(activity);
    write_work(code->mainWorkWrite, CMD_SET_WINDOW_FLAGS, values, mask);
}

and code->mainWorkRead is used for looper of message queue to receive pipe data from code->mainWorkWrite and trigger a callback method calling of looper. The real callback is ANativeActivityCallbacks defined in frameworks/native/include/android/native_activity.h. And loadNativeCode_native calls code->createActivityFunc(code.get(), rawSavedState, rawSavedSize) to call defined ANativeActivity initialized methods or default ANativeActivity_onCreate to initialize ANativeActivityCallbacks. We have saw logic to parse android.app.func_name at NativeActivity.java’s onCreate method, and its default value is ANativeActivity_onCreate.

After searching and analyzing, android_NativeActivity_setWindowFlags looks like called by native test code finally(for example external/deqp/framework/platform/android/tcuAndroidTestActivity.cpp). In another word, code->mainWorkWrite is used for native code.

Initialize JNI environment

The third thing that loadNativeCode_native does is to initialize JNI env for native code:

code->env = env;
code->clazz = env->NewGlobalRef(clazz);

const char* dirStr = env->GetStringUTFChars(internalDataDir, NULL);
code->internalDataPathObj = dirStr;
code->internalDataPath = code->internalDataPathObj.string();
env->ReleaseStringUTFChars(internalDataDir, dirStr);

if (externalDataDir != NULL) {
    dirStr = env->GetStringUTFChars(externalDataDir, NULL);
    code->externalDataPathObj = dirStr;
    env->ReleaseStringUTFChars(externalDataDir, dirStr);
}
code->externalDataPath = code->externalDataPathObj.string();

code->sdkVersion = sdkVersion;

code->javaAssetManager = env->NewGlobalRef(jAssetMgr);
code->assetManager = NdkAssetManagerForJavaObject(env, jAssetMgr);

It initializes AssetManager for native code too. And AssetManager will provide the ability to load Android resources for native code.

Pass app lifecycle

In NativeActivity.java, it calls onPauseNative at onPause method to pass pause state to native. And onPause_native in android_app_NativeActivity.cpp is the real implementation of onPauseNative:

static void
onPause_native(JNIEnv* env, jobject clazz, jlong handle)
{
    if (kLogTrace) {
        ALOGD("onPause_native");
    }
    if (handle != 0) {
        NativeCode* code = (NativeCode*)handle;
        if (code->callbacks.onPause != NULL) {
            code->callbacks.onPause(code);
        }
    }
}

onPause_native calls onPause of ANativeActivityCallbacks to pass pause state to app’s native code. We know ANativeActivity_onCreate in app native code will initialize those callbacks, but where is ANativeActivity_onCreate implementation?

NDK’s glue code

prebuilts/ndk/current/sources/android/native_app_glue/android_native_app_glue.c

At android_native_app_glue.c, we can see the implementation of ANativeActivity_onCreate and the initialization of callbacks, including onPause:

//...
static void onPause(ANativeActivity* activity) {
    LOGV("Pause: %p", activity);
    android_app_set_activity_state(ToApp(activity), APP_CMD_PAUSE);
}
//...
static void android_app_set_activity_state(struct android_app* android_app, int8_t cmd) {
    pthread_mutex_lock(&android_app->mutex);
    android_app_write_cmd(android_app, cmd);
    while (android_app->activityState != cmd) {
        pthread_cond_wait(&android_app->cond, &android_app->mutex);
    }
    pthread_mutex_unlock(&android_app->mutex);
}
//...
JNIEXPORT
void ANativeActivity_onCreate(ANativeActivity* activity, void* savedState, size_t savedStateSize) {
    LOGV("Creating: %p", activity);

    activity->callbacks->onConfigurationChanged = onConfigurationChanged;
    activity->callbacks->onContentRectChanged = onContentRectChanged;
    activity->callbacks->onDestroy = onDestroy;
    activity->callbacks->onInputQueueCreated = onInputQueueCreated;
    activity->callbacks->onInputQueueDestroyed = onInputQueueDestroyed;
    activity->callbacks->onLowMemory = onLowMemory;
    activity->callbacks->onNativeWindowCreated = onNativeWindowCreated;
    activity->callbacks->onNativeWindowDestroyed = onNativeWindowDestroyed;
    activity->callbacks->onNativeWindowRedrawNeeded = onNativeWindowRedrawNeeded;
    activity->callbacks->onNativeWindowResized = onNativeWindowResized;
    activity->callbacks->onPause = onPause;
    activity->callbacks->onResume = onResume;
    activity->callbacks->onSaveInstanceState = onSaveInstanceState;
    activity->callbacks->onStart = onStart;
    activity->callbacks->onStop = onStop;
    activity->callbacks->onWindowFocusChanged = onWindowFocusChanged;

    activity->instance = android_app_create(activity, savedState, savedStateSize);
}

When onPause in android_native_app_glue.c is called, it will send APP_CMD_PAUSE command to app’s native code with method named android_app_set_activity_state and android_app_write_cmd. Actually, android_app_write_cmd use pair pipe to pass data too, and we can see the initialization at android_app_create in android_native_app_glue.c. We know android_app_create is called at the end of ANativeActivity_onCreate, and it will create the instance of android_app defined in prebuilts/ndk/current/sources/android/native_app_glue/android_native_app_glue.h, and return it to activity->instance.

One thing about android_app_create should be pointed out specifically: this method creates custom thread for android_app:

pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
pthread_create(&android_app->thread, &attr, android_app_entry, android_app);

// Wait for thread to start.
pthread_mutex_lock(&android_app->mutex);
while (!android_app->running) {
   pthread_cond_wait(&android_app->cond, &android_app->mutex);
}
pthread_mutex_unlock(&android_app->mutex);

So app’s native code will run in separate thread, and it is the reason that android_app needs pair pipe for communication.

Go to app finally

https://github.com/android/ndk-samples/tree/master/native-activity

android_native_app_glue.c is provided by NDK, and app can use it directly like native-activity’s app/src/main/cpp/CMakeLists.txt:

add_library(native_app_glue STATIC
    ${ANDROID_NDK}/sources/android/native_app_glue/android_native_app_glue.c)

And android_app_create in android_native_app_glue.c calls pthread_create to run android_thread_entry method in single thread:

pthread_create(&android_app->thread, &attr, android_app_entry, android_app);

And android_app_entry will call android_main methods in native library to run app’s native code:

android_main(android_app);

If app’s native code implements android_main method, and it will be used as native code entry. We can see the implementation of native-activity at app/src/main/cpp/main.cpp:

/**
 * This is the main entry point of a native application that is using
 * android_native_app_glue.  It runs in its own thread, with its own
 * event loop for receiving input events and doing other things.
 */
void android_main(struct android_app* state) {
    struct engine engine{};

    memset(&engine, 0, sizeof(engine));
    state->userData = &engine;
    state->onAppCmd = engine_handle_cmd;
    state->onInputEvent = engine_handle_input;
    engine.app = state;
    //...
}

native-activity uses its engine_handle_cmd method as callback for app command, including app lifecycle. The onAppCmd is bound to looper at android_app_entry in android_native_app_glue.c:

ALooper* looper = ALooper_prepare(ALOOPER_PREPARE_ALLOW_NON_CALLBACKS);
ALooper_addFd(looper, android_app->msgread, LOOPER_ID_MAIN, ALOOPER_EVENT_INPUT, NULL,
        &android_app->cmdPollSource);
android_app->looper = looper;

When there are events coming to looper, android_main in app/src/main/cpp/main.cpp will call process method to trigger onAppCmd calling(see process_cmd in android_native_app_glue.c):

while ((ident=ALooper_pollAll(engine.animating ? 0 : -1, nullptr, &events,
                                (void**)&source)) >= 0) {

    // Process this event.
    if (source != nullptr) {
        source->process(state, source);
    }
    //...
}

The android_main initializes app’s native method as callback to process app commands. It also call ALooper_pollAll to wait looper events from NDK’s app glue and call source->process to leverage app glue’s process to call onAppCmd implementation, app’s engine_handle_cmd to process app lifecycle and other commands. The source is android_poll_source instance, and android_poll_source is defined in NDK’s app_native_app_glue.h, so app’s native library works under NDK’s app glue.

App’s engine_handle_cmd is responsible for passing commands. Unfortunately, native-activity doesn’t process APP_CMD_PAUSE, we can take look at another command:

case APP_CMD_INIT_WINDOW:
    // The window is being shown, get it ready.
    if (engine->app->window != nullptr) {
        engine_init_display(engine);
        engine_draw_frame(engine);
    }
    break;

When APP_CMD_INIT_WINDOW coming, it will call engine_init_display to initialize OpenGL ES environment, and call engine_draw_frame after it to draw contents. Other app commands use similar process and logic to process.

Share Surface between Java and native code

App’s native code uses OpenGL ES and EGL to draw contents on Android’s surface:

surface = eglCreateWindowSurface(display, config, engine->app->window, nullptr);

And it uses engine->app->window, aka android_app->window(see android_native_app_glue.h), as Android’s surface for eglCreateWindowSurface. So where does android_app->window comes from?

android_app->window is initialized with android_app->pendingWindow by android_app_pre_exec_cmd in android_native_app_glue.c. And android_app->pendingWindow is initialized by android_app_set_window in android_native_app_glue.c. The android_app_set_window is called by onNativeWindowCreated of android_native_app_glue.c.

If we come back to NativeActivity.java, we know NativeActivity.java uses getWindow().takeSurface(this) to take ownership of window surface that NativeActivity attached to, and call native callbacks when surface lifecycle changed, including surface created:

public void surfaceCreated(SurfaceHolder holder) {
    if (!mDestroyed) {
        mCurSurfaceHolder = holder;
        onSurfaceCreatedNative(mNativeHandle, holder.getSurface());
    }
}

The implementation of onSurfaceCreatedNative in android_app_NativeActivity.cpp will convert surface passed from Java to ANativeWindow:

//...
void setSurface(jobject _surface) {
    if (_surface != NULL) {
        nativeWindow = android_view_Surface_getNativeWindow(env, _surface);
    } else {
        nativeWindow = NULL;
    }
}
//...
static void
onSurfaceCreated_native(JNIEnv* env, jobject clazz, jlong handle, jobject surface)
{
    if (kLogTrace) {
        ALOGD("onSurfaceCreated_native");
    }
    if (handle != 0) {
        NativeCode* code = (NativeCode*)handle;
        code->setSurface(surface);
        if (code->nativeWindow != NULL && code->callbacks.onNativeWindowCreated != NULL) {
            code->callbacks.onNativeWindowCreated(code,
                    code->nativeWindow.get());
        }
    }
}
//...

onSurfaceCreated_native calls onNativeWindowCreated finally to pass ANativeWindow instance, converted from surface passed from Java part, to android_app_set_window. With previous analyzing, the window surface that NativeActivity.java bound to, will passed to android_app->window, and used for egl and OpenGL ES drawing. There is an official article called EGLSurfaces and OpenGL ES that describes the relationship between egl, OpenGL ES and ANativeWindow, and you can read it you have interested in it.

Custom NativeActivity.java

Actually, we can implement NativeActivity.java and add our custom logic to it. But we should take care about android:hasCode in AndroidManifest.xml’s <application> tag. If hasCode is false, frameworks will not load any custom Java code, and our customized NativeActivity.java will not be used. We must set it to true if we have customized NativeActivity.java or other Java code.

Summary

I only analyze app lifecycle process and surface sharing between Java and native code briefly at this article, and I don’t analyze more detailed content, because skeleton of NativeActivity is enough for me. NativeActivity provides a mechanism to pass Android’s specific lifecycle and input mechanism to app’s native code with NDK’s app glue. NativeActivity make native library Android platform aware and provide the ability to get platform related surface for egl and OpenGL ES drawing for app’s native library. It’s very useful and important for cross-platform GUI apps to add minimum platform related codes to let itself run Android platform too. If you have similar need, NativeActivity can be a candidate solution for you.

This article is very short and just used to show how to test uncompressed assets with Robolectric. What we need do is very simple:

1. Using Robolectric from 4.7(4.7.3 is the best recommended version currently with other big improvements). It contains changes to support opening fd from uncompressed file in apk. We can check Support to open fd for uncompressed file in asset and FileNotFoundException when opening AssetFileDescriptor for more details.
2. Using aaptOptions.noCompress to specify extension of files you don’t want to be compressed. We can check official documentation of AaptOptions#noCompress for more details.
3. Using AGP from 7.1.0-alpha08(including) to support aaptOptions#noCompress for unit tests. Before AGP 7.1.0-alpha08, it has a bug to compress all assets for unit tests, and we can check AGP compresses all assets for unit test .apk regardless of aaptOptions.noCompress for more details.

After that, we can write test code looks like:

@RunWith(AndroidJUnit4.class)
public class ExampleUnitTest {
    @Test
    public void openFd_shouldProvideFileDescriptorForAsset() throws IOException {
        Context context = ApplicationProvider.getApplicationContext();
        AssetManager assetManager = context.getAssets();
        AssetFileDescriptor assetFileDescriptor = assetManager.openFd("assetsHome.txt");
        assertThat(CharStreams.toString(new InputStreamReader(
                assetFileDescriptor.createInputStream(), Charset.forName("UTF-8"))))
                .isEqualTo("assetsHome!");
        assertThat(assetFileDescriptor.getLength()).isEqualTo(11);
    }
}

Taskbar is an awesome Android launcher which supports start app to freefrom windowing mode directly to provide desktop UI for Android user. And it is integrated into Android-x86 and BlissOS as alternative launcher to provide desktop experience for users. If you are not familiar with it, you can visit Taskbar’s Google Play Store page to download and experience it. This article is mainly used to introduce how Taskbar implements this feature, and which requirements of Android system needs to support Taskbar to start app in freeform windowing mode.

It needs Android device supports freeform

At section “3.8.14. Multi-windows” of Android 7.0 CDD, Google needs device implementation with screen size xlarge should support freeform mode:

Device implementations MUST NOT offer split-screen or freeform mode if both the screen height and width is less than 440 dp.

Device implementations with screen size xlarge SHOULD support freeform mode.

At Android 11 CDD section multi-windows, it also keeps this definition and restriction. So if Android device has xlarge screen size, and it is released with CTS passed, we can assume it supports freeform windowing mode, and Taskbar can work correctly to start app with freeform windowing mode. There are also many Android variants to provide desktop experience for users support freeform windowing mode directly, such as Android-x86 and BlissOS.

If you are a ROM developer or Android frameworks developer, and you want to make your device to support freeform windowing mode, you need config_freeformWindowManagement config is set to true at your frameworks/base/core/res/res/values/config.xml. It is recommended to set it with vendor overrides as what I did for boringdroid at its vendor_boringdroid project. You also should copy frameworks/native/data/etc/android.software.freeform_window_management.xml to device system/etc/permissions/android.software.freeform_window_management.xml similar likes boringdroid.mk at vendor_boringdroid

If you are a normal Android user, you can enable Enable freeform windows item at Settings’ Developer Options page by following Android Policy’s article “Freeform windows can be enabled in Android Q without hacks”.

It needs app supports multi-window

Android developer multi-window section shows how app to declare itself that supports multi-window, including freeform windowing mode. It is very important, because not all apps can work correctly when you resizing its window to specific size.

If you are a ROM developer or Android frameworks developer, and you want to force all activities resizable for freeform windowing mode or other multi-window modes, you can modify ActivityManagerService to force all activities resizable similar like boringdroid commit: Make sure all activities support resizable. From Android 10, we also can set display.settings.xml=freeform to your device, and set display to freeform, that will permit any Activity started on this display can enter freeform windowing mode. There is an example from goldfish config.ini.freeform.

If you are an app developer, you can follow official manual to prepare your app for large screens and multi-window to ensure your app can work perfectly when it is in freeform windowing mode and other multi-window modes.

If you are a normal user, and you want to force all activities are resizable and let it can be started into freeform windowing mode by Taskbar, you can enable Make all activities resizable for multi-window, regardless of manifest values similar likes to enable freeform windows forcibly.

Let’s start an app with freeform windowing mode

The following code snippets are both from Taskbar’s U.java.

When we start an app or an Activity, we can assign a Bundle to startActivity method. Android provides another class called ActivityOptions that can be serialized to Bundle to pass launching parameters related to this Activity to frameworks, and it will calculate the final windowing mode based on ActivityOptions and other display and device configs. If your display is freeform, and every Activity started on this display will enter freeform mode, if we don’t specific any fixed fullscreen/split screen/picture in picture windowing mode with ActivityOptions. If you display is not freeform, and the Android devices support freeform, and the Activity will be started supports freeform, we can add freeform windowing mode to ActivityOptions and pass it to startActivity, and frameworks will let this Activity enter freeform windowing mode. It’s very simle and clear, right?

But there are many restrictions for third-party apps, including Taskbar. The first difficulty is that old Android versions use stack id to represent freeform windowing mode at ActivityOptions, but new versions use windowing mode id to represent freeform windowing mode. So Taskbar should make a compatibility to set freeform windowing mode for Activity for different Android versions. The second thing is restriction on non-SDK interfaces. The methods of ActivityOptions to set freeform windowing mode is hidden, and we must use reflection to access them. But from Android 9, Android restricts app to access hidden APIs. Taskbar uses some tricks to bypass this restriction.

Let’s start to dive into Taskbar’s source code to see what it does to overcome those difficulties and let the function be realized.

The first station is U#allowReflection():

public static void allowReflection() {
    GlobalHelper helper = GlobalHelper.getInstance();
    if(helper.isReflectionAllowed()) return;

    try {
        Method forName = Class.class.getDeclaredMethod("forName", String.class);
        Method getDeclaredMethod = Class.class.getDeclaredMethod("getDeclaredMethod", String.class, Class[].class);

        Class<?> vmRuntimeClass = (Class<?>) forName.invoke(null, "dalvik.system.VMRuntime");
        Method getRuntime = (Method) getDeclaredMethod.invoke(vmRuntimeClass, "getRuntime", null);
        Method setHiddenApiExemptions = (Method) getDeclaredMethod.invoke(vmRuntimeClass, "setHiddenApiExemptions", new Class[]{String[].class});

        Object vmRuntime = getRuntime.invoke(null);
        setHiddenApiExemptions.invoke(vmRuntime, new Object[]{new String[]{"L"}});
    } catch (Throwable ignored) {}

    helper.setReflectionAllowed(true);
}

It’s main work is to use reflection to get VMRuntime#setHiddenApiExemptions without restriction, and pass L as input to exempt all hidden APIs. Actually this method only work correctly on API 29 and earlier. If you also want to bypass this restriction on API 30, you can check StackOverflow’s question: Bypass Android’s hidden API restrictions.

Now, Taskbar can use reflection to get hidden fields and call hidden methods. The next station is U#getFreeformWindowModeId():

// From android.app.ActivityManager.StackId
private static final int FULLSCREEN_WORKSPACE_STACK_ID = 1;
private static final int FREEFORM_WORKSPACE_STACK_ID = 2;

// From android.app.WindowConfiguration
private static final int WINDOWING_MODE_FULLSCREEN = 1;
private static final int WINDOWING_MODE_FREEFORM = 5;

private static int getFreeformWindowModeId() {
    if(getCurrentApiVersion() >= 28.0f)
        return WINDOWING_MODE_FREEFORM;
    else
        return FREEFORM_WORKSPACE_STACK_ID;
}

U#getFreeformWindowModeId() is used to get freeform windowing mode that defined in Android frameworks. Taskbar doesn’t use reflection to get those hidden fields from ActivityManager or WindowConfiguration, and copy their values to its source code directly. The U#getFreeformWindowModeId() will return different values for different Android versions.

When it gets correct freeform windowing mode id, and it will pass it to ActivityOptions at U#getActivityOptions(Context context, ApplicationType applicationType, View view):

if(stackId != -1) {
    allowReflection();
    try {
        Method method = ActivityOptions.class.getMethod(getWindowingModeMethodName(), int.class);
        method.invoke(options, stackId);
    } catch (Exception ignored) {}
}

The U#getActivityOptions(Context context, ApplicationType applicationType, View view) uses U#allowReflection() shown above to access hidden API of ActivityOptions. It use U#getWindowingModeMethodName() to get API to pass freeform windowing mode id for different Android versions:

private static String getWindowingModeMethodName() {
    if(getCurrentApiVersion() >= 28.0f)
        return "setLaunchWindowingMode";
    else
        return "setLaunchStackId";
}

Those APIs are not exposed publicly, so Google can change it if need. The user such as Taskbar must make itself methods compatible with different Android versions.

After setting correct freeform windowing mode id for ActivityOptions, the next station is to set launch bounds or window bounds to Activity’s ActivityOptions. It is at getActivityOptionsBundle(Context context,ApplicationType applicationType, View view, int left, int top, int right, int bottom):

ActivityOptions options = getActivityOptions(context, applicationType, view);
if(options == null) return null;

if(Build.VERSION.SDK_INT < Build.VERSION_CODES.N)
    return options.toBundle();

return options.setLaunchBounds(new Rect(left, top, right, bottom)).toBundle();

The launch bounds is the initial bounds of Activity’s window after it started. Actually, Taskbar can’t get final bounds before window closed. So it uses default bounds for Activity when every start. To fix this problem, I have added a commit Support persist window bounds to ignore launch bounds from ActivityOptions and keep bounds at frameworks/base.

Summary

It’s very clear how Taskbar start app with freeform windowing mode. If app and system supports freeform windowing mode, it uses reflection to get freeform windowing mode id and pass it to ActivityOptions. And it will use startActivity with Bundle generated from ActivityOptions to tell frameworks to start specific Activity or app to freeform windowing mode. When we start an app from Launcher, we actually start its main Activity, so there some mix-uses between Activity and app. If you’re clear about it, just going head to implement your custom launcher to start app in freeform windowing mode.

MaruOS is a project that leverages Android mobile phone’s computing unit to run Linux OS, especially Debian now, on Android OS with lxc. It uses its mflinger to allocate buffer from Android gralloc allocator, and then uses its mclient in Linux container to mmap this buffer and draw content from xf86-video-dummy to this buffer to show the Linux desktop to Android. But there are some Android’s gralloc allocator implementations doesn’t expose write permission to exposing buffer without using gralloc mapper. And Linux container can’t use gralloc mapper directly and easily. If we can use xserver implementation in Android to show x11 window on Android with the normal app buffer, we can fix this problem. So I start to test android-xserver with MaruOS, and it works fine. This article is used to show steps to using android-xserver with MaruOS.

Test environment

I use maru-0.7 with my phone Pixel XL (codename is marlin). Syncing maru-0.7 source code, preparing vendor binaries, building maru_marlin-userdebug, and flashing phone.

Run android-xserver

Following the README.md of android-xserver, building and running it on Pixel XL.

Setup xserver on Linux container

Firstly, we should find the local ip of the Pixel XL with command adb shell ifconfig:

marlin:/ # ifconfig
...
wlan0     ...
          inet addr:192.168.0.109  Bcast:192.168.0.255  Mask:255.255.255.0 
...

The local ip is 192.168.0.109.

And then we should use adb root to get root permission of the phone, and use the following command to start Linux container from adb shell:

adb shell
lxc-start --name default

The default username and password is maru and maru.

We should ensure lwm and xfe are installed on Linux container, and install them with the following commands if not:

sudo apt update
sudo apt install lwm xfe

The next step we should do is to assign the local ip as the xserver location to the x11:

maru@buster-container:~$ export DISPLAY=192.168.0.109:0
maru@buster-container:~$ echo $DISPLAY
192.168.0.109:0

Now, we can start lwm and xfe:

lwm &
xfe

Waiting some seconds, we can see the xfe window on the android-xserver:

If you are a ROM developer/maintainer, you maybe need to add some setting item to the official settings app for your custom configuration. For example, I need to add switch for user to enable/disable multi-window and BoringdroidSystemUI based on his/her need. If I add those switch to official settings app, I will manage the fork of official settings app, and apply patches when I upgrade based AOSP version, for example, upgrade from Android 10 to Android 11. So if there is a mechanism to plugin custom settings app to official settings app, it will help to release myself from annoying patch work. Fortunately, the official settings app provide a mechanism called for EXTRA_SETTINGS to help us to make it come true. The following first diagram is the official settings app dashboard page, and it loads the BoringdroidSettings dynamically, and the second diagram is the result after clicking the BoringdroidSettings dashboard entry, the shown BoringdroidSettings app page.

Code base

AOSP 10.0

Configure custom settings app with EXTRA_SETTINGS

It is very simple, the following code snippet is the example of BoringdroidSettings.

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    coreApp="true"
    package="com.boringdroid.settings"
    android:sharedUserId="android.uid.system">

    <application
        android:name=".BoringdroidSettingsApplication"
        android:allowBackup="false"
        android:defaultToDeviceProtectedStorage="true"
        android:hardwareAccelerated="true"
        android:icon="@drawable/ic_icon"
        android:label="@string/ic_name"
        android:requiredForAllUsers="true"
        android:supportsRtl="true"
        android:taskAffinity=""
        android:theme="@style/Theme.Settings"
        android:usesCleartextTraffic="true">

        <activity
            android:name=".BoringdroidSettings"
            android:exported="true"
            android:icon="@drawable/ic_icon"
            android:label="@string/ic_name">
            <intent-filter>
                <action android:name="com.android.settings.action.EXTRA_SETTINGS" />
            </intent-filter>

            <meta-data
                android:name="com.android.settings.category"
                android:value="com.android.settings.category.ia.homepage" />
            <meta-data
                android:name="com.android.settings.title"
                android:value="@string/boringdorid_dashboard_title" />
            <meta-data
                android:name="com.android.settings.summary"
                android:value="@string/boringdorid_dashboard_summary" />
        </activity>
    </application>
</manifest>

Firstly, we should add EXTRA_SETTINGS intent-filter for dashboard entry activity:

<intent-filter>
    <action android:name="com.android.settings.action.EXTRA_SETTINGS" />
</intent-filter>

And then we should add critical meta-data list for dashboard entry activity:

<meta-data
    android:name="com.android.settings.category"
    android:value="com.android.settings.category.ia.homepage" />
<meta-data
    android:name="com.android.settings.title"
    android:value="@string/boringdorid_dashboard_title" />
<meta-data
    android:name="com.android.settings.summary"
    android:value="@string/boringdorid_dashboard_summary" />

We must set com.android.settings.category with com.android.settings.category.ia.homepage to tell the official settings app, the activity wants to be added to dashboard. From the code, we also can use different category values to add activity to other sub page, but I don’t check it, if you need, you can try it.

We also should set com.android.settings.title and com.android.settings.summary to customize your shown title and summary.

The next step is to set android:icon="@drawable/ic_icon" for the activity that will be shown on dashboard.

The last what we should do is the set coreApp and shareUid, that the official settings will check.

After above steps, our custom activity will shown on official settings app dashboard page, and will be started after the user clicks it.

Develop custom settings app

If we can get system sign key, we can use gradle to develop custom settings app, and use Android Studio to build and install to our emulator or other physical devices without key problem. What we should do is to import androidx or other preference libraries to develop settings app. The BoringdroidSettings uses this way to develop normal functions. If you want to use Android.mk or Android.bp as build tool, we also can use androidx preference libraries to develop custom settings app, because the AOSP has those prebuilt libraries. The BoringdroidSettings also supports Android.bp for Boringdroid system build. Another example is MaruSettings.

How does EXTRA_SETTINGS work?

com.android.settingslib.drawer.TileUtils defines the EXTRA_SETTINGS:

/**
 * Settings will search for system activities of this action and add them as a top level
 * settings tile using the following parameters.
 *
 * <p>A category must be specified in the meta-data for the activity named
 * {@link #EXTRA_CATEGORY_KEY}
 *
 * <p>The title may be defined by meta-data named {@link #META_DATA_PREFERENCE_TITLE}
 * otherwise the label for the activity will be used.
 *
 * <p>The icon may be defined by meta-data named {@link #META_DATA_PREFERENCE_ICON}
 * otherwise the icon for the activity will be used.
 *
 * <p>A summary my be defined by meta-data named {@link #META_DATA_PREFERENCE_SUMMARY}
 */
public static final String EXTRA_SETTINGS_ACTION = "com.android.settings.action.EXTRA_SETTINGS";

From the comment, we know it is used to search system activities, that will be added as a top level settings tile. The comment also describe the meta-datas to set title/icon/summary, what we set above.

The TileUtils.getCategories retrieves all tiles with specific actions, including EXTRA_SETTINGS. It calls the TileUtils.getTilesForAction to retrieve the EXTRA_SETTINGS meta-datas.

@VisibleForTesting
static void getTilesForAction(Context context,
        UserHandle user, String action, Map<Pair<String, String>, Tile> addedCache,
        String defaultCategory, List<Tile> outTiles, boolean requireSettings) {
    final Intent intent = new Intent(action);
    if (requireSettings) {
        intent.setPackage(SETTING_PKG);
    }
    final PackageManager pm = context.getPackageManager();
    List<ResolveInfo> results = pm.queryIntentActivitiesAsUser(intent,
            PackageManager.GET_META_DATA, user.getIdentifier());
    for (ResolveInfo resolved : results) {
        if (!resolved.system) {
            // Do not allow any app to add to settings, only system ones.
            continue;
        }
        ActivityInfo activityInfo = resolved.activityInfo;
        Bundle metaData = activityInfo.metaData;
        String categoryKey = defaultCategory;

        // Load category
        if ((metaData == null || !metaData.containsKey(EXTRA_CATEGORY_KEY))
                && categoryKey == null) {
                Log.w(LOG_TAG, "Found " + resolved.activityInfo.name + " for intent "
                    + intent + " missing metadata "
                    + (metaData == null ? "" : EXTRA_CATEGORY_KEY));
            continue;
        } else {
            categoryKey = metaData.getString(EXTRA_CATEGORY_KEY);
        }
        // other code
    }
}

It skips non-system processes:

if (!resolved.system) {
    // Do not allow any app to add to settings, only system ones.
    continue;
}

So we should add shareUid to share system process.

It also skip component without EXTRA_CATEGORY_KEY meta-data, so we also must set it.

In com.android.settings.dashboard.DashboardFragment, it has multi entries to com.android.settings.dashboard.DashboardFeatureProviderImpl.getTilesForCategory to get tiles with specific category key. Different dashboard fragment has different key, for example, the top level dashboard of official settings app use the key com.android.settings.category.ia.homepage, defined by com.android.settingslib.drawer.CategoryKey.CATEGORY_HOMEPAGE. The relationship between key and dashboard fragment instance is defined by com.android.settings.dashboard.DashboardFragmentRegistry:

static {
    PARENT_TO_CATEGORY_KEY_MAP = new ArrayMap<>();
    PARENT_TO_CATEGORY_KEY_MAP.put(TopLevelSettings.class.getName(),
            CategoryKey.CATEGORY_HOMEPAGE);
    PARENT_TO_CATEGORY_KEY_MAP.put(
            NetworkDashboardFragment.class.getName(), CategoryKey.CATEGORY_NETWORK);
    // other code
}

And the above code also tells use that we can change category key value to add custom setting activity to specific sub dashboard, such as system dashboard.

After that, the DashboardFragment will use Tile to show all items with the same category key. The left things are normal clicking processing, if you want to learn it, just reading the code.

Summary

With the EXTRA_SETTINGS, we can let official settings app to plugin our custom settings app into it very simple. And we can focus on develop custom settings content, and get rid of porting work when upgrading based AOSP version. And it supports to add different type setting activity to different dashboard page, such as top level, system, network, and etc. If you want to arrange settings item for official settings app, it is not suitable for you. And it is useful for people only wants to add custom setting items to official settings app.

That’s all. Enjoy it.

The manual has been moved to Bliss Wiki, that will be maintained.

This article is a draft content to describe how to install Bliss OS on VirtualBox. It combines android-x86 VirtualBox howto and Bliss OS Install Bliss OS on a VM (virtualbox) part.

Create a new VM

If you have not already created a VirtualBox virtual machine for Bliss OS yet, do so as follows:

1. Click the “New” button, and name your new virtual machine however you like. Set Type to Linux, and Version to Linux 2.6 / 3.x / 4.x. Note that you should choose the appropriate bit type for the version of Bliss OS that you downloaded.
2. Specify how much RAM will be allocated to your virtual machine when you run it. Android doesn’t specify a bare-minimum requirement for memory, just keep in mind what apps you plan on running. 2GB (2048MB) is a good place to start, and you can change this later if you need to.
3. Create a new Hard disk image which will act as your machine’s storage. The recommended starting size of 8GB is enough. Click through the rest of the options for creating your Hard disk.

Your virtual machine has now been created. It still needs to be initially installed at this point.

Settings

Tested on VirtualBox 64-bit for Ubuntu 20.04 with Intel and AMD (Ryzen R7 4800H) processors, version 6.1. Bliss OS version v11.11, 64-bit.

Select your machine, then click the Settings button and refer to the below recommended configuration to make sure your settings match.

1. [System] Recommended: Processor(s) should be set above 1 if you have more than one virtual processor in your host system. Failure to do so means every single app (like Google Chrome) might crush if you try to use it.

2. [Display]:

   2.1. Optional: Video Memory may be increased beyond the minimum selected automatically. The affects of this are unknown.

   2.2 Mandatory: Unless guest additions are installed, change the default VMSVGA to VBoxVGA.

   2.3 Optional: Enable 3D Acceleration may be checked. The Linux Guest Additions must (VirtualBox v6.1+) / may (VirtualBox v6.0 and below) need to be installed to get any benefit from this.

   Failure to do so means you won’t even be able to launch Bliss OS in the first place.

3. [Storage] Find the first “Empty” item (this should have an icon of a CD). In the Attributes, click on the CD icon with a small down arrow, and pick “Choose Optical Virtual Disk File…”. Specify the Bliss OS ISO that you downloaded.

4. [Audio] Intel HD Audio seems to be natively supported in Bliss OS.

5. [Network] By default, your installation of Bliss OS will be able to automatically connect to the internet. If not, you can try to enable WiFi in “Settings/Network & Internet”, and connect to showing VirtWifi. If you do not want to connect to the internet in VirtualBox, uncheck “Enable Network Adapter” under the Adapter 1 tab.

Install

Now we can click the start on VirtualBox to start the VM.

When entering installation page,

1. select “Installation - Install Bliss-OS to harddisk”.
2. Press D to detect devices.
3. Press C to create and modify partitions. If asked, select “No” for GPT.
4. Select “New”, “Primary”, “full size”, and then make it bootable.
5. “Write to disk”.
6. “Quit”.
7. Select the partition and then reformat to ext4.
8. Select “Yes” to install GRUB.
9. Select “Yes” to make the /system directory writable.

After than, just closing the installing window, and restart this VM instance, and select “Advanced Options” and “Boot from local device” to select Bliss OS with different options to start.

To here, you should have a fully working Bliss OS install in a VM, or at least something that works… even if it may be slow. Again, Android on VM is generally not a good idea and you will get a lot more performance if you install Bliss OS to the actual hardware.

This article based on AOSP 9.0.

PIP is another multi-window feature I used very much when watching videos. This article will analyze its logic to show its details.

Enter picture-in-picture(pip)

We can enter pip by API Activity.enterPictureInPictureMode. If the activity supports pip, it will call ActivityManagerService.enterPictureInPictureMode to do the real work.

ActivityManagerService.enterPictureInPictureMode

// Adjust the source bounds by the insets for the transition down
final Rect sourceBounds = new Rect(r.pictureInPictureArgs.getSourceRectHint());
mStackSupervisor.moveActivityToPinnedStackLocked(r, sourceBounds, aspectRatio,
        "enterPictureInPictureMode");
final PinnedActivityStack stack = r.getStack();
stack.setPictureInPictureAspectRatio(aspectRatio);
stack.setPictureInPictureActions(actions);

The ActivityManagerService.enterPictureInPictureMode will move activity to pinned stack, and then update PinnedActivityStack states. The PinnedActivityStack is the implementation of ActivityStack, and will process specific logic for pip.

ActivityStackSupervisor.moveActivityToPinnedStackLocked

PinnedActivityStack stack = display.getPinnedStack();
if (stack != null) {
    moveTasksToFullscreenStackLocked(stack, !ON_TOP);
}
// Other code
resizeStackLocked(stack, task.getOverrideBounds(), null /* tempTaskBounds */,
        null /* tempTaskInsetBounds */, !PRESERVE_WINDOWS,
        true /* allowResizeInDockedMode */, !DEFER_RESUME);

if (task.mActivities.size() == 1) {
    task.reparent(stack, ON_TOP, REPARENT_MOVE_STACK_TO_FRONT, !ANIMATE, DEFER_RESUME,
            false /* schedulePictureInPictureModeChange */, reason);
} else {
    final TaskRecord newTask = task.getStack().createTaskRecord(
            getNextTaskIdForUserLocked(r.userId), r.info, r.intent, null, null, true);
    r.reparent(newTask, MAX_VALUE, "moveActivityToStack");
    newTask.reparent(stack, ON_TOP, REPARENT_MOVE_STACK_TO_FRONT, !ANIMATE,
            DEFER_RESUME, false /* schedulePictureInPictureModeChange */, reason);
}

The ActivityStackSupervisor.moveActivityToPinnedStackLocked will move all tasks in pinned stack to fullscreen, and resize pinned stack to the assigned bounds. It also reparents task of activity to the pinned stack. If origin task has multiple activities, it will move top activity to pinned stack, and keep other activities in origin task.

The following screenshot is an example:

Input

In system ui, there is a system ui service called PipUI to process the ui of pip. The PipUI is entry for PipManager to receive command from frameworks.

Pip input consumer

The PipManager will create an instance of InputConsumerController called mInputConsumerController, to manage pip input consumer to receive the input from input flinger with its registerInputConsumer.

PipManager.initialize

mInputConsumerController = InputConsumerController.getPipInputConsumer();
mInputConsumerController.registerInputConsumer();

InputConsumerController.registerInputConsumer
            
final InputChannel inputChannel = new InputChannel();
try {
    mWindowManager.destroyInputConsumer(mName);
    mWindowManager.createInputConsumer(mToken, mName, inputChannel);
} catch (RemoteException e) {
    Log.e(TAG, "Failed to create input consumer", e);
}
mInputEventReceiver = new InputEventReceiver(inputChannel, Looper.myLooper());
if (mRegistrationListener != null) {
    mRegistrationListener.onRegistrationChanged(true /* isRegistered */);
}

In InputMonitor.accept, it will use pip stack bounds to restrict pip input consumer touchable region:

InputMonitor.accept
            
if (w.inPinnedWindowingMode()) {
    if (mAddPipInputConsumerHandle
            && (inputWindowHandle.layer <= pipInputConsumer.mWindowHandle.layer)) {
        // Update the bounds of the Pip input consumer to match the window bounds.
        w.getBounds(mTmpRect);
        pipInputConsumer.mWindowHandle.touchableRegion.set(mTmpRect);
        addInputWindowHandle(pipInputConsumer.mWindowHandle);
        mAddPipInputConsumerHandle = false;
    }
    // Other code
}

The pip consumer has the fixed name called pip_input_consumer, so InputMonitor can retrieve pip consumer by name.

The InputMonitor.createInputConsumer will set FLAG_NOT_TOUCH_MODAL to pip window handle layout params flags to ensure other events fall through, and pip input consumer target consumes all events for pip/pinned stack.

InputMonitor.createInputConsumer

case INPUT_CONSUMER_PIP:
    // The touchable region of the Pip input window is cropped to the bounds of the
    // stack, and we need FLAG_NOT_TOUCH_MODAL to ensure other events fall through
    consumer.mWindowHandle.layoutParamsFlags |= FLAG_NOT_TOUCH_MODAL;
    break;

When InputConsumerController.InputEventReceiver receives the input events from input flinger, it will call InputConsumerController.TouchListener to notify the input events, that set by inputConsumerController.setRegistrationListener in PipTouchHandler.PipTouchHandler.

If we click the pip window as normal, it will show pip menu by starting PipMenuActivity over the pip window, looks like following screenshot:

If we do nothing after showing pip menu some time, it will dismiss the pip menu.

If we double tap the pip window, it will dismiss pip, and move tasks in pip/pinned stack to fullscreen stack by calling ActivityManagerService.dismissPip:

ActivityManagerService.dismissPip

if (animate) {
    stack.animateResizePinnedStack(null /* sourceHintBounds */,
            null /* destBounds */, animationDuration, false /* fromFullscreen */);
} else {
    mStackSupervisor.moveTasksToFullscreenStackLocked(stack, true /* onTop */);
}

KEYCODE_WINDOW

For key event, in PhoneWindowManager.interceptKeyBeforeQueueing, it will response to KEYCODE_WINDOW to show pip menu when pip is visible by PhoneWindowManager.showPictureInPictureMenu. And PipManager.showPictureInPictureMenu will process it.

Focusable

In WindowConfiguration.canReceiveKeys, it will not allow pip/pinned stack to receive keys:

WindowConfiguration

public boolean canReceiveKeys() {
    return mWindowingMode != WINDOWING_MODE_PINNED;
}

And this method will be used to check whether one ConfigurationContainer is focusable.

ActivityStackSupervisor.isFocusable

return container.getWindowConfiguration().canReceiveKeys() || alwaysFocusable;

AppWindowToken

boolean windowsAreFocusable() {
    return getWindowConfiguration().canReceiveKeys() || mAlwaysFocusable;
}

So if the pip/pinned window doesn’t set it always focusable(set FLAG_ALWAYS_FOCUSABLE), it is not focusable.

Always on top

In WindowConfiguration.isAlwaysOnTop, it will set pip/pinned stack always on top.

WindowConfiguration

public boolean isAlwaysOnTop() {
    return mWindowingMode == WINDOWING_MODE_PINNED;
}

For example, in DisplayContent.positionChildAt, it will set pip/pinned stack to the top over other stacks in this display:

DisplayContent.positionChildAt

if (child.getWindowConfiguration().isAlwaysOnTop()
        && position != POSITION_TOP) {
    // This stack is always-on-top, override the default behavior.
    Slog.w(TAG_WM, "Ignoring move of always-on-top stack=" + this + " to bottom");

    // Moving to its current position, as we must call super but we don't want to
    // perform any meaningful action.
    final int currentPosition = mChildren.indexOf(child);
    super.positionChildAt(currentPosition, child, false /* includingParents */);
    return;
}

Receive pip stack event from ActivityManagerService

In PipManager, there is an TaskStackChangeListener implementation called mTaskStackListener. In PipManager.initialize, it will register mTaskStackListener to ActivityManagerService to receive pip stack event.

PipManager.initialize

ActivityManagerWrapper.getInstance().registerTaskStackListener(mTaskStackListener);

Summary

The Activity.enterPictureInPictureMode will notify ActivityManagerService to move activity to pinned stack. If there are tasks in pinned stack, the AMS will move existing tasks to fullscreen stack. If the task of activity exists has multiple activities, it will keep other activities to origin task, and create new task for pip/pinned activity.

The pip/pinned window can’t receive keys, and system ui will create pip input consumer to receive input for pip/pinned stack. When input event received, pip ui will show PipMenuActivity to display pip menu. If we double tap the pip/pinned window, it will dismiss pip and move tasks in pip stack to fullscreen stack through ActivityManagerService.

The pip ui also uses TaskStackChangeListener to receive pip event from ActivityManagerService.

This article based on AOSP 9.0.

Split screen is my best favorite multi-window feature when I use Android phone. This article will analyze its logic show how this awesome feature is implemented.

SystemUI

The split screen’s entry is in system ui’s recents. In RecentsView, if the user select one app to start in split screen mode with ui choice, RecentsViewl.onBusEvent will try to start the task of this app to split screen mode.

RecentsView.onBusEvent

final ActivityOptions options = ActivityOptionsCompat.makeSplitScreenOptions(
    dockState.createMode == SPLIT_SCREEN_CREATE_MODE_TOP_OR_LEFT);
if (ActivityManagerWrapper.getInstance().startActivityFromRecents(event.task.key.id,
        options)) {
    // Other code
}

The ActivityManagerWrapper.startActivityFromRecents calls ActivityManagerService.startActivityFromRecents directly.

ActivityManagerService

The ActivityManagerService.startActivityFromRecents calls ActivityStackSupervisor.startActivityFromRecents directly. The starting activity from recents logic is the same as normal starting activity in most occasion. We will show the important points for recents and split screen.

Restore task from recent_tasks files

The RecentsView passes the task id of app to the ActivityManagerService, and ActivityStackSupervisor.startActivityFromRecents will try to restore the task with the specified id from recent_tasks if the task with the same id doesn’t exist. The work is done by ActivityStackSupervisor.anyTaskForIdLocked:

ActivityStackSupervisor.anyTaskForIdLocked

final TaskRecord task = mRecentTasks.getTask(id);
// Other code
if (!restoreRecentTaskLocked(task, aOptions, onTop)) {
    if (DEBUG_RECENTS) Slog.w(TAG_RECENTS,
            "Couldn't restore task id=" + id + " found in recents");
    return null;
}

RecentTasks.getTask

TaskRecord getTask(int id) {
    final int recentsCount = mTasks.size();
    for (int i = 0; i < recentsCount; i++) {
        TaskRecord tr = mTasks.get(i);
        if (tr.taskId == id) {
            return tr;
        }
    }
    return null;
}