Docs

System Overview & Architecture

This page describes CROWDio’s high-level design, component responsibilities, interaction model, technology stack, and design principles.

---

What is CROWDio?

CROWDio is a distributed computing platform designed to aggregate and orchestrate computational resources from Android-based smart mobile devices.

It targets large-scale, heterogeneous mobile environments where devices differ in CPU capacity, memory, battery level, and network conditions. The system follows a master–worker (coordinator–executor) architectural pattern, enabling scalable task execution, fault tolerance, and adaptive scheduling.

System goals:

• Heterogeneity-awareness (CPU, memory, network, battery)
• Fault tolerance through declarative checkpointing with resume
• Pluggable scheduling strategies (FIFO → MCDM)
• Low-latency communication using persistent WebSockets
• Minimal developer overhead via SDK annotations

---

High-Level Component Diagram

graph TB
    subgraph Client["Client Application"]
        SDK["developer_sdk/api.py\nCrowdComputeClient"]
    end

    subgraph Foreman["Foreman (Orchestrator)"]
        API["REST API\nFastAPI :8000"]
        WS["WebSocket Server\n:9000"]
        SCH["Scheduler\nFIFO / RR / LL / MCDM"]
        JM["Job Manager"]
        CM["Checkpoint Manager"]
        DB["SQLite\ncrowdcompute.db"]
        DASH["Dashboard\nHTML UI"]
    end

    subgraph Workers["Workers"]
        PCW["PC Worker\npc_worker/main.py"]
        MOB["Android Worker\nMobile App + Chaquopy"]
    end

    SDK -- WebSocket :9000 --> WS
    WS --> JM
    JM --> SCH
    SCH --> JM
    JM --> DB
    JM --> CM
    CM --> DB
    WS -- assign_task --> PCW
    WS -- assign_task --> MOB
    PCW -- task_result / checkpoint --> WS
    MOB -- task_result / checkpoint --> WS
    API --> DB
    API --> DASH

---

Three Primary Subsystems

Subsystem	Description
Developer SDK	Declarative task submission; abstracts connection, serialization, job submission, result aggregation
Foreman (control plane)	Single source of truth for job state, worker availability, checkpoints, scheduling decisions
Workers (execution plane)	Stateless execution agents — perform tasks, report progress, emit checkpoints

---

Control Plane vs. Data Plane

Plane	What it carries	Interface
Control plane	REST commands, dashboard, admin, config	HTTP localhost:8000
Data plane	Job submissions, task assignments, results, heartbeats, checkpoints	WebSocket localhost:9000

---

State Machine — Job Lifecycle

stateDiagram-v2
    [*] --> PENDING: Client submits job
    PENDING --> RUNNING: Scheduler dispatches first task
    RUNNING --> RUNNING: Tasks executing in parallel
    RUNNING --> PARTIAL: Some tasks failed (retrying)
    PARTIAL --> RUNNING: Retry dispatched
    RUNNING --> COMPLETED: All tasks succeeded
    RUNNING --> FAILED: Fatal error, no retry
    COMPLETED --> [*]
    FAILED --> [*]

State Machine — Task Lifecycle

stateDiagram-v2
    [*] --> QUEUED: Job created
    QUEUED --> ASSIGNED: Dispatcher sends assign_task
    ASSIGNED --> RUNNING: Worker begins execution
    RUNNING --> COMPLETED: task_result received
    RUNNING --> FAILED: task_error received (retries exhausted)
    RUNNING --> CHECKPOINTED: task_checkpoint emitted
    CHECKPOINTED --> RUNNING: Checkpoint stored, continues
    FAILED --> ASSIGNED: Retry — resume_task with checkpoint
    COMPLETED --> [*]
    FAILED --> [*]

---

Database Schema (Summary)

Table	Key Columns
jobs	id, status, created_at, completed_at, client_id
tasks	id, job_id, status, worker_id, attempt, result
workers	id, type, platform, status, last_heartbeat
checkpoints	id, task_id, sequence, type (BASE/DELTA/COMPACTED), data_hex
recovery_events	id, task_id, worker_id, recovered_at, method

---

Technology Stack

Component	Technology	Rationale
Foreman server	Python + FastAPI	Async-first, OpenAPI auto-docs, high developer productivity
WebSocket layer	websockets library	Persistent bidirectional real-time communication
Persistence	SQLite + SQLAlchemy	Lightweight, embedded, zero-config — suitable for local deployments
Worker communication	JSON over WebSocket	Human-readable, versioned, schema-flexible
Android worker runtime	Kotlin + Chaquopy	Native Android service layer + embedded CPython runtime
Python execution (Android)	Chaquopy	Execute user-defined Python on Android without native code
Client SDK	Python asyncio	Non-blocking, compatible with modern async workflows
Scheduling	Custom MCDM engine	Pluggable strategy pattern; FIFO/RR/EDAS/ARAS/MABAC
Function serialization	Source-code text	Portable; workers remain lightweight and SDK-dependency-free

---

Design Principles

Principle	Description
Asynchronous-First	All I/O is non-blocking via Python asyncio and Kotlin coroutines
Pluggability	Scheduling strategies and policies are interchangeable at runtime
Declarative Control	Developers specify fault-tolerance behavior via SDK annotations
Incremental Checkpointing	Delta-based persistence minimizes network and storage overhead
Heterogeneity Awareness	Scheduling adapts to diverse device capabilities in real time
Fail-Safe Recovery	Tasks resume automatically after failure — no developer intervention
Minimal Intrusion	SDK design requires zero changes to user function signatures