Docs

Evaluation & Experiments

CROWDio includes a built-in evaluation framework for benchmarking the distributed system under various conditions. This page documents objectives, setup, metrics, and full experimental results.

4.1 Objective of Evaluation

The evaluation phase validates the performance, scalability, and fault-tolerance characteristics of the CROWDio framework under heterogeneous device environments. Specifically, the evaluation aims to:

Assess the performance impact of checkpointing — measure execution time with/without checkpointing and quantify overhead under different checkpoint frequencies.
Analyze checkpoint frequency trade-offs — compare 1-second, 2-second, 5-second intervals and identify the balance between performance overhead and recovery granularity.
Evaluate system scalability — observe performance as workload size increases.
Measure throughput — analyze task completion rate and degradation under larger workloads.
Assess performance under heterogeneous devices — evaluate behavior across 5 Android workers of varying capability.
Validate across workload types — CPU-intensive, AI/NLP, and data-parallel.

4.2 Experimental Setup

4.2.1 System Architecture Overview

The environment consists of:

Component	Description
Foreman	Central orchestrator — task partitioning, scheduling, checkpoint management, result aggregation, worker monitoring
Android Workers	6 heterogeneous Android devices executing distributed subtasks
Developer SDK	Job submission, checkpoint config, result callbacks, and task partition logic

4.2.2 Hardware Configuration

All distributed experiments used 6 Android devices with intentionally heterogeneous performance characteristics to simulate real Mobile Crowd Computing (MCdC) environments:

Device	Cores	CPU Frequency	RAM
Samsung Galaxy A34	8	2.0 GHz	7.3 GB
Samsung Galaxy A32	8	1.8 GHz	5.5 GB
Samsung Galaxy A51	8	1.74 GHz	7.4 GB
Motorola E40	8	1.82 GHz	3.4 GB
Samsung Galaxy S6 Lite	8	2.0 GHz	3.6 GB
Samsung Galaxy A9+	8	1.8 GHz	3.3 GB

!!! note “Single-Device Baseline” Single-device baseline experiments were run using Termux (a Linux-based terminal emulator for Android), which provides a lightweight, controlled execution environment with minimal background interference — enabling precise benchmarking of CPU-bound workloads isolated from framework overhead.

4.2.3 Workloads Evaluated

(1) Monte Carlo Simulation — CPU-bound

Task: Estimation of Euler’s number (e)
Trial sizes: 1M, 2M, 5M, 10M, 50M, 100M iterations
Purpose: Evaluate scalability and checkpoint overhead

(2) Sentiment Analysis — AI/NLP

Dataset: Help Desk Tickets dataset (Mendeley Data, V2)
Task: Text sentiment classification across independently processed batches
Purpose: Evaluate distributed AI task execution

(3) Image Processing — Data-Parallel

Dataset: Kaggle Weather Image Dataset (6,862 images)
Task: Tile-based image filtering (sharpen, blur, edge detection, grayscale, contrast)
Processing: 5 batches, each distributed to one worker
Purpose: Evaluate embarrassingly parallel workloads

4.3 Evaluation Metrics

Metric	Formula / Description
Job Execution Time (s)	Total wall-clock time to complete the full job
Throughput (tasks/s)	`Total Tasks ÷ Execution Time`
Jain’s Fairness Index	`J(x) = (Σxᵢ)² ÷ (n · Σxᵢ²)` where xᵢ = load on worker i, n = workers
Average Task Time (s)	Mean time per individual task across all workers

Jain’s Fairness Index interpretation: Value close to 1 = highly balanced distribution. Value close to 0 = highly unequal distribution.

5. Results and Analysis

All reported execution times were averaged over 10 independent runs to ensure statistical reliability and minimize the impact of transient fluctuations in device performance and network conditions.

5.1 Scheduling Algorithm Evaluation

5.1.1 Experimental Setup

Cluster: 6 Android devices with varying CPU capability
Jobs: Each job decomposed into 2 subtasks
Objective: Minimize total execution latency via intelligent device selection

5.1.2 Algorithm Overview

FIFO (Baseline)

Tasks dispatched strictly in arrival order. No evaluation of worker state, battery, or CPU. Can cause head-of-line blocking where a high-performance device idles while a slow device processes the front-of-queue task.

WRR — Weighted Round Robin (MCDM)

Dynamically assigns a suitability score to each of the 6 Android devices based on real-time metrics transmitted via HEARTBEAT (CPU load, battery level, available RAM, network latency). Tasks are routed to devices with higher scores.

5.1.3 Results — FIFO vs. WRR

Trials (Monte Carlo)	Single Device (no framework)	FIFO Baseline	WRR (MCDM)	Performance Delta
1 Million	2.058 s	2.0 s	1.2 s	+40%
10 Million	6.128 s	6.2 s	5.5 s	+11.3%
100 Million	37.883 s	113.4 s	48.9 s	+56%

5.1.4 Analysis

Load Level	Observation
Low (1M)	WRR reduced latency by 40% — intelligent device selection is effective even for small tasks
Medium (10M)	Minor dip (+12%) — MCDM decision overhead briefly exceeded routing benefits at this scale
High (100M)	WRR achieved 56% reduction (48.9s vs 113.4s). FIFO suffered severe bottlenecks due to blind assignment to underpowered devices

!!! tip “Key Insight” FIFO consistently underperforms on heterogeneous clusters. MCDM-based scheduling is essential for optimizing throughput when devices have varying capabilities.

5.2 Checkpointing Evaluation — Monte Carlo (Euler’s Number Estimation)

5.2.1 Effect of Checkpointing on Execution Time (1M Trials)

Mode	Execution Time
Checkpointing disabled	2.058 s
Enabled — 5 s interval	4.136 s
Enabled — 2 s interval	4.897 s
Enabled — 1 s interval	4.305 s
Enabled — 0.5 s interval	4.939 s

Checkpointing overhead is ~2–3 seconds — small and practical. Overhead increases with higher frequency due to state serialization, network transmission, and storage costs.

5.2.2 Throughput Analysis

Trials	Execution Time	Throughput
1M (no checkpoint)	2.058 s	0.18 tasks/s
1M (5 s interval)	4.136 s	0.17 tasks/s
1M (2 s interval)	4.897 s	0.17 tasks/s
2M (2 s interval)	7.161 s	0.18 tasks/s
5M (2 s interval)	15.314 s	0.15 tasks/s

Throughput decreases as workload grows — expected due to increasing CPU utilization and growing aggregation overhead.

5.2.3 Fairness Analysis

All configurations produced Jain’s Fairness Index = 0.889, indicating good but not perfect load distribution. Slight imbalance is expected from device heterogeneity, but MCDM scheduling mitigates extreme disparities.

5.2.4 Average Task Time

Trials	Avg. Task Time
1M (no checkpoint)	0.305 s
1M (5 s interval)	1.479 s
1M (2 s interval)	1.428 s
2M (2 s interval)	2.945 s
5M (2 s interval)	7.312 s

Average task time scales linearly with workload — confirming proportional computational complexity growth. Checkpointing itself does not significantly affect per-task duration.

5.2.5 Checkpoint Frequency Impact

5M Trials:

Interval	Execution Time
5 s	15.267 s
2 s	15.431 s
1 s	15.204 s
0.5 s	15.610 s
0.1 s	16.770 s
0.05 s	17.880 s

2M Trials:

Interval	Execution Time
5 s	7.419 s
2 s	7.161 s
1 s	7.415 s
0.5 s	7.524 s
0.1 s	7.685 s
0.05 s	8.520 s

!!! tip “Recommended Interval” The 5-second interval delivers the best performance-to-recovery-granularity balance. Intervals below 0.5 s introduce noticeable overhead without meaningful recovery benefit for most workloads.

5.2.6 Scalability with Increasing Trials

Trials	Execution Time
1M	2.058 s
2M	2.543 s
5M	3.687 s
10M	6.128 s
50M	21.447 s
100M	37.883 s

Execution time grows proportionally to workload with no instability or exponential growth — confirming predictable computational scaling.

5.2.7 Overall Analysis

The CROWDio performance curve consistently lies between the best (Galaxy A34) and worst (Galaxy A32) single-device results. This reflects the trade-off between parallel execution gains and distributed coordination overhead (including Chaquopy cross-language runtime overhead).

Key finding: When execution times of individual devices are averaged and compared against CROWDio distributed results, the performance curves intersect as workload grows. Beyond a workload threshold, CROWDio outperforms the average single-device execution — demonstrating that distributed execution becomes increasingly advantageous under higher computational intensity.

5.3 Image Processing

Dataset

Source: Kaggle Weather Image Recognition Dataset
Images: 6,862 total
Strategy: 5 batches, one per worker

Configuration

Workers: 5 Android devices
Scheduling: FIFO
Modes compared: Single device (Termux) vs. CROWDio distributed

Results

Execution Mode	Total Processing Time
Galaxy A34 (single device, Termux)	2.61 minutes
Galaxy A32 (single device, Termux)	3.78 minutes
5 Android Workers (CROWDio)	161 minutes

!!! warning “Workload Granularity Matters” For this image processing workload, distributed execution increased total time from ~3.78 minutes to ~161 minutes. Small batch sizes (5 images per batch) led to high dispatch and serialization frequency, causing network overhead to dominate. This highlights that task granularity must be carefully tuned before applying a distributed framework to data-parallel workloads on resource-constrained devices.

5.4 Sentiment Analysis

Dataset: Help Desk Tickets (Mendeley Data, V2)
Workers: 5 Android devices
Scheduling: FIFO
Model: Each worker processes an independent batch of tickets

This configuration demonstrates CROWDio’s ability to manage independent AI workloads across multiple mobile devices in parallel. The evaluation is ongoing to assess performance across varying workload sizes and device combinations.

5.5 Overall Findings

Finding	Detail
Checkpointing overhead	~2–3 s overhead; manageable and justified for fault tolerance in mobile environments
Optimal checkpoint interval	5–10 s provides best performance/recovery balance
WRR vs FIFO at scale	WRR achieves 56.9% latency reduction at 100M trials
MCDM necessity	FIFO consistently underperforms on heterogeneous clusters; MCDM is essential
Scalability	Linear execution time growth confirmed — no instability
Fairness	Jain’s Index ≈ 0.889 across all configurations
Image processing	Small batch sizes on Android cause coordination overhead to dominate
Crossover threshold	CROWDio outperforms average single-device beyond a workload size threshold

5.6 Discussion

Resource-aware scheduling and checkpointing are critical for efficient performance in heterogeneous mobile environments. MCDM algorithms such as WRR significantly outperform queue-based approaches by leveraging high-performance devices while avoiding weaker nodes — especially under heavy workloads.

Checkpointing overhead, while small, is well justified: in real MCdC scenarios, device mobility, battery drain, and network instability make fault tolerance non-negotiable. The overhead of 5–10% execution time is a worthwhile investment for deterministic resumption.

The image processing experiment highlights that task granularity matters: small batch sizes on Android create serialization-dominated runtimes. Larger, coarser tasks benefit more from distribution.

CROWDio provides an intuitive SDK and developer tooling that enables developers to harness nearby Android mobile workers for high-computation tasks, delivering both cost-effective compute and a streamlined developer experience — bridging the gap between theoretical MCdC research and practical real-world deployment.

Evaluation REST API

Live metrics are exposed at runtime via the Foreman API:

Endpoint	Description
`GET /api/evaluation/metrics`	Aggregate metrics across all workers and jobs
`GET /api/evaluation/load-distribution`	Task count per worker
`GET /api/evaluation/throughput-history`	Time-series throughput (tasks/s)
`GET /api/evaluation/worker-performance`	Per-worker performance breakdown

curl http://localhost:8000/api/evaluation/metrics

Experiment Modules

Located under evaluation/experiments/:

Module	Purpose
`scalability.py`	Throughput and latency as worker count scales 1→N
`heterogeneity.py`	Mixed PC + Android pool, load-balance effectiveness
`energy_constraints.py`	Energy-aware scheduling with battery-limited Android workers
`failure_simulation.py`	Worker failure injection at configurable rates — checkpoint recovery validation

# Scalability experiment
python -m evaluation.experiments.scalability --workers 1 2 4 8 --tasks-per-run 500 --host localhost

# Failure simulation with checkpointing
python -m evaluation.experiments.failure_simulation --workers 6 --failure-rate 0.2 --tasks 400 --checkpoint-enabled --host localhost