AI food recognition and calorie counting by photo in mobile app

AI Food Recognition and Calorie Counting from Photos in Mobile Applications

Food tracking via photos is one of the most technically complex "simple" tasks in mobile AI. User expects: photograph borscht—get macros. Between these points lies a chain: dish recognition, ingredient identification, portion estimation, nutritional database search. Each link adds error.

Technical Recognition Chain

Correct architecture isn't "one model for everything" but a pipeline of specialized steps.

Step 1: dish detection and classification. CoreML on iOS (EfficientDet or YOLOv8 classification model), TFLite on Android. For MVP—cloud API: Clarifai Food Model, Google Cloud Vision with food tags, or specialized Logmeal API.

Step 2: portion estimation. Harder. Without reference object (coin, hand, standard plate) estimating grams is nearly impossible. Two practical solutions: ask user for dish type (20cm plate, 200ml glass) or use ARKit/ARCore for depth estimation. ARKit depth gives acceptable results for volume dishes—15–25% error, better than user manual entry (they usually underestimate).

Step 3: nutritional data lookup. USDA FoodData Central—free API with 700,000+ products. Open Food Facts—open-source base, good for packaged goods. For Russian market, need domestic dishes: borscht, pelmeni, olivier—not in USDA in familiar format.

// iOS: complete recognition pipeline
struct FoodRecognitionPipeline {

    func analyze(image: UIImage, portionContext: PortionContext?) async throws -> MealAnalysis {

        // 1. Food recognition via Logmeal API
        let foodItems = try await logmealClient.recognizeFood(image: image)

        // 2. Portion estimation
        let portionEstimates: [PortionEstimate]
        if let context = portionContext {
            portionEstimates = estimatePortionFromContext(foodItems, context: context)
        } else {
            portionEstimates = try await estimatePortionWithAR(image: image)
        }

        // 3. Nutritional data
        let nutritionData = try await withThrowingTaskGroup(of: NutritionResult.self) { group in
            for (item, portion) in zip(foodItems, portionEstimates) {
                group.addTask {
                    try await self.fetchNutrition(food: item, grams: portion.estimatedGrams)
                }
            }
            return try await group.reduce(into: []) { $0.append($1) }
        }

        return MealAnalysis(
            items: foodItems,
            portions: portionEstimates,
            nutrition: nutritionData.aggregate(),
            confidence: foodItems.map(\.confidence).min() ?? 0
        )
    }
}

Parallel nutrition requests via TaskGroup matter: 3 items sequentially = 3x delay.

Composite Dishes Challenge

Borscht in photo—beet, cabbage, carrot, potato, meat, sour cream in unknown proportions. Two solution approaches:

Recipe database. LLM or custom model breaks dish into recipe ingredients. Works for standard dishes, poorly for home cooking variations.

User correction. After auto-recognition, user sees predicted composition and can remove or add ingredients. Swipe-to-remove, slider for grams. This UX is fundamentally better than "98% accuracy" without edit capability.

// Android: meal composition editor UI
@Composable
fun MealCompositionEditor(
    items: List<FoodItem>,
    onItemRemoved: (FoodItem) -> Unit,
    onPortionChanged: (FoodItem, Float) -> Unit
) {
    LazyColumn {
        items(items, key = { it.id }) { item ->
            SwipeToDismiss(
                state = rememberDismissState { if (it == DismissValue.DismissedToStart) {
                    onItemRemoved(item); true } else false
                },
                background = { DeleteBackground() },
                dismissContent = {
                    FoodItemRow(
                        item = item,
                        onPortionChange = { grams -> onPortionChanged(item, grams) }
                    )
                }
            )
        }
    }
}

HealthKit and Health Connect Integration

Logged meals should flow into health ecosystem.

// iOS: log meal to HealthKit
func logMealToHealthKit(_ meal: MealAnalysis) async throws {
    let store = HKHealthStore()
    let caloriesType = HKQuantityType(.dietaryEnergyConsumed)
    let proteinType = HKQuantityType(.dietaryProtein)
    let carbsType = HKQuantityType(.dietaryCarbohydrates)
    let fatType = HKQuantityType(.dietaryFatTotal)

    let metadata: [String: Any] = [
        HKMetadataKeyFoodType: meal.primaryItem?.name ?? "Mixed Meal"
    ]

    let samples = [
        HKQuantitySample(type: caloriesType,
                        quantity: .init(unit: .kilocalorie(), doubleValue: meal.nutrition.calories),
                        start: .now, end: .now, metadata: metadata),
        HKQuantitySample(type: proteinType,
                        quantity: .init(unit: .gram(), doubleValue: meal.nutrition.protein),
                        start: .now, end: .now)
        // + carbs, fat
    ]

    try await store.save(samples)
}

Request permissions early via HKHealthStore.requestAuthorization. Common mistake: request on first app open before user sees value. Apple doesn't forbid it, but conversion is higher if request appears at first meal entry.

Accuracy Barriers and Honest Display

Even good models error on non-standard dishes, poor lighting, unusual angles. Hide uncertainty—mistake. Show confidence score next to result—right:

struct NutritionDisplayView: View {
    let analysis: MealAnalysis

    var body: some View {
        VStack(alignment: .leading, spacing: 12) {
            if analysis.confidence < 0.6 {
                ConfidenceWarningBanner(
                    message: "Low recognition confidence. Check dish composition."
                )
            }
            CalorieSummaryCard(nutrition: analysis.nutrition)
            MacroBreakdownChart(nutrition: analysis.nutrition)
            IngredientList(items: analysis.items, editable: true)
        }
    }
}

Timeline Estimates

Basic integration (one recognition API + USDA nutrition base + simple UI)—1–2 weeks. Full implementation with AR portion estimation, composite dishes, user correction, HealthKit/Health Connect, eating history, and daily macro targets—1–2 months.