Adding Bushel Example (#132)

* docs: complete CloudKit schema design and data source research for Bushel demo

Add comprehensive planning documentation for Task 5 (Bushel Version History Tool):

- cloudkit-schema-plan.md: Complete CloudKit schema for RestoreImage, XcodeVersion,
  and SwiftVersion record types with fields, indexes, relationships, and implementation phases
- data-sources-api-research.md: API research for xcodereleases.com (JSON API),
  swiftversion.net (HTML scraping), and MistKit integration patterns with code examples
- mobileasset-wiki.md: Reference documentation on Apple's MobileAsset framework and MESU server
- firmware-wiki.md: Reference documentation on Apple firmware manifests and version history

Update Task 5 with refined scope focusing on macOS restore images for virtualization,
including data sources (ipsw.me via IPSWDownloads, Mr. Macintosh, MESU XML) and
updated subtasks aligned with implementation phases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add public write API and Bushel demo for CloudKit record operations

Add comprehensive public API for creating, updating, and deleting CloudKit records, along with a complete demo application showing real-world usage patterns.

## MistKit Public API Enhancements

### New Public Types
- Add `RecordOperation` struct with factory methods for create/update/delete operations
- Provides clean abstraction over internal OpenAPI-generated types
- Supports all operation types: create, update, forceUpdate, replace, forceReplace, delete, forceDelete

### CloudKitService Write Operations
- Add `modifyRecords(_:)` for batch record operations
- Add convenience methods: `createRecord()`, `updateRecord()`, `deleteRecord()`
- Proper error handling with typed `CloudKitError`
- Batch processing support (handles CloudKit's 200-operation limit)

### RecordFieldConverter Enhancements
- Add `toComponentsFieldValue()` for bidirectional field conversion
- Support all CloudKit field types: string, int64, double, boolean, bytes, date, location, reference, asset, list
- Proper date/timestamp conversion (milliseconds since epoch)
- Nested list support with type preservation

### Sendable Conformance
- Add Sendable to `FieldValue` and nested types (Location, Reference, Asset)
- Add Sendable to `CloudKitService` for concurrency safety
- Enable safe usage in Swift 6 strict concurrency mode

### CustomFieldValue Improvements
- Add memberwise initializer for programmatic construction
- Enable write operations without Decodable dependency

## Bushel Demo Application

Complete CLI tool demonstrating MistKit capabilities for syncing macOS restore images, Xcode versions, and Swift compiler versions to CloudKit.

### Features
- Multi-source data fetching (ipsw.me, xcodereleases.com, swift.org, MESU, Mr. Macintosh)
- CloudKit batch upload with progress reporting
- Record relationships using CloudKit References
- JSON export for offline analysis
- Proper error handling and retry logic

### Architecture
- Clean separation: DataSources → Models → CloudKit → Commands
- Uses ONLY public MistKit APIs (demonstrates proper abstraction)
- Swift 6 strict concurrency throughout
- Sendable-conforming types for thread safety

### Commands
- `sync`: Fetch and upload all data to CloudKit
- `export`: Fetch data and export to JSON (no CloudKit upload)

### Documentation
- Comprehensive README.md with architecture diagrams
- XCODE_SCHEME_SETUP.md for development configuration
- CloudKit schema definitions
- Usage examples and troubleshooting guide

### Dependencies
- IPSWDownloads for ipsw.me API integration
- SwiftSoup for HTML parsing
- ArgumentParser for CLI interface

## Files Changed

### MistKit Core (8 files)
- RecordOperation.swift (new, 122 lines)
- RecordOperation+OpenAPI.swift (new, 63 lines)
- CloudKitService+WriteOperations.swift (new, 151 lines)
- RecordFieldConverter.swift (+131 lines)
- CloudKitResponseProcessor.swift (+42 lines)
- CloudKitService.swift (Sendable conformance)
- FieldValue.swift (Sendable conformance)
- CustomFieldValue.swift (memberwise init)

### Bushel Demo (20 files, 2,437 lines)
- Package.swift and dependencies
- 5 data source fetchers
- 3 model types with CloudKit field mapping
- CloudKit integration layer (service, builder, engine)
- 2 CLI commands (sync, export)
- Comprehensive documentation (README + Xcode setup guide)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update iOS version in MistKit workflow

* feat: add automated CloudKit schema setup with cktool for Bushel demo

Provides declarative schema definition (.ckdb), automated import script, and comprehensive documentation for setting up CloudKit container schema programmatically, enabling easy deployment to development/production environments.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add TheAppleWiki data source and refactor to Server-to-Server authentication

Major Changes:

1. TheAppleWiki Integration
   - Add TheAppleWiki.com as new data source for historical IPSW data
   - Create IPSWParser, TheAppleWikiFetcher, and domain models
   - Integrate into DataSourcePipeline with parallel fetching
   - Add device filtering for VirtualMac variants
   - Implement file size parsing and prerelease detection

2. Server-to-Server Authentication Refactoring
   - Replace APITokenManager with ServerToServerAuthManager
   - Update CLI to accept --key-id and --key-file flags
   - Change env vars from CLOUDKIT_API_TOKEN to CLOUDKIT_KEY_ID + CLOUDKIT_KEY_FILE
   - Add PEM file validation and helpful error messages
   - Update both SyncCommand and ExportCommand

3. CloudKit Schema Fixes
   - Add required DEFINE SCHEMA header to schema.ckdb
   - Remove system fields (CloudKit adds these automatically)
   - Fix setup-cloudkit-schema.sh validation commands
   - Update to use 'xcrun cktool get-teams' for token check
   - Add --file flag to validate-schema and import-schema commands

4. Documentation & Security
   - Add comprehensive IMPLEMENTATION_NOTES.md for future reference
   - Create .gitignore with *.pem and .env protection
   - Update README with Server-to-Server authentication setup
   - Add security best practices for private key management

Files Changed:
- New: TheAppleWiki/ data source directory (IPSWParser, models, fetcher)
- New: IMPLEMENTATION_NOTES.md (session documentation)
- New: .gitignore (private key protection)
- Modified: Authentication throughout (BushelCloudKitService, SyncEngine, commands)
- Modified: Schema and setup script (format fixes, command corrections)
- Modified: README (comprehensive S2S auth documentation)

This refactoring demonstrates production-ready CloudKit integration using
recommended authentication methods and adds historical firmware data coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve data source parsing issues in Bushel sync

Fixed multiple data source fetchers to handle API format changes and
incomplete data, enabling successful sync from all external sources.

Changes:
- MESUFetcher: Switch from XML to PropertyList parsing for Apple MESU
  API format change (plist with nested dictionaries)
- TheAppleWikiFetcher: Remove VirtualMac device filter since UniversalMac
  restore images work for all Apple Silicon devices
- XcodeReleasesFetcher: Make optional all fields that can be missing in
  older Xcode releases (compilers, links, sdks, and nested fields)
- MrMacintoshFetcher: Fix HTML parsing for 3-column table structure and
  improve date parsing to handle formats without year
- DataSourcePipeline: Add detailed per-source logging for easier debugging
- schema.ckdb: Add ___recordName QUERYABLE to all record types for
  CloudKit query support

All fetchers now handle missing/optional data gracefully and provide
clear error messages when failures occur.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [skip ci] Adding previous conversation

* docs: add comprehensive CloudKit Server-to-Server authentication guide

Add detailed documentation for setting up CloudKit S2S authentication,
including schema configuration, permissions, and common troubleshooting.

Schema Changes:
- Update all record types (RestoreImage, XcodeVersion, SwiftVersion)
- Grant READ, CREATE, WRITE to both _creator and _icloud roles
- This enables Server-to-Server authentication to create/modify records

Key Findings:
- S2S authentication requires BOTH _creator AND _icloud permissions
- Schema syntax: GRANT READ, CREATE, WRITE TO "role"
- Boolean fields must use INT64 type (0/1)
- Operation type .forceReplace provides idempotency

Documentation includes:
- Complete setup walkthrough from key generation to deployment
- CloudKit permissions model explained for novice developers
- Common errors and solutions (ACCESS_DENIED, schema syntax, etc.)
- Swift implementation examples using MistKit
- Testing and verification procedures
- Production readiness checklist

This resolves the ACCESS_DENIED errors that were blocking CloudKit sync
operations and provides a reusable reference for future projects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(bushel): add tutorial-quality verbose logging and documentation polish

Enhances the Bushel demo for use in the upcoming blog post and as a reference
implementation for novice developers and Claude Code (Celestra demo).

## New Features

### Professional Logging System
- Add Logger.swift with os.Logger subsystems (cloudkit, datasource, sync)
- Implement --verbose/-v flag for both sync and export commands
- Add educational logging with explain() for teaching CloudKit concepts
- Include verbose(), info(), success(), warning(), error() helpers
- Thread-safe global isVerbose flag with proper Swift 6 concurrency

### Educational Logging Throughout
- SyncEngine: Log sync phases, data pipeline, MistKit batch operations
- BushelCloudKitService: Log dependency ordering, batch details, CloudKit responses
- Verbose mode reveals Server-to-Server auth, batching limits, API calls
- Tutorial comments explaining MistKit patterns and CloudKit concepts

### Tutorial-Quality Documentation
- Add "What You'll Learn" section with 5 key objectives
- Create Quick Start guide with Learning Mode examples
- Document verbose flag usage and benefits
- Add comprehensive troubleshooting section for common beginner mistakes
- Expand Usage section with all command-line flag examples
- Add "Using Bushel as a Reference" guide for copying patterns
- Include learning path for beginners with key files to study
- Link to Celestra sibling demo (coming soon)

## Documentation Improvements
- README expanded from ~430 to ~590 lines
- Added troubleshooting guide with 5 common issues and solutions
- Enhanced learning resources with beginner-focused content
- Documented all reusable patterns (auth, batching, logging, records)

## GitHub Issues Created
Created 5 issues for deferred enhancements:
- #136: Comprehensive test suite
- #137: Incremental sync with change tracking
- #138: Configuration file support (.bushelrc)
- #139: Advanced CloudKit features (deduplication, zones, conflicts)
- #140: Architecture diagrams for documentation

## Task Management
- Completed Task 5 (Bushel Version History Tool) in Task Master
- All subtasks 5.1-5.5 marked complete
- Ready to proceed to Task 6 (Celestra RSS Tool)

## Build Status
- ✅ Zero errors, zero warnings
- ✅ Swift 6 strict concurrency compliant
- ✅ All new code properly annotated

This makes Bushel ready for:
1. Blog Post Part 3 feature with professional screenshots
2. Tutorial resource for novice iOS developers learning CloudKit
3. Reference template for Claude Code implementing Celestra

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: convert boolean to INT64 helper method for CloudKit compatibility

CloudKit Web Services doesn't have a native BOOLEAN type - it uses INT64
with 0/1 values. This change:

- Converts FieldValue.boolean enum case to a static helper method
- Makes isSigned optional in RestoreImageRecord (unknown for some sources)
- Updates RecordBuilder to conditionally include isSigned field
- Improves merge logic to prefer non-nil isSigned values
- Removes dead code from CustomFieldValue
- Cleans up unused imports and OpenAPI type definitions

All tests passing ✅

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(bushel): improve isSigned merge logic and add data source timestamps

Core Improvements:
- Fixed MESU fetcher XML parsing to correctly navigate plist structure
- Added sourceUpdatedAt field to RestoreImageRecord for tracking data freshness
- Implemented sophisticated merge priority: MESU > most recent timestamp > prefer false on conflict
- MrMacintoshFetcher now extracts page update date from HTML
- Fixed Package.swift dependency reference (MistKit-Bushel)

New Features:
- Added ClearCommand for deleting all CloudKit records
- Added deleteAllRecords() and clear() methods to CloudKit services

Data Source Improvements:
- MESU now correctly identifies signed builds (e.g., 25B78 = macOS 26.1)
- Merge logic respects MESU as authoritative regardless of age
- Non-MESU sources compared by sourceUpdatedAt for freshness
- When sources disagree with same timestamp, prefers false (safer)

Documentation:
- Added TODO-AppleDB-LastModified.md for future AppleDB integration
- Documents planned Last-Modified header usage across all fetchers
- Includes complete implementation guide for AppleDB replacement

Technical Details:
- DataSourcePipeline.swift: Lines 176-219 implement merge priority rules
- MESUFetcher.swift: Fixed XML path navigation for VirtualMac2,1
- MrMacintoshFetcher.swift: Parses "UPDATED: MM/DD/YY" from HTML

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(bushel): add HTTP Last-Modified tracking and AppleDB integration

- Add HTTPHeaderHelpers utility for fetching Last-Modified headers from data sources
- Update IPSWFetcher, MESUFetcher, and MrMacintoshFetcher to capture HTTP Last-Modified timestamps
- Add AppleDB as new data source with GitHub commit date tracking for macOS-specific updates
  - AppleDBFetcher fetches from api.appledb.dev with device-specific signing status
  - Uses GitHub API to track when macOS data was last updated (more accurate than Last-Modified)
  - Falls back to HTTP Last-Modified if GitHub API fails
  - Handles SDK entries without build numbers gracefully
- Deprecate TheAppleWikiFetcher in favor of AppleDB (more reliable and up-to-date)
- Update DataSourcePipeline to include AppleDB in fetch pipeline

Test results show significant improvements:
- AppleDB contributes 236 images (5x more than ipsw.me)
- All timestamp tracking working correctly across sources
- Merge logic properly prioritizes by sourceUpdatedAt timestamps
- Total unique images: 238 (up from 173)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(bushel): add data source metadata tracking with fetch throttling

Implement comprehensive metadata tracking system to monitor when data
sources were last fetched and updated, enabling intelligent throttling
to prevent unnecessary API calls.

## New Features

### DataSourceMetadata Model
- Track source name, record type, fetch times, and update times
- Store record counts, fetch durations, and error messages
- Use consistent recordName format: "metadata-{source}-{recordType}"

### CloudKit Schema
- Add DataSourceMetadata record type with queryable fields
- Fields: sourceName, recordTypeName, lastFetchedAt, sourceUpdatedAt,
  recordCount, fetchDurationSeconds, lastError
- Note: recordName system field must be marked queryable via Dashboard

### FetchConfiguration System
- Per-source throttling intervals (AppleDB: 6h, MESU: 1h, others: 12h)
- Environment variable override support
- Force fetch bypass option

### DataSourcePipeline Integration
- Wrap all fetcher calls with metadata tracking
- Query existing metadata before fetch
- Check throttling rules before fetching
- Time fetch operations
- Update metadata on success or failure
- Sync metadata to CloudKit

### CLI Enhancements
- Add --force flag to bypass throttling
- Add --min-interval flag to override default intervals
- Add --source flag to fetch from specific source only
- Add status command to view metadata and next fetch times

### StatusCommand
- Display last fetched time with "X ago" formatting
- Show source last updated time
- Report record counts and fetch durations
- Calculate next eligible fetch time
- Support --detailed and --errors-only flags

### FieldValue Extensions
- Convenience accessors for FieldValue enum (stringValue, int64Value, etc.)
- Enable cleaner CloudKit response parsing

## Known Issues

### MistKit Logging Redaction
The KEY_ID_REDACTED replacement in MistKit logging is too aggressive,
redacting critical error information including field names and types
from CloudKit error responses. This makes debugging CloudKit errors
nearly impossible.

See METADATA_TRACKING_STATUS.md for complete implementation details
and debugging steps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve CloudKit metadata tracking issues and add debug logging

This commit fixes several critical issues preventing DataSourceMetadata records from being created in CloudKit and adds comprehensive debugging capabilities.

## Core Fixes

### CloudKit Field Type Issues
- Add explicit type annotations for INT64 and DOUBLE fields in RecordFieldConverter
- Fix date/timestamp conversion to use integer milliseconds instead of floating-point
- CloudKit DATE/TIME fields require integer milliseconds, not double precision

### Debug Logging Enhancements
- Add MISTKIT_DISABLE_LOG_REDACTION environment variable to disable log redaction
- Add --no-redaction flag to both sync and status commands for easier debugging
- Refine maskKeyIdPattern regex from [A-Za-z0-9]{8,} to [a-fA-F0-9]{40,}
- Previous pattern was too aggressive and redacted legitimate field names

### Package Configuration
- Fix MistKit package reference in Bushel/Package.swift (MistKit-Bushel → MistKit)

## Technical Details

The key issue was that CloudKit TIMESTAMP fields require integer milliseconds, but we were sending double-precision floating-point values. This caused CloudKit to reject records with "Invalid value, expected type TIMESTAMP" errors.

Additionally, INT64 and DOUBLE field types were missing explicit type annotations (type: nil), which CloudKit requires for proper field validation.

The overly broad log redaction pattern was masking actual CloudKit error messages, making debugging impossible. The refined pattern now only redacts actual CloudKit key IDs (40+ hex characters).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: remove obsolete metadata tracking documentation

Remove conversation transcript and status document as all issues have been resolved and the metadata tracking feature is now fully functional.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(bushel): add PR #132 code review fix documentation

Add comprehensive documentation for addressing code review feedback:
- Task Master PRD with 8 prioritized tasks (P0-P2)
- Detailed implementation guide with code examples and acceptance criteria
- Remove obsolete TODO-AppleDB-LastModified.md

Coverage includes:
- Type conversion fixes (Boolean/Int64, safe casting)
- Enhanced error handling and retry logic
- Security improvements (key permissions, logging redaction)
- Code refactoring and complexity reduction
- Validation layers for references and fields
- Batch operation enhancements
- Documentation and performance optimizations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Code Review Fixes (#142)

* refactor: replace RecordFieldConverter with initializer-based conversions

Replace the 363-line RecordFieldConverter utility enum with idiomatic
Swift initializer extensions, resolving all major linting violations.

Changes:
- Add Components.Schemas.FieldValue+FieldValue.swift for MistKit → OpenAPI conversion
- Add FieldValue+Components.swift for OpenAPI → MistKit conversion
- Add CustomFieldValue.CustomFieldValuePayload+FieldValue.swift for list nesting
- Remove RecordFieldConverter.swift (363 lines with 8 serious violations)
- Refactor CloudKitResponseProcessor.processModifyRecordsResponse using static error handler array
- Add CloudKitError.init?(_ response:) with @sendable handlers (complexity reduced from 25 → 3)
- Fix multiline arguments brackets in CloudKitService+WriteOperations.swift
- Remove TODO comment in FieldValue.swift
- Fix public import warning in CustomFieldValuePayload.swift

Benefits:
- More idiomatic Swift API (initializer-based conversions)
- Cleaner call sites: FieldValue(x) vs RecordFieldConverter.convertToFieldValue(x)
- Better maintainability with loop-based error handling
- Zero compilation errors, only minor style warnings remain

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: unify error handling with CloudKitResponseType protocol

Replace operation-specific error handlers with a generic protocol-based approach that works for all CloudKit operations. This reduces code duplication and makes error handling more maintainable.

- Add CloudKitResponseType protocol defining common error response accessors
- Implement protocol conformance for getCurrentUser and modifyRecords operations
- Refactor CloudKitError initializers to use generic protocol-based error extraction
- Remove CloudKitError+ModifyRecords.swift (replaced by generic approach)
- Simplify CloudKitResponseProcessor.processGetCurrentUserResponse
- Add proper copyright headers to operation extension files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: leogdion

.taskmaster/docs/cloudkit-schema-plan.md

@@ -0,0 +1,133 @@
+# Firmware
+
+Source: TheAppleWiki - https://theapplewiki.com/wiki/Firmware
+(Saved from PDF: Firmware - The Apple Wiki.pdf)
+
+A **firmware** is an IPSW file or OTA update ZIP file that contains everything needed to install the core operating system of a device.
+
+## Firmware Manifests
+
+To check for iOS, iPod, Apple TV, and HomePod mini updates, iTunes, Finder, Apple Configurator, and Apple Devices contact the main manifest at:
+- **https://s.mzstatic.com/version**
+
+This manifest also contains carrier bundles.
+
+There are separate manifests for:
+- **macOS**: https://mesu.apple.com/assets/macos/com_apple_macOSIPSW/com_apple_macOSIPSW.xml
+- **bridgeOS**: https://mesu.apple.com/assets/bridgeos/com_apple_bridgeOSIPSW/com_apple_bridgeOSIPSW.xml
+- **visionOS**: https://mesu.apple.com/assets/visionos/com_apple_visionOSIPSW/com_apple_visionOSIPSW.xml
+
+The first-generation Apple TV also uses a firmware manifest, found at:
+- **https://mesu.apple.com/version.xml**
+
+It uses separate files, rather than consolidating them into an IPSW archive.
+
+## History
+
+The original use of the version manifest was to notify the user of iPod firmware updates. The notification directs the user to the Apple Support website to download iPod Updater. With iTunes 7.0, iPod Updater functionality was merged into iTunes, using IPSW files to hold the firmware. iTunes 7.3 extends the manifest and IPSW file format to support the iPhone.
+
+## Limitations
+
+There is no public manifest of IPSWs for any Apple Watch, HomePod (other than HomePod mini), or Apple TV 4K and later. These devices exclusively use OTA updates, and can be recovered using recoveryOS. Some IPSW files have been discovered for these devices through various means, but lack of a user-accessible Lightning or USB-C port renders it difficult to make use of them.
+
+## Firmware List (by device)
+
+### Release Firmware
+
+**Apple TV**
+- 1.x · 2.x · 3.x · 4.x · 5.x · 6.x · 7.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**Apple Vision**
+- 1.x · 2.x · 26.x
+
+**Apple Watch**
+- 1.x · 2.x · 3.x · 4.x · 5.x · 6.x · 7.x · 8.x · 9.x · 10.x · 11.x · 26.x
+
+**HomePod**
+- 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**Mac**
+- 1.x · 2.x · 3.x · 4.x · 6.x · 7.x · 8.x · 9.x · 10.0.x Cheetah · 10.1.x Puma · 10.2.x Jaguar · 10.3.x Panther · 10.4.x Tiger · 10.5.x Leopard · 10.6.x Snow Leopard · 10.7.x Lion · 10.8.x Mountain Lion · 10.9.x Mavericks · 10.10.x Yosemite · 10.11.x El Capitan · 10.12.x Sierra · 10.13.x High Sierra · 10.14.x Mojave · 10.15.x Catalina · 11.x Big Sur · 12.x Monterey · 13.x Ventura · 14.x Sonoma · 15.x Sequoia · 26.x Tahoe
+
+**Mac Server**
+- 1.x · 10.0.x Cheetah · 10.1.x Puma · 10.2.x Jaguar · 10.3.x Panther · 10.4.x Tiger · 10.5.x Leopard · 10.6.x Snow Leopard · 10.7.x Lion · macOS Server
+
+**iBridge (T2 chip)**
+- 1.x (embeddedOS) · 2.x · 3.x · 4.x · 5.x · 6.x · 7.x · 8.x · 9.x · 10.x
+
+**iPad**
+- 3.x · 4.x · 5.x · 6.x · 7.x · 8.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**iPad Air**
+- 7.x · 8.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**iPad Pro**
+- 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**iPad mini**
+- 6.x · 7.x · 8.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**iPhone**
+- 1.x · 2.x · 3.x · 4.x · 5.x · 6.x · 7.x · 8.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x · 16.x · 17.x · 18.x · 26.x
+
+**iPod touch**
+- 1.x · 2.x · 3.x · 4.x · 5.x · 6.x · 7.x · 8.x · 9.x · 10.x · 11.x · 12.x · 13.x · 14.x · 15.x
+
+### Other Firmware Types
+
+- Preinstalled Firmware
+- iPod Firmware
+- iPod Recovery Firmware
+- Mac Security Updates
+- Mac Server Security Updates
+
+### Beta Firmware
+
+### OTA Updates
+
+### Rapid Security Responses
+
+### Beta Rapid Security Responses
+
+### RecoveryOSUpdates
+
+### Firmware Keys
+
+## External Links
+
+### Manifests
+
+**iOS, iPod, Apple TV, and HomePod mini**:
+- https://s.mzstatic.com/version
+- Old URL: https://itunes.apple.com/WebObjects/MZStore.woa/wa/com.apple.jingle.appserver.client.MZITunesClientCheck/version
+
+**bridgeOS**:
+- https://mesu.apple.com/assets/bridgeos/com_apple_bridgeOSIPSW/com_apple_bridgeOSIPSW.xml
+
+**macOS**:
+- https://mesu.apple.com/assets/macos/com_apple_macOSIPSW/com_apple_macOSIPSW.xml
+
+**visionOS**:
+- https://mesu.apple.com/assets/visionos/com_apple_visionOSIPSW/com_apple_visionOSIPSW.xml
+
+## Key Insights for CloudKit Schema
+
+1. **Multiple firmware types**: IPSW files (full OS), OTA updates (incremental)
+2. **macOS versioning**: Uses both legacy (10.x) and modern (11+) numbering
+3. **Latest version number**: 26.x appears across all platforms (likely future release)
+4. **Apple Silicon coverage**: Big Sur 11.x onwards for Apple Silicon Macs
+5. **Manifest structure**:
+   - iOS/iPod/Apple TV/HomePod mini: Single manifest at s.mzstatic.com
+   - macOS/bridgeOS/visionOS: Separate MESU XML files
+6. **Beta firmware**: Listed separately from release firmware
+7. **OTA vs IPSW**: Different update mechanisms, both tracked in manifests
+
+## Relevance to MistKit Project
+
+For Bushel's macOS virtualization use case:
+- Focus on **macOS manifest only**: `mesu.apple.com/assets/macos/com_apple_macOSIPSW/`
+- Version coverage: Big Sur 11.x through current (26.x)
+- IPSW files specifically for restore/install, not OTA updates
+- MESU provides only latest signed version
+- Need community sources (ipsw.me, Mr. Macintosh) for historical data
+- Beta firmware requires separate tracking (not in main MESU manifest)

.taskmaster/docs/data-sources-api-research.md

@@ -0,0 +1,182 @@
+# MobileAsset Framework
+
+Source: TheAppleWiki - https://theapplewiki.com/wiki/MobileAsset
+
+> This page is a work in progress
+
+Apple's operating systems have a framework called **MobileAsset**
+used to download and update system data, independently of OS updates.
+
+For example, the timezone database and the keyboard autocorrect language models
+are updated silently in the background and don't need an OS update.
+High-quality speech synthesizer voices for Siri and accessibility features
+are downloaded on demand when a language is selected,
+rather than being included with the OS,
+since they are large and most people only need one or two languages.
+OTA updates to iOS itself and firmware for Apple accessories are also mobile assets.
+
+## Asset types
+
+Assets are grouped using an *asset type* string in reverse-domain format,
+starting with `com.apple.MobileAsset`,
+such as `com.apple.MobileAsset.TimeZoneUpdate`.
+There are currently more than a hundred asset types known.
+
+There may be multiple *assets* for each asset type.
+For example, the asset type for AirTag firmware
+has a single asset with the latest version of the firmware,
+but the asset type for Siri voices has hundreds of assets,
+for the different voice languages, qualities, and compatible OS versions.
+
+## Servers
+
+There are two server systems from which asset metadata is retrieved.
+"Mesu" uses static plist files in a simple CDN,
+and the newer "Pallas" uses POST requests to a dynamic server.
+Some assets are only in mesu, some are only in pallas,
+some are actively updated in both,
+and some are present in both but mesu doesn't get updated anymore
+(still there only for compatibility with older operating systems).
+Apple seems to be moving to Pallas for new asset types.
+
+Both servers only provide the list of available assets
+and their metadata, not the content.
+The metadata includes a URL to the actual asset file,
+which is hosted on `updates.cdn-apple.com`
+(or `appldnld.apple.com` for some very old stuff).
+
+### Mesu
+
+The original asset server is `mesu.apple.com`.
+It's a file server (probably Amazon S3 with a CDN) which serves XML plists,
+containing the metadata for all the assets with a given asset type.
+For example, every Siri voice for every language is listed in the same file,
+and every iOS OTA update for every device is in the same file.
+
+For iOS, these XML plists are downloaded from `mesu.apple.com/assets/{type}/{type}.xml`,
+where `{type}` is the asset type, with dots replaced with underscores.
+Other operating systems use `mesu.apple.com/assets/{os}/{type}/{type}.xml`,
+where `{os}` is one of: `audio`, `watch`, `tv`, `macos`, `visionos`.
+Some old visionOS assets are under "xros" instead.
+
+Example:
+`https://mesu.apple.com/assets/macos/com_apple_MobileAsset_TimeZoneUpdate/com_apple_MobileAsset_TimeZoneUpdate.xml`
+
+They are simply static files;
+no special user agent, request headers or query parameters are required,
+and cache headers like ETag, Last-Modified and If-Modified-Since work as expected.
+
+In the past, when iOS OTA updates used mesu,
+there were also other directories like "iOS12DeveloperBeta"
+in place of the {os} to serve OTAs of iOS betas.
+This is not used anymore on newer iOS versions,
+since OTA software updates (both beta and release) use Pallas exclusively.
+The only known remaining exception is beta firmware for AirPods,
+which are in `AirPods2022Seed/`.
+
+The plist files in mesu have a dictionary containing these keys:
+* `AssetType` (string): The dot-separated asset type string. Not always present.
+* `Signature` (data): RSA signature of the asset metadata.
+* `Certificate` (data): X.509 certificate used to verify the signature.
+* `SigningKey` (string): Always "`AssetManifestSigning`".
+* `FormatVersion` (integer): Always "1" (but not always present).
+* `Assets` (array of dict): The list of assets. See below for details on the content of each dict.
+
+There are three additional plists
+in the mesu server that are not MobileAssets,
+as their content is completely different:
+* `macos/com_apple_macOSIPSW/com_apple_macOSIPSW.xml`
+* `bridgeos/com_apple_bridgeOSIPSW/com_apple_bridgeOSIPSW.xml`
+* `visionos/com_apple_visionOSIPSW/com_apple_visionOSIPSW.xml`
+
+They contain URLs for .ipsw files for the latest version of
+macOS (Apple Silicon), bridgeOS (T2 firmware) and visionOS.
+
+### Pallas
+
+The newer system is codenamed Pallas.
+It runs in `gdmf.apple.com`
+(the meaning of "gdmf" is unknown).
+To get assets, you do a POST request sending JSON,
+and get back a signed "JSON Web Token" (JWT) as response.
+
+An example minimal request:
+```json
+{
+    "ClientVersion": 2,
+    "AssetAudience": "0c88076f-c292-4dad-95e7-304db9d29d34",
+    "AssetType": "com.apple.MobileAsset.TimeZoneUpdate"
+}
+```
+
+Unlike mesu, with pallas the OS can make more specific requests,
+such as sending the OS version number
+and only getting assets compatible with that version.
+Or in the case of OS OTA updates,
+getting updates only for the specific hardware device,
+and deltas with the current version as prerequisite.
+
+## Asset metadata
+
+The XML plists from mesu and the JSON responses from pallas both have
+an `"Assets"` key with an array of dictionaries.
+Each dictionary in the array describes a downloadable asset file.
+The content of these dictionaries can vary depending on asset type,
+but it's the same schema for Mesu and Pallas.
+
+Some asset types (currently) have no assets.
+This is represented with a dummy entry in the array:
+```json
+"Assets": [
+  {
+    "__Empty": "Empty"
+  }
+]
+```
+
+Strangely, `com.apple.MobileAsset.DeviceCheck` in mesu
+is missing the `Assets` key altogether.
+
+Some dictionary keys are common to every asset
+(even though they may not be always present),
+and others are specific to certain asset types.
+The common keys are:
+* `__BaseURL` (string): The prefix of the URL that the asset file can be downloaded from. Always present.
+* `__RelativePath` (string): The path to download the asset file from, relative to the BaseURL. Always present.
+* `_DownloadSize` (integer): Size of the file to download, in bytes. Always present.
+* `_UnarchivedSize` (integer): How much space this asset takes once the archive is extracted. Always present.
+* `_Measurement` (data): SHA-1 hash of the asset file to download. Always present. This is a <data> element in mesu plists, and a base64 string in pallas JSON responses.
+* `_MeasurementAlgorithm` (string): Hashing algorithm used in `_Measurement`. Always present, always set to "SHA-1". It's unknown if the code even supports other algorithms.
+* `_CompressionAlgorithm` (string): Outer format of the asset file. Always "zip" in Mesu assets, can also be "aea", "AppleArchive", or "AppleEncryptedArchive" in some Pallas assets. Always present.
+* `_IsZipStreamable` (bool): Exact meaning unknown; either `true` or missing. Seems to be `true` on every `zip`- and `aea`-format asset, except old ones hosted in the `appldnld` server.
+* `_MasteredVersion` (string): Some version number. Present in every mesu asset, except some with asset type `SoftwareUpdate`, and some newer "auto assets" (`_IsMAAutoAsset==true`).
+* `_CompatibilityVersion` (integer): A version number increased when the content of the asset changes in an incompatible way; newer versions of the OS start using assets with a newer content version while older OS versions keep using the old assets.
+* `__CanUseLocalCacheServer` (bool): Presumably, whether this asset is eligible for Content Caching. It seems to be either true or missing.
+
+The full URL to download an asset file is `__BaseURL + __RelativePath`.
+The reason why it's split into two is that `__RelativePath`
+can also be requested from a local caching server.
+
+Most asset types have extra keys specific to their own purposes,
+sometimes *a lot* of extra keys.
+
+## Key Insights for CloudKit Schema
+
+1. **MESU is NOT a historical database** - It only contains the current signed release
+2. **Three special IPSW plists** that are NOT MobileAssets:
+   - macOS: `mesu.apple.com/assets/macos/com_apple_macOSIPSW/com_apple_macOSIPSW.xml`
+   - bridgeOS: `mesu.apple.com/assets/bridgeos/com_apple_bridgeOSIPSW/com_apple_bridgeOSIPSW.xml`
+   - visionOS: `mesu.apple.com/assets/visionos/com_apple_visionOSIPSW/com_apple_visionOSIPSW.xml`
+3. **MESU structure**: Simple static XML files, no authentication required
+4. **Hash algorithm**: MESU only provides SHA-1, not SHA-256
+5. **Download URLs**: All hosted on `updates.cdn-apple.com`
+6. **Beta releases**: No longer in MESU for iOS (moved to Pallas), unknown for macOS
+
+## Relevance to MistKit Project
+
+For the Bushel CloudKit schema:
+- Use MESU as a **freshness detector** for new releases only
+- Don't rely on MESU for historical data (it doesn't have any)
+- Primary source should be ipsw.me API (via IPSWDownloads package)
+- MESU only provides SHA-1, need ipsw.me for SHA-256
+- MESU won't have beta releases - need Mr. Macintosh for those