Skip to content

cli.md

Latest commit

 

History

History
500 lines (368 loc) · 13 KB

cli.md

File metadata and controls

500 lines (368 loc) · 13 KB

Osiris CLI Reference

Command-line interface for Osiris v0.2.0 pipeline system.

Global Options

osiris [--help] [--version] <command> [options]
  • --help: Show help message
  • --version: Display Osiris version (v0.2.0)

Core Commands

init - Initialize Configuration

Initialize Osiris configuration files:

osiris init [OPTIONS]

Options:

  • --no-comments: Generate configuration without comment lines
  • --stdout: Output configuration to stdout instead of creating file
  • --json: Output initialization results in JSON format

Creates/Updates:

  • osiris.yaml - Main configuration file including:
    • Logging and session settings
    • LLM configuration
    • AIOP configuration with defaults for export settings
    • Pipeline and discovery settings
  • Creates backup as osiris.yaml.backup if file exists

AIOP Configuration: The init command generates a complete AIOP configuration section with defaults:

aiop:
  enabled: true            # Auto-generate AIOP after each run
  policy: core             # core|annex|custom
  max_core_bytes: 300000   # Size limit for Core package
  timeline_density: medium # Event filtering level
  metrics_topk: 100        # Top metrics to keep
  # ... additional settings

Examples:

# Standard initialization
osiris init

# Generate config without comments for production
osiris init --no-comments

# Preview config without creating file
osiris init --stdout

# Get JSON output for automation
osiris init --json

chat - Conversational Pipeline Creation

Start interactive conversation to create pipelines:

osiris chat [--pro-mode]

Options:

  • --pro-mode: Use custom LLM prompts from .osiris_prompts/

compile - Compile OML to Manifest

Compile an OML pipeline into an executable manifest:

osiris compile <pipeline.oml> [--output DIR] [--verbose]

Example:

osiris compile pipeline.oml
# Output: logs/run_<timestamp>/compile_<timestamp>/manifest.yaml

run - Execute Pipeline

Execute a compiled manifest or OML file:

# Run the last compiled manifest
osiris run --last-compile

# Run specific manifest
osiris run manifest.yaml

# Run with E2B cloud execution
osiris run --last-compile --e2b

# E2B with custom resources
osiris run --last-compile --e2b --e2b-cpu 4 --e2b-mem 8

# E2B with auto-install dependencies
osiris run --last-compile --e2b --e2b-install-deps

Options:

  • --last-compile: Use the most recently compiled manifest
  • --e2b: Execute in E2B cloud sandbox
  • --e2b-cpu N: Number of CPU cores (default: 2)
  • --e2b-mem GB: Memory in GB (default: 4)
  • --e2b-timeout SECONDS: Execution timeout (default: 900)
  • --e2b-install-deps: Auto-install missing Python packages
  • --e2b-env KEY=VALUE: Set environment variable in sandbox
  • --e2b-pass-env NAME: Pass environment variable from current shell
  • --dry-run: Show execution plan without running
  • --verbose: Show detailed execution output

validate - Validate Configuration

Check Osiris configuration for errors:

osiris validate [--mode MODE]

Options:

  • --mode: Validation mode (strict, warn, off), default: warn

Environment override:

OSIRIS_VALIDATION=strict osiris validate

Connection Commands

connections list - List Connections

Display configured connections with masked secrets:

osiris connections list [--json]

Example output:

Osiris Connections
╭─────────┬───────────┬──────────────────┬─────────╮
│ Family  │ Alias     │ Host             │ Default │
├─────────┼───────────┼──────────────────┼─────────┤
│ mysql   │ default   │ localhost:3306   │ ✓       │
│ mysql   │ primary   │ db.example.com   │         │
│ supabase│ main      │ *.supabase.co    │ ✓       │
╰─────────┴───────────┴──────────────────┴─────────╯

Options:

  • --json: Output in JSON format

connections doctor - Test Connections

Validate connection configurations and test connectivity:

osiris connections doctor

Checks:

  • Configuration syntax validity
  • Required fields present
  • Environment variables resolved
  • Network connectivity (if possible)

Log Commands

logs list - List Sessions

Display all pipeline execution sessions:

osiris logs list [--recent N] [--verbose]

Options:

  • --recent N: Show only N most recent sessions
  • --verbose: Include detailed session information

logs show - Display Session Details

Show events and metrics for a specific session:

osiris logs show --session <session_id> [--filter TYPE]

Options:

  • --session: Session ID (e.g., run_1234567890)
  • --filter: Filter by event type (e.g., "error", "step_complete")

logs html - Generate HTML Report

Create interactive HTML report for session analysis:

# Generate and open report
osiris logs html --open

# Generate for specific session
osiris logs html --session <session_id> --open

# Output to custom directory
osiris logs html --output reports/

Options:

  • --session: Specific session ID (default: most recent)
  • --output: Output directory (default: logs/)
  • --open: Open report in browser after generation

The HTML report includes:

  • Session overview with pipeline metrics
  • Execution timeline visualization
  • Connection details (masked secrets)
  • Performance metrics and charts
  • Artifact browser
  • E2B-specific metrics (bootstrap time, badges)

logs gc - Garbage Collection

Clean up old session logs:

osiris logs gc [--keep-recent N] [--older-than DAYS]

Options:

  • --keep-recent N: Keep N most recent sessions (default: 10)
  • --older-than DAYS: Remove sessions older than DAYS

logs aiop - Export AI Operation Package

Generate a structured, multi-layered AI Operation Package (AIOP) for LLM consumption:

osiris logs aiop [--session SESSION_ID | --last] [OPTIONS]

Required Arguments (one of):

  • --session SESSION_ID: Export specific session by ID
  • --last: Export the most recent session

Output Options:

  • --output PATH: Write to file instead of stdout
  • --format {json|md}: Output format (default: json)
    • json: Full AIOP in JSON-LD format with all layers
    • md: Human-readable Markdown run-card summary

Size Control:

  • --max-core-bytes N: Maximum size for Core package (default: 300000)
  • --timeline-density {minimal|medium|verbose}: Event filtering (default: medium)
    • minimal: Key events only (start, end, errors)
    • medium: Important events and step transitions
    • verbose: All events including debug
  • --metrics-topk N: Keep top N metrics per step (default: 10)
  • --schema-mode {summary|detailed}: Schema detail level (default: summary)

Annex Policy (for large runs):

  • --policy {core|annex}: Export policy (default: core)
    • core: Single package with truncation if needed
    • annex: Core + NDJSON shard files for full data
  • --annex-dir DIR: Directory for annex files (default: .aiop-annex)
  • --compress {none|gzip}: Compression for annex files (default: none)

Configuration Precedence:

CLI flags > $OSIRIS_AIOP_* environment > osiris.yaml > defaults

Environment Variables:

  • OSIRIS_AIOP_MAX_CORE_BYTES: Maximum Core size in bytes
  • OSIRIS_AIOP_TIMELINE_DENSITY: Timeline filtering level
  • OSIRIS_AIOP_METRICS_TOPK: Number of top metrics to keep
  • OSIRIS_AIOP_SCHEMA_MODE: Schema detail level

Exit Codes:

  • 0: Success - AIOP generated successfully
  • 1: General error (e.g., I/O failure)
  • 2: Invalid arguments or session not found
  • 4: AIOP was truncated due to size limits (warning)

Examples:

# Export last run as JSON
osiris logs aiop --last

# Generate Markdown run-card to file
osiris logs aiop --last --format md --output runcard.md

# Export with annex shards and compression
osiris logs aiop --session run_123 --policy annex --compress gzip

# Control package size with aggressive truncation
osiris logs aiop --last --max-core-bytes 50000 --timeline-density minimal

# Check if truncation occurred
osiris logs aiop --last --format json | jq '.metadata.truncated'

Configuration Precedence: AIOP settings follow this precedence order:

  1. CLI flags (highest priority) - Command line arguments
  2. Environment variables - OSIRIS_AIOP_* prefixed variables
  3. YAML configuration - Settings in osiris.yaml under aiop: section
  4. Built-in defaults (lowest priority)

Example:

# YAML sets timeline_density: high
# ENV sets OSIRIS_AIOP_TIMELINE_DENSITY=low
# CLI sets --timeline-density medium
# Result: medium (CLI wins)

AIOP Structure: The exported package contains four layers:

  1. Evidence Layer: Timeline, metrics, errors, artifacts
  2. Semantic Layer: DAG, components, OML specification
  3. Narrative Layer: Natural language description with citations
  4. Metadata Layer: Package metadata, size hints, truncation markers, config_effective

The metadata.config_effective field shows the actual configuration used and its source:

{
  "timeline_density": {
    "value": "medium",
    "source": "CLI"  // or "ENV", "YAML", "DEFAULT"
  }
}

See AIOP Examples for complete structure.

Pro Mode Commands

dump-prompts - Export LLM Prompts

Export system prompts for customization:

osiris dump-prompts --export

Creates .osiris_prompts/ directory with:

  • conversation_system.txt - Main AI personality
  • sql_generation_system.txt - SQL generation rules
  • user_prompt_template.txt - Context building template

Use with osiris chat --pro-mode to apply custom prompts.

Example Workflows

Standard Pipeline Development

# 1. Initialize configuration
osiris init

# 2. Set up connections and secrets
vi osiris_connections.yaml
vi .env

# 3. Create pipeline via conversation
osiris chat

# 4. Compile the generated OML
osiris compile output/pipeline.oml

# 5. Run locally
osiris run --last-compile

# 6. Generate report
osiris logs html --open

E2B Cloud Execution

# Set E2B API key
export E2B_API_KEY="your-key-here"

# Compile pipeline
osiris compile pipeline.oml

# Run in E2B with custom resources
osiris run --last-compile --e2b \
  --e2b-cpu 4 \
  --e2b-mem 8 \
  --e2b-timeout 3600 \
  --e2b-install-deps

# View results
osiris logs html --open

Connection Management

# List all connections
osiris connections list

# Test connectivity
osiris connections doctor

# View as JSON for scripting
osiris connections list --json | jq '.connections.mysql'

Environment Variables

Key environment variables used by Osiris:

  • E2B_API_KEY: API key for E2B cloud execution
  • OSIRIS_VALIDATION: Override validation mode (strict/warn/off)
  • MYSQL_PASSWORD, MYSQL_DATABASE: MySQL credentials
  • SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY: Supabase credentials

Output Locations

All output locations respect the filesystem contract configured in osiris.yaml. Default paths are shown below.

CLI Session Logs

CLI commands (like osiris connections list, osiris connections doctor) create session logs under:

<base_path>/<run_logs_dir>/<session_id>/

Default: run_logs/<session_id>/ (relative to current directory)

Configuration (in osiris.yaml):

filesystem:
  base_path: ""              # Base path for all files (default: current directory)
  run_logs_dir: "run_logs"   # Directory for CLI session logs

Structure:

run_logs/
└── connections_1234567890/
    ├── osiris.log       # Main session log
    ├── events.jsonl     # Structured events
    └── metrics.jsonl    # Performance metrics

Pipeline Execution Logs

Pipeline runs (via osiris run) use the full filesystem contract with profile support:

<base_path>/<run_logs_dir>/[<profile>/]<pipeline_slug>/<run_ts>_<run_id>-<manifest_short>/

Structure:

run_logs/
└── dev/                    # Profile (if profiles.enabled)
    └── orders_etl/         # Pipeline slug
        └── 20250116T120000Z_001-abc1234/
            ├── osiris.log
            ├── debug.log
            ├── events.jsonl
            ├── metrics.jsonl
            └── artifacts/

Build Artifacts

Compiled pipelines and manifests:

<base_path>/<build_dir>/[<profile>/]pipelines/<pipeline_slug>/<manifest_short>-<manifest_hash>/

Default: build/pipelines/<pipeline_slug>/...

Other Outputs

  • Generated pipelines: <base_path>/<outputs.directory>/ (default: output/)
  • AIOP exports: <base_path>/<aiop_dir>/... (default: aiop/)
  • HTML reports: Generated in session directories
  • Session cache: <base_path>/.osiris/sessions/
  • Discovery cache: <base_path>/.osiris/cache/

See Filesystem Contract for full configuration options.