ADR-004: ComfyUI Mock Mode for Development and Testing

Date: 2025-11-06 Status: 🤔 Proposed Decision Makers: Engineering Team Approval Required: Before implementation Related ADRs: None

Executive Summary

This ADR proposes a mock execution mode for ComfyUI workers that enables rapid development and testing without requiring full ComfyUI installation, GPU hardware, or model downloads. Mock outputs are defined per-component in ECLI and returned by workers as if real generation occurred.

Current Gap:

❌ Long setup time - ComfyUI + custom nodes + models = 8-15 minute cold start
❌ Hardware dependency - Requires GPU or slow CPU mode
❌ No fast iteration - Testing workflow changes requires full execution
❌ Difficult local dev - Developers need full ComfyUI stack locally

Proposed Solution:

🎯 Component-level mock outputs - Each workflow has a mockOutput.json in ECLI
🚀 Worker-side mock detection - COMFYUI_MOCK_MODE=true bypasses real ComfyUI
📦 Azure blob copy - Mock outputs copied to expected job output locations
🔄 Transparent to API - API/frontend unaware, receives normal job results

Impact:

Before: 8-15min setup, GPU required, slow iteration
After: <30s setup, CPU-only, instant "generation" results

Context
Problem Statement
Decision
Technical Design
Implementation Details
Consequences
Alternatives Considered

Context

Current ComfyUI Workflow Execution

API receives job → Redis queue → Worker claims job
                                       ↓
                          Worker.processJob(jobData)
                                       ↓
                    ComfyUIRestStreamConnector.processJob()
                                       ↓
                    Submit workflow to ComfyUI via REST API
                                       ↓
                    ComfyUI executes nodes (10s-10min)
                                       ↓
                    ComfyUI saves output to Azure blob
                                       ↓
                    ComfyUI returns output URLs
                                       ↓
                    Worker returns JobResult to API
                                       ↓
                    API returns result to frontend

Current Pain Points

Development Friction:

Full ComfyUI installation required (~2-5GB download)
Custom nodes installation (64+ nodes, ~3-5 minutes)
Model downloads (2-15GB per model, 5+ minutes)
GPU required for reasonable performance
Total setup: 8-15 minutes minimum

Testing Friction:

Every workflow change requires full execution
Can't test job queue logic without ComfyUI
Difficult to reproduce specific outputs
Slow feedback loops

Local Development Friction:

Developers need full ComfyUI stack
Hardware requirements (GPU preferred)
Inconsistent environments between devs

Problem Statement

How do we enable rapid development and testing of the job queue system, API, and frontend without requiring full ComfyUI execution?

Requirements:

✅ Fast setup (< 1 minute)
✅ No GPU required
✅ Reproducible outputs
✅ Transparent to API/frontend
✅ Easy to add new mock outputs
✅ Per-component/workflow mocking

Decision

We will implement component-level mock mode with the following architecture:

1. Mock Output Storage (File System)

Decision: Use file-based storage in container (Simplest approach)

Mock outputs stored in container file system:

/workspace/comfyui-mocks/
├── txt2img-flux.png
├── upscale-esrgan.png
├── video-gen.mp4
├── image-merge.png
└── ...

Naming Convention: {workflow_name}.{extension}

Why File System over Redis/Database:

✅ Simplest implementation - no external dependencies
✅ Zero configuration - just mount/copy files into container
✅ No API changes needed - workers have direct file access
✅ Version controlled - mock files in repo alongside code
✅ Fast lookup - direct file path construction
✅ Easy to update - replace files, rebuild container

Storage Location:

Development: /workspace/comfyui-mocks/ (local directory)
Container: Baked into base machine image or mounted volume

2. Worker Mock Mode Detection

Environment Variable:

bash

COMFYUI_MOCK_MODE=true  # Enables mock mode
COMFYUI_MOCK_DIR=/workspace/comfyui-mocks  # Mock files directory

Worker Behavior:

typescript

// In ComfyUIRestStreamConnector.processJob()
if (process.env.COMFYUI_MOCK_MODE === 'true') {
  return this.processMockJob(jobData, progressCallback);
}
// Otherwise: normal ComfyUI execution

3. Mock Job Execution Flow

Worker claims job → Detects COMFYUI_MOCK_MODE=true
                              ↓
           Extract workflow_name from jobData.payload
                              ↓
      Construct mock file path: `/workspace/comfyui-mocks/{workflow_name}.png`
                              ↓
           Check if mock file exists
                              ↓
         Read mock file from disk
                              ↓
         Generate target blob path from job context
         (e.g., user-123/job-456/output.png)
                              ↓
      Upload mock file to Azure blob at target location
                              ↓
      Return JobResult with new CDN URL
                              ↓
      API receives result (thinks it was real generation)

4. Azure Blob Upload Mechanism

Upload Implementation:

typescript

async uploadMockOutputToJobLocation(
  workflowName: string,
  jobContext: { storage: { cdnUrl, prefix }, jobId }
): Promise<string> {
  // 1. Construct mock file path
  const mockDir = process.env.COMFYUI_MOCK_DIR || '/workspace/comfyui-mocks';
  const mockFilePath = `${mockDir}/${workflowName}.png`;

  // 2. Check if mock file exists
  if (!fs.existsSync(mockFilePath)) {
    throw new Error(`No mock file found for workflow: ${workflowName} at ${mockFilePath}`);
  }

  // 3. Read mock file from disk
  const mockFileBuffer = await fs.promises.readFile(mockFilePath);

  // 4. Generate target path
  const targetPath = `${jobContext.prefix}/${jobContext.jobId}/output.png`;

  // 5. Upload to Azure blob storage
  await azureBlobClient.upload(targetPath, mockFileBuffer);

  // 6. Return new CDN URL
  return `${jobContext.storage.cdnUrl}/${targetPath}`;
}

Technical Design

Directory Structure

Mock Files Location:

/workspace/comfyui-mocks/
├── txt2img-flux.png          # Text-to-image workflow mock
├── upscale-esrgan.png         # Upscaler workflow mock
├── video-gen.mp4              # Video generation mock
├── image-merge.png            # Image merging mock
├── background-removal.png     # Background removal mock
└── README.md                  # Documentation for adding mocks

File Naming Convention:

Format: {workflow_name}.{extension}
Workflow name must match exactly (case-sensitive)
Extension determines MIME type (.png, .jpg, .mp4, .webp, etc.)

Development Setup:

bash

# Create mock directory
mkdir -p /workspace/comfyui-mocks

# Add sample mock outputs
cp sample-images/flux-output.png /workspace/comfyui-mocks/txt2img-flux.png
cp sample-images/upscaled.png /workspace/comfyui-mocks/upscale-esrgan.png

Container Setup:

dockerfile

# In machine Dockerfile
COPY comfyui-mocks/ /workspace/comfyui-mocks/

Worker Connector Changes

New Method: processMockJob()

typescript

// In ComfyUIRestStreamConnector
async processMockJob(jobData: JobData, progressCallback: ProgressCallback): Promise<JobResult> {
  const startTime = Date.now();

  // 1. Extract workflow name
  const workflowName = jobData.payload?.workflow_name || jobData.metadata?.workflow_name;

  if (!workflowName) {
    throw new Error('No workflow_name found in job payload or metadata');
  }

  logger.info(`[MOCK MODE] Processing mock job for workflow: ${workflowName}`);

  // 2. Construct mock file path
  const mockDir = process.env.COMFYUI_MOCK_DIR || '/workspace/comfyui-mocks';
  const mockFilePath = path.join(mockDir, `${workflowName}.png`);

  // 3. Check if mock file exists
  if (!fs.existsSync(mockFilePath)) {
    throw new Error(`No mock file found for workflow: ${workflowName} at ${mockFilePath}`);
  }

  // 4. Simulate progress updates
  await progressCallback({ job_id: jobData.id, progress: 10, message: 'Mock: Initializing' });
  await this.sleep(500);
  await progressCallback({ job_id: jobData.id, progress: 50, message: 'Mock: Processing' });
  await this.sleep(500);

  // 5. Upload mock file to job-specific Azure blob location
  const outputUrl = await this.uploadMockOutputToJobLocation(
    workflowName,
    jobData.ctx // Contains storage config + prefix
  );

  await progressCallback({ job_id: jobData.id, progress: 100, message: 'Mock: Complete' });

  logger.info(`[MOCK MODE] Completed mock job for ${workflowName}: ${outputUrl}`);

  // 6. Return result identical to real ComfyUI output
  return {
    success: true,
    data: {
      image_url: outputUrl,
      mime_type: 'image/png',
      content_type: 'image',
      mock_mode: true // For debugging
    },
    processing_time_ms: Date.now() - startTime,
    service_metadata: {
      service_version: this.version,
      service_type: this.service_type,
      mock_execution: true
    }
  };
}

Implementation Details

Phase 1: Core Mock Mode (MVP)

Setup:

✅ Create /workspace/comfyui-mocks/ directory structure
✅ Add sample mock files for 2-3 core workflows
✅ Add mock directory to .gitignore (or version control sample mocks)

Worker Changes:

✅ Add COMFYUI_MOCK_MODE environment variable support
✅ Add COMFYUI_MOCK_DIR environment variable (defaults to /workspace/comfyui-mocks)
✅ Implement processMockJob() in ComfyUIWebSocketConnector
✅ Implement uploadMockOutputToJobLocation() helper
✅ Add file existence validation with clear error messages

Container Changes:

✅ Copy mock files into container image (optional, for baked mocks)
✅ Or mount mock directory as volume (for local development)

Testing:

✅ Unit tests for processMockJob()
✅ Unit tests for mock file discovery and validation
✅ Integration test: Submit job → Worker returns mock → Verify blob upload
✅ E2E test: API → Queue → Mock Worker → Result

Phase 2: Enhanced Mocking (Future)

File Extensions and MIME Type Detection:

bash

/workspace/comfyui-mocks/
├── txt2img-flux.png      # Auto-detect: image/png
├── video-gen.mp4         # Auto-detect: video/mp4
├── audio-gen.wav         # Auto-detect: audio/wav

Multiple Mock Variants:

bash

/workspace/comfyui-mocks/
├── txt2img-flux.default.png
├── txt2img-flux.high-quality.png
├── txt2img-flux.error.png

Mock Selection:

bash

COMFYUI_MOCK_VARIANT=high-quality  # Which mock variant to use

Metadata Files (Optional):

json

// /workspace/comfyui-mocks/txt2img-flux.meta.json
{
  "width": 512,
  "height": 512,
  "description": "Sample flux output",
  "variants": ["default", "high-quality", "error"]
}

Consequences

Positive

✅ Rapid Local Development

No ComfyUI installation needed
CPU-only development machines work
Instant "generation" results

✅ Faster Testing

Test job queue logic without GPU
Reproducible outputs for E2E tests
Faster CI/CD pipelines

✅ Cost Reduction

No GPU instances needed for development
Fewer model downloads (bandwidth savings)
Faster developer onboarding

✅ Transparent to API

No API changes required
Frontend receives normal responses
Easy to toggle mock mode on/off

Negative

❌ Maintenance Overhead

Need to update mock files when workflows change output format
Mock files need to be kept in sync with workflow expectations

❌ Not True E2E

Doesn't test actual ComfyUI execution
Won't catch ComfyUI-specific bugs
Need real execution tests for production validation

❌ Storage Requirements

Mock files consume disk space (typically 0.5-5MB per workflow)
Need to manage mock file versions

Mitigation Strategies

Mock Output Staleness:

Automated tests to verify mock outputs match current workflow output formats
Version mock files alongside workflow versions
Document mock file update process in README

Storage Management:

Compress mock files where appropriate
Use symbolic links for duplicate mocks
Consider on-demand download for large mocks (Phase 2)

Production Safety:

COMFYUI_MOCK_MODE defaults to false
Production environments never set mock mode
CI includes both mock and real execution tests
Clear [MOCK MODE] prefix in all log messages

Alternatives Considered

Alternative 1: Simulation Connector (Rejected)

Approach: Use existing SimulationConnector for all services

Rejected Because:

❌ SimulationConnector is service-agnostic (no workflow awareness)
❌ Doesn't understand component-level mocking
❌ Doesn't copy blobs to expected locations
❌ Not ComfyUI-specific

Alternative 2: Mock ComfyUI Server (Rejected)

Approach: Create fake ComfyUI HTTP/WebSocket server that returns mocks

Rejected Because:

❌ More complex implementation (full server mock)
❌ Slower (still goes through WebSocket protocol)
❌ Doesn't gain us anything over direct mocking
❌ Harder to maintain

Alternative 3: Redis-Based Mock Storage (Rejected)

Approach: Store mock output metadata in Redis, pointing to CDN URLs

Rejected Because:

❌ Requires ECLI to publish mock data on startup
❌ Adds external dependency (though Redis already used)
❌ More complex than file system
❌ Mock data lost if Redis flushes
❌ Harder to version control mock outputs

Alternative 4: Database Mock Storage (Rejected)

Approach: Store mock outputs in database with workflow schema

Rejected Because:

❌ Workers don't currently have DB access
❌ Requires schema migrations
❌ Slower implementation
❌ More complex than file system
❌ Overkill for simple mock data

Open Questions

Q: Should mock mode also skip model downloads? A: Yes - mock mode should bypass all ComfyUI installation (covered by GPU_MODE=mock + COMPONENTS="")
Q: How do we handle workflows without mock outputs defined? A: Throw clear error: "No mock output defined for workflow: {name}"
Q: Should API know about mock mode? A: No - API is agnostic, workers handle mocking transparently
Q: What about collection-based components? A: Collections resolve to individual workflows, each has its own mock output

Decision

Status: 🤔 Proposed

We will implement Phase 1: Core Mock Mode as described above:

Mock files stored in /workspace/comfyui-mocks/ directory
File naming convention: {workflow_name}.{extension}
Workers detect COMFYUI_MOCK_MODE=true and bypass ComfyUI
Mock files uploaded to Azure blob with job-specific paths
API/frontend receive identical responses as real execution

Implementation Priority: HIGH (enables fast local development)

Estimated Effort:

Mock directory setup: 1 hour
Worker changes: 4 hours
Testing: 3 hours
Total: ~1 day (simplified from original 2-day estimate)

References

Approval Required From:

[ ] Tech Lead
[ ] DevOps (for environment variable rollout)
[ ] Frontend (for awareness of mock mode in testing)

ADR-004: ComfyUI Mock Mode for Development and Testing ​

Executive Summary ​

Table of Contents ​

Context ​

Current ComfyUI Workflow Execution ​

Current Pain Points ​

Problem Statement ​

Decision ​

1. Mock Output Storage (File System) ​

2. Worker Mock Mode Detection ​

3. Mock Job Execution Flow ​

4. Azure Blob Upload Mechanism ​

Technical Design ​

Directory Structure ​

Worker Connector Changes ​

Implementation Details ​

Phase 1: Core Mock Mode (MVP) ​

Phase 2: Enhanced Mocking (Future) ​

Consequences ​

Positive ​

Negative ​

Mitigation Strategies ​

Alternatives Considered ​

Alternative 1: Simulation Connector (Rejected) ​

Alternative 2: Mock ComfyUI Server (Rejected) ​

Alternative 3: Redis-Based Mock Storage (Rejected) ​

Alternative 4: Database Mock Storage (Rejected) ​

Open Questions ​

Decision ​

References ​

ADR-004: ComfyUI Mock Mode for Development and Testing

Executive Summary

Table of Contents

Context

Current ComfyUI Workflow Execution

Current Pain Points

Problem Statement

Decision

1. Mock Output Storage (File System)

2. Worker Mock Mode Detection

3. Mock Job Execution Flow

4. Azure Blob Upload Mechanism

Technical Design

Directory Structure

Worker Connector Changes

Implementation Details

Phase 1: Core Mock Mode (MVP)

Phase 2: Enhanced Mocking (Future)

Consequences

Positive

Negative

Mitigation Strategies

Alternatives Considered

Alternative 1: Simulation Connector (Rejected)

Alternative 2: Mock ComfyUI Server (Rejected)

Alternative 3: Redis-Based Mock Storage (Rejected)

Alternative 4: Database Mock Storage (Rejected)

Open Questions

Decision

References