AI-DLC Rules / Shared Rules

Overconfidence Prevention Guide

Avoiding AI overreach

Overconfidence Prevention Guide

Problem Statement

AI-DLC was exhibiting overconfidence by not asking enough clarifying questions, even for complex project intent statements. This led to assumptions being made instead of gathering proper requirements.

Root Cause Analysis

The overconfidence issue was caused by directives in multiple stages that encouraged skipping questions:

  1. Functional Design: "Skip entire categories if not applicable"
  2. User Stories: "Use categories as inspiration, NOT as mandatory checklist"
  3. Requirements Analysis: Similar patterns encouraging minimal questioning
  4. NFR Requirements: "Only if" conditions that discouraged thorough analysis

These directives were telling the AI to avoid asking questions rather than encouraging comprehensive requirements gathering.

Solution Implemented

Updated Question Generation Philosophy

OLD APPROACH: "Only ask questions if absolutely necessary" NEW APPROACH: "When in doubt, ask the question - overconfidence leads to poor outcomes"

Key Changes Made

1. Requirements Analysis Stage

  • Changed from "only if needed" to "ALWAYS create questions unless exceptionally clear"
  • Added comprehensive evaluation areas (functional, non-functional, business context, technical context)
  • Emphasized proactive questioning approach

2. User Stories Stage

  • Removed "skip entire categories" directive
  • Added comprehensive question categories to evaluate
  • Enhanced answer analysis requirements
  • Strengthened follow-up question mandates

3. Functional Design Stage

  • Replaced "only if" conditions with comprehensive evaluation
  • Added more question categories (data flow, integration points, error handling)
  • Strengthened ambiguity detection and resolution requirements

4. NFR Requirements Stage

  • Expanded question categories beyond basic NFRs
  • Added reliability, maintainability, and usability considerations
  • Enhanced answer analysis for technical ambiguities

New Guiding Principles

  1. Default to Asking: When there's any ambiguity, ask clarifying questions
  2. Comprehensive Coverage: Evaluate ALL relevant categories, don't skip areas
  3. Thorough Analysis: Carefully analyze ALL user responses for ambiguities
  4. Mandatory Follow-up: Create follow-up questions for ANY unclear responses
  5. No Proceeding with Ambiguity: Don't move forward until ALL ambiguities are resolved

Implementation Guidelines

For Question Generation

  • Evaluate ALL question categories, don't skip any
  • Ask questions wherever clarification would improve quality
  • Include comprehensive question categories in each stage
  • Default to inclusion rather than exclusion of questions

For Answer Analysis

  • Look for vague responses: "depends", "maybe", "not sure", "mix of", "somewhere between"
  • Detect undefined terms and references to external concepts
  • Identify contradictory or incomplete answers
  • Create follow-up questions for ANY ambiguities

For Follow-up Questions

  • Create separate clarification files when ambiguities are detected
  • Ask specific questions to resolve each ambiguity
  • Don't proceed until ALL unclear responses are clarified
  • Be thorough - better to over-clarify than under-clarify

Quality Assurance

Red Flags to Watch For

  • Stages completing without asking any questions on complex projects
  • Proceeding with vague or ambiguous user responses
  • Skipping entire question categories without justification
  • Making assumptions instead of asking for clarification

Success Indicators

  • Appropriate number of clarifying questions for project complexity
  • Thorough analysis of user responses with follow-up when needed
  • Clear, unambiguous requirements before proceeding to implementation
  • Reduced need for changes during later stages due to better upfront clarification

Maintenance

This guide should be referenced when:

  • Adding new stages to AI-DLC
  • Updating existing stage instructions
  • Reviewing AI-DLC performance for overconfidence issues
  • Training team members on AI-DLC question generation principles

Key Takeaway

It's better to ask too many questions than to make incorrect assumptions. The cost of asking clarifying questions upfront is far less than the cost of implementing the wrong solution based on assumptions.