Skip to content

Application - Full Case Studies

Summary

This capstone chapter synthesizes all course concepts through detailed case studies of AI projects that succeeded or failed based on problem framing decisions. You'll analyze complete applications of The Loop framework across classic ML and GenAI domains, identify common pivot patterns that transcend technology choices, and develop a practical Monday Morning Checklist for applying problem framing skills to any AI initiative.

The chapter moves from theoretical understanding to professional practice, demonstrating how experienced practitioners diagnose misaligned projects early, pivot systematically rather than reactively, and build organizational muscle for better upstream decision-making. By studying both successful pivots and costly failures, you'll develop pattern recognition abilities that make problem framing feel less like following a checklist and more like exercising practiced judgment.

Concepts Covered

This chapter integrates concepts from across the entire course:

  1. Problem Framing (Chapters 1-3)
  2. The Loop Framework (Chapter 3)
  3. Outcome Metrics (Chapter 3)
  4. Problem Deconstruction (Chapter 3)
  5. Alternatives Menu (Chapter 2)
  6. Trade-off Analysis (Chapter 3)
  7. Signals and Diagnosis (Chapter 4)
  8. Pivot Decisions (Chapter 5)
  9. Persist Decisions (Chapter 5)
  10. Kill Signals (Chapter 4)
  11. Pattern Bridges (synthesis concept)
  12. Monday Morning Checklist (applied framework)
  13. Case Study Analysis (professional skill)
  14. Cautionary Tales (learning from failures)
  15. ML Pivot (Chapter 5)

Prerequisites

This chapter assumes mastery of all previous chapters:

Learning Objectives

By the end of this chapter, you will be able to:

  1. Analyze complete AI case studies using The Loop framework to identify where framing succeeded or failed
  2. Diagnose common failure patterns across diverse AI projects (classic ML and GenAI)
  3. Identify pivot patterns that recur across different problem domains and solution types
  4. Apply the Monday Morning Checklist to assess new AI projects systematically
  5. Synthesize course concepts into a coherent problem framing practice ready for professional application
  6. Evaluate peer problem framings using the same critical lens applied to case studies
  7. Create a complete problem framing portfolio demonstrating systematic thinking from ambiguity to implementation

Introduction: What Separates Success from Failure

You've seen the statistics: 85% of AI projects fail to deliver business value. The technical literature focuses on model performance, data quality, and infrastructure challenges. But when experienced practitioners diagnose failures post-mortem, a different pattern emerges.

Projects fail because teams built the wrong thing—technically competent solutions to incorrectly framed problems. A churn prediction model with 92% accuracy that doesn't reduce churn. A fraud detection system so sensitive it blocks legitimate transactions. A chatbot that answers questions perfectly but doesn't solve the underlying knowledge access problem.

The difference between successful and unsuccessful AI practitioners is not model architecture knowledge or coding skill. It's the systematic habit of asking upstream questions before committing to implementation:

  • What outcome are we actually pursuing?
  • What are we assuming about the problem structure?
  • What alternatives exist beyond the obvious approach?
  • How will we know if we're solving the right problem?
  • What signals would tell us to pivot?

This chapter shows you these habits in action through detailed case studies. You'll see how practitioners applied The Loop to navigate ambiguity, how they recognized early signals of misalignment, and how they made evidence-based decisions to persist or pivot. More importantly, you'll see the common patterns that transcend specific domains—the repeatable structures that make problem framing a transferable skill rather than domain-specific intuition.

Classic ML Pivot Case Studies

Case Study 1: Churn Prediction → Customer Success Intervention System

Initial Framing (Incorrect)

A B2B SaaS company framed their retention challenge as: "We need to predict which customers will churn so we can intervene." The team built a classification model using historical usage data, support tickets, and billing patterns to predict 30-day churn risk.

Loop Analysis - Original Frame

Outcome: Reduce customer churn by 15% within 6 months Atomic Unit: Individual customer account Decomposition: - Identify at-risk customers (model task) - Trigger alerts to customer success team - Execute retention campaigns (out of scope for ML team)

Alternatives Considered: 1. Logistic regression (interpretable coefficients for CS team) 2. Gradient boosting (higher accuracy, less interpretable) 3. Rule-based risk scoring (simple, no training required)

Trade-offs: Team chose gradient boosting for accuracy, accepting interpretability loss

Signals Defined: - Model accuracy (AUC-ROC on held-out test set) - Churn rate reduction (6-month trend) - Customer success team adoption (% of alerts acted on)

What Actually Happened

The model achieved 89% AUC-ROC and was deployed to production. The customer success team received alerts for high-risk accounts. After 3 months:

  • Churn rate: unchanged
  • CS team alert response rate: 23%
  • Most common CS team feedback: "We already knew these customers were struggling"

Diagnosis Through Signals

The weak signal came not from model metrics (which looked excellent) but from user behavior: the CS team ignored most alerts. Interviews revealed:

  • CS team already tracked usage patterns manually
  • The model identified customers currently struggling, not those who would struggle soon
  • By the time predictions triggered, CS team had exhausted intervention options
  • The real constraint was what intervention to apply, not who needs help

The Pivot

The team returned to The Loop with new understanding:

New Outcome: Enable CS team to intervene 60-90 days before risk escalates (not 30 days)

New Decomposition: - Identify leading indicators of future struggle (not current struggle) - Recommend specific interventions based on struggle type - Track intervention effectiveness to learn what works

New Alternatives: 1. Early warning system (predict engagement decline before revenue impact) 2. Intervention recommendation engine (match struggle patterns to proven interventions) 3. Causal inference model (identify which interventions actually reduce churn vs. selection effects)

New Trade-offs: Accepted lower precision in early warnings in exchange for more lead time; prioritized actionable insights over prediction accuracy

Outcome After Pivot

The team built an early warning system that flagged accounts showing usage decline patterns 60 days before historical churn points, paired with recommended interventions based on similar past customers who were saved. Six months post-pivot:

  • Churn reduction: 12% (close to original goal)
  • CS team adoption: 81% of recommendations acted on
  • New discovery: Product onboarding gaps were the #1 addressable churn driver

Key Lessons

  1. Prediction accuracy doesn't equal business value: The original model was technically excellent but strategically misaligned
  2. User behavior is a signal, not noise: CS team ignoring alerts was data, not a training problem
  3. Atomic unit matters: Moving from "at-risk customer" to "intervention opportunity" changed the entire problem structure
  4. Leading indicators > lagging indicators: Predicting current state is easier but less valuable than predicting future risk while it's preventable

Pattern: This case demonstrates the Prediction → Action pivot—moving from forecasting outcomes to enabling interventions.

Case Study 2: Fraud Detection → Risk Scoring and Adaptive Limits

Initial Framing (Incorrect)

A fintech company framed payment fraud as: "We need to detect fraudulent transactions before they complete." The team built a binary classifier to approve or reject transactions in real-time based on historical fraud patterns.

Loop Analysis - Original Frame

Outcome: Reduce fraud losses by 40% while maintaining customer experience

Atomic Unit: Individual transaction

Decomposition: - Extract transaction features (amount, merchant, location, time) - Train classifier on historical fraud labels - Deploy real-time prediction API - Block transactions predicted as fraud

Alternatives Considered: 1. Rule-based system (high precision, low recall) 2. Random forest (balanced performance) 3. Neural network (highest accuracy potential) 4. Anomaly detection (catch novel fraud patterns)

Trade-offs: Chose random forest for balance of accuracy and interpretability; accepted latency constraint of <100ms

Signals Defined: - Fraud detection rate (% of fraud caught) - False positive rate (% of legitimate transactions blocked) - Customer complaint rate (proxy for UX impact)

What Actually Happened

The model achieved 94% fraud detection with 2% false positive rate—excellent by industry standards. But three months post-deployment:

  • Fraud losses: reduced 35% (close to goal)
  • Customer complaints: increased 300%
  • Revenue impact: $2M monthly from abandoned purchases after false declines
  • VIP customer churn: +8% quarter-over-quarter

Diagnosis Through Signals

The false positive rate looked low (2%), but the absolute volume was crushing customer experience. Analysis revealed:

  • 2% of 50M monthly transactions = 1M false declines
  • Customers who experienced false declines had 40% cart abandonment on retry
  • High-value customers (who spent more, triggering fraud signals) were disproportionately affected
  • The binary approve/reject decision treated all risks equally

The Pivot

New Outcome: Minimize total cost (fraud losses + customer friction + lost revenue) rather than just fraud losses

New Decomposition: - Estimate transaction risk on continuous scale (not binary) - Adjust friction level based on risk score (not blanket block) - Segment customers by value and risk tolerance - Monitor customer behavior post-friction

New Alternatives: 1. Risk scoring + adaptive friction (step-up authentication for risky transactions) 2. Customer-specific risk thresholds (VIP customers get higher limits) 3. Velocity limits + real-time risk adjustment (dynamic fraud detection) 4. Hybrid approach: rules for obvious fraud, ML for gray area, humans for VIPs

New Trade-offs: Accepted slightly higher fraud losses in exchange for better customer experience and revenue retention; prioritized precision over recall for high-friction interventions

Outcome After Pivot

The team rebuilt the system as a risk scoring platform with four response tiers:

  1. Very low risk: instant approval
  2. Low-medium risk: SMS verification
  3. Medium-high risk: additional authentication
  4. Very high risk: human review

Customer segmentation allowed VIP customers higher risk thresholds before friction was introduced. Six months post-pivot:

  • Total cost reduction: 52% (fraud + friction + revenue)
  • Customer complaints: returned to baseline
  • VIP churn: back to pre-deployment levels
  • Fraud losses: reduced 31% (less than original goal but better total outcome)

Key Lessons

  1. Optimize the right objective: Minimizing fraud losses alone created greater total cost
  2. Binary decisions are often wrong: Continuous risk scores enable proportional responses
  3. Atomic unit drives solution: Moving from "transaction" to "customer transaction in context" enabled segmentation
  4. Second-order effects matter: False positive rate was low but impact was devastating

Pattern: This case demonstrates the Classification → Scoring pivot—moving from binary decisions to continuous risk assessment with adaptive responses.

Case Study 3: Recommendation Engine → Search Optimization

Initial Framing (Incorrect)

An e-commerce platform framed product discovery as: "We need to recommend products customers will buy." The team built a collaborative filtering recommendation system using purchase history and browsing behavior.

Loop Analysis - Original Frame

Outcome: Increase conversion rate by 20% through personalized recommendations

Atomic Unit: User session

Decomposition: - Collect user behavior data (clicks, purchases, time-on-page) - Train collaborative filtering model - Display personalized recommendations on homepage and product pages - Track click-through and conversion rates

Alternatives Considered: 1. Collaborative filtering (user-user similarity) 2. Content-based filtering (item similarity) 3. Matrix factorization (latent factors) 4. Deep learning embeddings (learn complex patterns)

Trade-offs: Chose matrix factorization for scalability and cold-start performance; accepted lack of interpretability

Signals Defined: - Recommendation click-through rate (CTR) - Conversion rate from recommendations - Revenue attributed to recommendations - Coverage (% of products recommended)

What Actually Happened

The recommendation engine launched with strong initial metrics:

  • Recommendation CTR: 8.3% (vs. 2.1% baseline for non-personalized)
  • Conversion rate: +12% for users who clicked recommendations

But after six months, concerning patterns emerged:

  • Overall site conversion rate: +4% (far below 20% goal)
  • Search usage: declining 15% quarter-over-quarter
  • Customer surveys: "Hard to find what I'm looking for"
  • Product catalog coverage: recommendations showed only 12% of inventory

Diagnosis Through Signals

The recommendation system was working too well—it optimized for engagement with familiar products at the expense of discovery. Deep dive analysis revealed:

  • Recommendations reinforced past purchases (users who bought hammers saw more hammers)
  • Search abandonment was high (users gave up when search didn't find specific needs)
  • Conversion gains came from existing intent, not discovery
  • Users with specific needs (search-driven) had worse experience than browsers (recommendation-driven)

The real problem: customers already knew what they wanted, but search couldn't find it.

The Pivot

New Outcome: Help customers find what they need, whether they know what it is or not

New Decomposition: - Intent-driven customers (searching for specific items) → improve search relevance - Discovery-driven customers (browsing, open to suggestions) → improve recommendations - Measure success separately for each journey

New Alternatives: 1. Semantic search (understand query intent, not just keywords) 2. Query expansion (suggest related terms for failed searches) 3. Hybrid search-recommendation (recommend when search returns few results) 4. Faceted navigation improvements (help customers filter effectively)

New Trade-offs: Redirected 60% of engineering effort to search quality; accepted that recommendations serve a narrower use case than originally assumed

Outcome After Pivot

The team implemented semantic search using product embeddings, query expansion for ambiguous searches, and better failure handling when search returned no results. Recommendations were repositioned as "inspiration" rather than primary discovery mechanism. Six months post-pivot:

  • Overall conversion rate: +18% (close to original goal)
  • Search success rate: +34%
  • Search usage: recovered to baseline levels
  • Customer satisfaction scores: +22 points
  • Recommendation CTR: dropped to 4.1% (but total conversions still higher)

Key Lessons

  1. Success on proxies can mask failure on outcomes: Recommendation metrics looked good while site goals suffered
  2. User segmentation matters: Different customer intents need different solutions
  3. Optimization can create blindness: Focusing on recommendations prevented seeing search quality issues
  4. Coverage metrics reveal bias: Only 12% catalog coverage was a red flag ignored during initial deployment

Pattern: This case demonstrates the Engagement Optimization → User Need Fulfillment pivot—moving from maximizing clicks to solving actual customer problems.

GenAI Pivot Case Studies

Case Study 4: Customer Support Chatbot → Retrieval-Augmented Knowledge Base

Initial Framing (Incorrect)

A software company framed support automation as: "We need a chatbot that can answer customer questions to reduce support ticket volume." The team fine-tuned a language model on historical support tickets and documentation.

Loop Analysis - Original Frame

Outcome: Reduce support ticket volume by 30% through automated responses

Atomic Unit: Customer question

Decomposition: - Collect training data (support tickets, documentation, FAQs) - Fine-tune LLM on company-specific language and solutions - Deploy chatbot interface on support portal - Measure deflection rate (questions answered without human)

Alternatives Considered: 1. Rule-based FAQ matching (simple, limited coverage) 2. Retrieval-only system (accurate but not conversational) 3. Fine-tuned model (conversational, domain-aware) 4. RAG (Retrieval-Augmented Generation) (combines retrieval accuracy with generation)

Trade-offs: Chose fine-tuning for end-to-end conversational experience; accepted higher compute cost and maintenance burden

Signals Defined: - Deflection rate (% of questions answered without escalation) - Answer accuracy (human evaluation sample) - User satisfaction (thumbs up/down) - Response latency

What Actually Happened

The chatbot launched with promising early metrics:

  • Deflection rate: 38% (exceeded goal)
  • User satisfaction: 72% thumbs-up rate
  • Response latency: <2 seconds

But after three months, the signals deteriorated:

  • Deflection rate: dropped to 19%
  • User satisfaction: dropped to 51%
  • Support team escalations: increasingly frustrated customers
  • Common complaint: "The bot gave me outdated information"

Diagnosis Through Signals

The fine-tuned model had learned patterns from historical tickets, but couldn't adapt to:

  • Product updates (features changed, but model training was static)
  • Edge cases not in training data (hallucinated solutions)
  • Multi-step troubleshooting (gave generic advice instead of systematic diagnosis)

The real problem emerged from qualitative analysis: customers didn't want conversation—they wanted correct, current, specific answers.

The Pivot

New Outcome: Help customers find accurate solutions faster (not necessarily through conversation)

New Decomposition: - Knowledge retrieval (find relevant documentation) - Answer synthesis (summarize retrieved content) - Confidence assessment (know when to escalate vs. answer) - Content freshness (detect when documentation is outdated)

New Alternatives: 1. Pure retrieval (return documentation links, no generation) 2. RAG with conservative generation (cite sources, admit uncertainty) 3. Structured troubleshooting trees (guide diagnosis step-by-step) 4. Hybrid: retrieval for known issues, human for edge cases

New Trade-offs: Accepted less conversational UX in exchange for accuracy and transparency; prioritized source citation over fluent generation

Outcome After Pivot

The team rebuilt the system using RAG with strict source citation requirements. When confidence was low, the system showed relevant documentation and offered human escalation. For common issues, it displayed step-by-step troubleshooting guides. Six months post-pivot:

  • Deflection rate: 41% (sustained above goal)
  • User satisfaction: 84% (higher than original peak)
  • Escalation quality: support team reported better-informed customers
  • Documentation coverage: system surfaced gaps in docs, leading to content improvements
  • Hallucination rate: <2% (vs. 18% with fine-tuned approach)

Key Lessons

  1. Conversational ≠ helpful: Users cared about accuracy, not chatbot personality
  2. Fine-tuning creates staleness: Static training can't keep up with product evolution
  3. Transparency builds trust: Showing sources increased confidence even when answers were uncertain
  4. Failure modes matter more than success cases: 18% hallucination rate destroyed trust despite 82% accuracy

Pattern: This case demonstrates the Generation → Retrieval pivot—moving from learned responses to grounded information access.

Case Study 5: Document Summarization → Structured Information Extraction

Initial Framing (Incorrect)

A legal services firm framed contract review as: "We need to summarize contracts so lawyers can review them faster." The team built a multi-document summarization system using LLMs to generate executive summaries of legal contracts.

Loop Analysis - Original Frame

Outcome: Reduce contract review time by 50%

Atomic Unit: Individual contract document

Decomposition: - Ingest contract PDFs - Extract text and structure - Generate executive summary highlighting key terms - Display summary to reviewing lawyer

Alternatives Considered: 1. Extractive summarization (select important sentences) 2. Abstractive summarization (generate new summary text) 3. Template-based extraction (fill standard contract fields) 4. Clause classification (identify contract types)

Trade-offs: Chose abstractive summarization for readability and conciseness; accepted risk of missed details

Signals Defined: - Summary quality (lawyer rating 1-5) - Review time reduction (before/after comparison) - Lawyer adoption rate (% using summaries) - Error catch rate (missed critical terms)

What Actually Happened

Initial deployment showed mixed results:

  • Summary quality: 3.8/5.0 average rating
  • Review time: 20% reduction (far below 50% goal)
  • Adoption rate: 62%
  • Error catch rate: lawyers still read full contracts

Qualitative feedback revealed the core issue: lawyers didn't need summaries—they needed specific information quickly.

Typical lawyer workflow: 1. Check liability cap amounts 2. Verify termination clauses 3. Compare payment terms to standard 4. Flag unusual provisions

Summaries provided narrative overview but didn't answer specific questions efficiently.

Diagnosis Through Signals

The 20% time reduction came from lawyers skimming summaries first, but they still needed to search the full document for specific clauses. The summarization framing assumed lawyers wanted to understand contracts generally, but the actual task was finding specific provisions quickly.

The Pivot

New Outcome: Enable lawyers to extract specific contract provisions 10x faster than manual search

New Decomposition: - Clause identification (detect liability, termination, payment, etc.) - Structured extraction (pull specific values: dates, amounts, parties) - Comparison to standards (flag deviations from firm templates) - Search and navigation (jump to relevant sections)

New Alternatives: 1. Structured extraction with validation (extract key fields, verify completeness) 2. Clause-level QA (answer specific questions about contract terms) 3. Comparison dashboard (show standard vs. actual terms side-by-side) 4. Hybrid: extraction for standard clauses, QA for ad-hoc questions

New Trade-offs: Accepted higher upfront effort to define extraction schema; prioritized precision over coverage (extract fewer things correctly vs. everything approximately)

Outcome After Pivot

The team built a structured extraction system that identified 25 critical contract elements (liability caps, termination notice periods, payment terms, etc.) and displayed them in a standardized dashboard. When lawyers needed non-standard information, a QA interface allowed natural language questions grounded in the contract text. Six months post-pivot:

  • Review time: 58% reduction (exceeded goal)
  • Adoption rate: 94%
  • Lawyer satisfaction: 4.6/5.0
  • Error catch rate: improved (structured format made gaps obvious)
  • New use case: automated contract comparison for negotiations

Key Lessons

  1. Summarization solves a different problem than search: Lawyers didn't need less text—they needed specific answers
  2. Structured output > narrative output: Standardized extraction enabled comparison and review workflows
  3. Coverage/precision trade-off: Extracting 25 things correctly was better than 100 things approximately
  4. User workflow determines UX: Understanding how lawyers use information shaped the interface

Pattern: This case demonstrates the Summarization → Extraction pivot—moving from condensed narrative to structured data retrieval.

Case Study 6: Content Generation → Content Assistance

Initial Framing (Incorrect)

A marketing agency framed blog content creation as: "We need AI to generate blog posts automatically to scale content production." The team built a system that generated full blog articles from topic keywords and outlines.

Loop Analysis - Original Frame

Outcome: Increase content production by 5x while maintaining quality

Atomic Unit: Individual blog post

Decomposition: - Input: topic keywords, target audience, desired length - Generation: produce full blog post with introduction, body, conclusion - Review: human editor approves or requests regeneration - Publish: deploy to CMS

Alternatives Considered: 1. Template-based generation (fill structured templates) 2. Full generative model (end-to-end article creation) 3. Outline expansion (generate from detailed human outline) 4. Section-by-section generation (break into chunks)

Trade-offs: Chose outline expansion for balance of quality and automation; accepted that human involvement was still needed

Signals Defined: - Content volume (posts per week) - First-pass approval rate (% of generations published without major edits) - Engagement metrics (traffic, time-on-page, shares) - Editorial time per post

What Actually Happened

The system increased output volume dramatically:

  • Content volume: 6x baseline (exceeded goal)
  • First-pass approval rate: 34%
  • Editorial time: 45 minutes per post (down from 3 hours)

But engagement metrics revealed problems:

  • Traffic: unchanged (more content didn't attract more readers)
  • Time-on-page: decreased 22%
  • Shares: decreased 41%
  • SEO rankings: declining for competitive keywords

Diagnosis Through Signals

The generated content was generic, lacked unique insights, and didn't match the agency's editorial voice. Editors spent most time rewriting for voice and adding original analysis—exactly the high-value work they hoped to automate.

The fundamental misalignment: clients valued the agency for unique perspective and expertise, not just content volume. Generic AI-generated posts competed poorly with existing content and damaged the brand.

The Pivot

New Outcome: Enable writers to produce higher-quality content faster (not more content automatically)

New Decomposition: - Research assistance (gather relevant sources and data) - Outline generation (suggest structure based on topic) - Draft sections (generate first drafts that writers refine) - Editing assistance (improve clarity, fix grammar, suggest rephrasings)

New Alternatives: 1. Research + outline tool (help planning, writers do drafting) 2. Section-level assistance (generate rough drafts for revision) 3. Editing copilot (improve human-written drafts) 4. Hybrid: AI for research and structure, humans for insights and voice

New Trade-offs: Accepted lower content volume in exchange for higher per-post quality; prioritized editorial control over automation

Outcome After Pivot

The team rebuilt the system as a writing assistant that helped with research (finding relevant studies, statistics, examples), generated structural outlines, and offered alternative phrasings during editing. Writers retained full creative control. Six months post-pivot:

  • Content volume: 2.8x baseline (lower than auto-generation but sustainable)
  • Editorial time: 1.5 hours per post (more than auto-generation, less than baseline)
  • Engagement metrics: all exceeded baseline (traffic +18%, time-on-page +12%, shares +31%)
  • Client retention: improved (content quality strengthened brand)
  • SEO rankings: recovered and exceeded pre-AI levels

Key Lessons

  1. More ≠ better: Volume without quality damaged the brand
  2. Augmentation > automation: Helping experts was more valuable than replacing them
  3. Voice and perspective matter: Generic content competed poorly with existing web content
  4. User control is a feature: Writers trusted and adopted tools that enhanced rather than replaced their judgment

Pattern: This case demonstrates the Full Automation → Human-AI Collaboration pivot—moving from end-to-end generation to expert assistance.

Pattern Bridges: Common Pivot Patterns Across Domains

The case studies above span different domains (B2B SaaS, fintech, e-commerce, support, legal, marketing) and different AI approaches (classification, recommendation, chatbot, summarization, generation). Yet they share recurring patterns—structural similarities that make pivots recognizable even in unfamiliar contexts.

These pattern bridges are transferable mental models. When you encounter a new AI problem, recognizing which pattern it matches helps you anticipate likely failure modes and consider alternative framings proactively.

Pattern 1: Prediction → Action (Churn Case)

Structure: Initial framing focuses on forecasting outcomes. Pivot shifts to enabling interventions.

Why It Recurs: - Prediction feels like the ML task (it's what models do) - But business value comes from acting on predictions, not just knowing them - Action requires lead time, specificity, and intervention options

Other Examples: - Equipment failure prediction → Maintenance scheduling optimization - Hospital readmission prediction → Discharge planning improvement - Employee attrition prediction → Retention program targeting

Recognition Signals: - Stakeholders ignore predictions they "already knew" - High prediction accuracy but no outcome improvement - Users ask "what should I do about this?" after seeing predictions

Framing Questions: - What action will stakeholders take with this prediction? - How much lead time do they need to intervene? - What interventions are available and effective? - Is the constraint prediction quality or intervention capacity?

Pattern 2: Classification → Scoring (Fraud Case)

Structure: Initial framing uses binary decisions (approve/reject, fraud/legitimate). Pivot shifts to continuous risk scores with proportional responses.

Why It Recurs: - Classification is simpler to implement (binary threshold) - But real-world costs are non-uniform (false positives on VIP customers hurt more) - Proportional responses (more friction for higher risk) balance trade-offs better

Other Examples: - Loan approval (binary) → Credit risk scoring (tiered terms) - Content moderation (remove/allow) → Risk-based review queues (auto/review/escalate) - Applicant screening (hire/reject) → Interview prioritization (tiers)

Recognition Signals: - Low false positive rate but high impact from false positives - Uniform treatment causing outsized harm to important segments - Calls for "manual review" on edge cases

Framing Questions: - Are all errors equally costly? - Can we adjust response intensity to risk level? - What intermediate actions exist between accept and reject? - Which segments deserve different treatment?

Pattern 3: Engagement → Need Fulfillment (Recommendation Case)

Structure: Initial framing optimizes for interaction metrics (clicks, time-on-site). Pivot shifts to solving user goals (find what they need).

Why It Recurs: - Engagement metrics are easy to measure and optimize - But engagement can be shallow (clicking without value) - User satisfaction comes from accomplishing goals, not just interacting

Other Examples: - Video recommendations (watch time) → Content satisfaction (completion, ratings) - News feed (scroll depth) → Information value (learned something useful) - Search results (CTR) → Query success (found answer)

Recognition Signals: - Metrics improving but user satisfaction declining - Users "engaging" more but accomplishing less - Surveys show frustration despite high usage

Framing Questions: - What is the user trying to accomplish? - Are engagement metrics proxies for that goal or distractions? - What would "success" look like from the user's perspective? - Are we optimizing the platform's goals or the user's goals?

Pattern 4: Generation → Retrieval (Chatbot Case)

Structure: Initial framing uses learned generation (fine-tuning, training). Pivot shifts to grounded retrieval (RAG, search, citations).

Why It Recurs: - Generation feels more "AI-like" (creative, conversational) - But generation risks hallucination and staleness - Many use cases value accuracy over fluency

Other Examples: - Code generation → Code search with examples - Report writing → Template filling with data retrieval - Question answering → Source citation with excerpts

Recognition Signals: - Hallucination complaints (users catch factual errors) - Requests for "where did this come from?" - Accuracy degrading over time (model staleness)

Framing Questions: - Does the user need original content or access to existing knowledge? - How important is factual accuracy vs. conversational quality? - Can we ground responses in verifiable sources? - How will we keep information current?

Structure: Initial framing condenses information (summaries, abstracts). Pivot shifts to structured extraction (specific fields, searchable data).

Why It Recurs: - Summarization seems helpful (less to read) - But users often want specific information, not shorter narrative - Structured extraction enables search, comparison, and automation

Other Examples: - Meeting summaries → Action item extraction - Research paper summaries → Method and results extraction - Financial document summaries → Numerical data extraction

Recognition Signals: - Users still search full documents after reading summaries - Requests for "Can you pull out X?" for specific fields - Need to compare across documents (easier with structured data)

Framing Questions: - What specific questions do users have about these documents? - Do they need narrative understanding or specific data points? - Will they compare multiple documents (suggesting structured output)? - What format makes the information most actionable?

Pattern 6: Full Automation → Human-AI Collaboration (Content Case)

Structure: Initial framing replaces human work entirely. Pivot shifts to augmenting human expertise.

Why It Recurs: - Automation promises maximum efficiency (remove human bottleneck) - But expertise, judgment, and creativity are hard to automate - Augmentation leverages AI for drudgework, human for high-value decisions

Other Examples: - Automated diagnosis → Decision support for doctors - Automated code review → Suggestion tools for developers - Automated translation → Translation assistance for professional translators

Recognition Signals: - Quality issues (AI alone doesn't match expert output) - Experts reject or ignore the automated system - Value comes from unique perspective AI can't replicate

Framing Questions: - What parts of the workflow are tedious vs. requiring expertise? - Can AI handle routine cases while escalating edge cases? - Do humans trust fully automated outputs? - Where does irreplaceable human judgment matter?

Pattern Recognition in Practice

When encountering a new AI problem, ask:

  1. Does it match a known pattern? (Use the six above as starting templates)
  2. What failure mode does this pattern predict? (Each has characteristic warning signs)
  3. What alternative framing would avoid that failure? (Each pattern has an alternative structure)
  4. What signals would tell you which framing is correct? (Design experiments to test)

This pattern-based thinking accelerates problem framing by providing starting hypotheses rather than blank-slate exploration. You're not reinventing analysis for each new problem—you're applying structured frameworks informed by past failures.

The Monday Morning Checklist

You walk into work Monday morning and someone says: "We need an AI solution for [business problem]." How do you respond?

This checklist operationalizes The Loop framework into a repeatable routine for any AI problem framing scenario. Use it as a forcing function to slow down and ask upstream questions before committing to implementation.

Phase 1: Outcome Clarity (5 minutes)

Before you discuss solutions, establish the destination.

  • [ ] What is the measurable outcome we're pursuing?
  • Not: "improve customer experience"

  • Yes: "reduce support ticket volume by 25% while maintaining satisfaction >4.0/5.0"

  • [ ] Who defines success for this outcome?

  • Is it the person asking for AI, or a different stakeholder?

  • Do they agree on what success looks like?

  • [ ] What is the timeline and acceptable cost?

  • When do we need to show results?

  • What resources (people, compute, data) are available?

  • [ ] What happens if we don't pursue this?

  • Is this urgent or aspirational?
  • What's the opportunity cost of not solving it?

Red flags: - Multiple conflicting goals ("improve quality and reduce cost and ship faster") - Vague outcomes that can't be measured - Solution already decided ("we need a chatbot" instead of "we need better support")

If red flags appear: Stop and clarify outcome with stakeholders before proceeding.

Phase 2: Problem Deconstruction (15 minutes)

Before you explore solutions, understand the problem structure.

  • [ ] What is the atomic unit of this problem?
  • What is the smallest independently meaningful component?

  • Will we measure success at that granularity or aggregate?

  • [ ] What are we assuming about the problem?

  • What would have to be true for this framing to make sense?

  • Which assumptions can we validate quickly?

  • [ ] Who are the stakeholders and users?

  • Who will use the AI output?
  • Who will be affected by it?

  • Whose behavior needs to change for success?

  • [ ] What constraints exist?

  • Data availability (do we have labels? volume? quality?)
  • Latency requirements (real-time? batch?)
  • Interpretability needs (black box acceptable? need explanations?)

  • Fairness/safety requirements (regulated domain? ethical risks?)

  • [ ] What is the current workflow?

  • How is this problem solved today?
  • What's broken about the current approach?
  • What works well that we shouldn't break?

Red flags: - No clear atomic unit (problem is too abstract) - Unvalidated assumptions presented as facts - Key stakeholders not identified or consulted - No understanding of current state (assuming AI is obviously better)

If red flags appear: Do discovery work (interviews, observation, data exploration) before proceeding.

Phase 3: Alternatives Menu (20 minutes)

Before you commit to a solution, explore the possibility space.

  • [ ] What are 5+ different ways to approach this?
  • Include non-ML alternatives (rules, heuristics, process changes)
  • Include different ML archetypes (classification, ranking, generation, etc.)

  • Include hybrid approaches (ML + rules, human-in-loop)

  • [ ] What is the simplest viable approach?

  • What's the baseline (current state or naive solution)?
  • What's the minimal ML solution that could work?

  • What would a rule-based system look like?

  • [ ] What is the most sophisticated approach?

  • What would we do with unlimited resources?

  • What emerging research could apply here?

  • [ ] Do we even need AI for this?

  • Can we solve it with better data, processes, or UX?
  • Is the constraint actually technical?

Red flags: - Only one solution considered - Defaulting to whatever is trendy (LLMs, deep learning) without justification - Skipping simple baselines - "AI" is the goal rather than the outcome

If red flags appear: Force divergent thinking—research different approaches, consult experts in other domains, look for analogous solved problems.

Phase 4: Trade-off Analysis (20 minutes)

Before you choose, understand what you're giving up.

For each viable alternative, assess:

  • [ ] Development cost
  • Data requirements (collection, labeling, cleaning)
  • Engineering effort (person-months to build)

  • Compute cost (training and inference)

  • [ ] Operational characteristics

  • Latency (real-time? seconds? minutes?)
  • Accuracy/quality (good enough for outcome?)

  • Reliability (failure modes? edge cases?)

  • [ ] Maintenance burden

  • How often must it be retrained?
  • How much monitoring does it need?

  • What breaks when data distribution shifts?

  • [ ] Organizational fit

  • Do we have expertise to build and maintain this?
  • Does it align with existing infrastructure?

  • Will stakeholders trust and adopt it?

  • [ ] Risk profile

  • What's the worst-case failure mode?
  • Are there fairness, safety, or compliance risks?
  • Can we mitigate or is risk inherent?

Create a comparison matrix: Rows = alternatives, Columns = criteria, Cells = qualitative assessment

Red flags: - Choosing solely on accuracy without considering cost/latency/maintenance - Ignoring organizational constraints (building what the team can't maintain) - Underestimating ongoing costs (training once vs. continuous retraining)

If red flags appear: Build trade-off matrix explicitly and review with stakeholders—make decisions transparent.

Phase 5: Signal Design (15 minutes)

Before you build, define how you'll know if you're on the right track.

  • [ ] What are the success metrics?
  • Model metrics (accuracy, precision, recall, etc.)
  • Business metrics (conversion, retention, cost reduction)

  • User metrics (satisfaction, adoption, engagement)

  • [ ] What are the leading indicators?

  • What can we measure early that predicts long-term success?

  • What would tell us we're headed in the wrong direction?

  • [ ] What are the kill signals?

  • What threshold would make us stop or pivot?

  • What failure mode would invalidate the approach?

  • [ ] How will we instrument this?

  • What data do we need to collect to measure these signals?

  • Can we measure them in a pilot before full deployment?

  • [ ] What is the evaluation timeline?

  • When will we check signals? (weekly? monthly?)
  • How long until we expect meaningful signal? (avoid premature judgment)

Red flags: - Only tracking model metrics (ignoring business outcomes) - No plan for measuring impact (assuming deployment = success) - No defined failure criteria (never considering stopping)

If red flags appear: Design instrumentation and evaluation plan before building—treat measurement as part of the product.

Phase 6: Commit or Defer (5 minutes)

Final check before proceeding.

  • [ ] Can we clearly answer all previous checklist items?
  • If not, what gaps remain?

  • Can we fill gaps quickly or do we need more discovery?

  • [ ] Do stakeholders agree on the framing?

  • Have we aligned on outcomes, constraints, and approach?

  • Are there unresolved conflicts?

  • [ ] Do we have the resources to execute?

  • Data, talent, compute, time

  • Or do we need to acquire them first?

  • [ ] Is this the right priority?

  • Given other projects, is this the best use of resources?
  • What are we not doing if we pursue this?

Three possible outcomes:

  1. Commit: Framing is clear, trade-offs understood, resources available → build it
  2. Defer: Good framing but not the right time/priority → revisit later
  3. Reframe: Gaps in understanding, misalignment, or better alternatives identified → return to earlier phases

Avoid: Proceeding when framing is unclear or stakeholders are misaligned. The cost of building the wrong thing exceeds the cost of more discovery.

Checklist in Practice: Example Walkthrough

Scenario: "We need a recommendation system for our learning platform to personalize content."

Phase 1: Outcome Clarity - Measurable outcome: Increase course completion rate from 35% to 50% in 6 months - Success owner: VP of Product (measured via analytics dashboard) - Timeline: Pilot in 3 months, full rollout in 6 months - Cost: 2 engineers for 3 months, $10k compute budget - Opportunity cost: Delaying mobile app improvements

Phase 2: Problem Deconstruction - Atomic unit: Individual learner session (not learner or course) - Assumptions: (1) Personalization improves completion, (2) We have signal for good recommendations, (3) Learners will follow recommendations - Stakeholders: Learners (need relevant content), instructors (want completions), ops team (maintains system) - Constraints: Must work with cold-start (new users), <500ms latency, explainable (learners want to know "why this?") - Current workflow: Manual course browsing, search, instructor recommendations

Phase 3: Alternatives Menu 1. Collaborative filtering (user-user similarity) 2. Content-based filtering (course similarity) 3. Sequence modeling (what comes after course X?) 4. Popularity + basic rules (trending + match user goals) 5. Instructor-curated paths (no ML, just better UX)

Phase 4: Trade-off Analysis

Alternative Dev Cost Latency Accuracy Maintenance Interpretability
Collaborative Medium Low Medium Medium Low
Content-based Low Low Medium Low High
Sequence model High Medium High High Low
Popularity Low Low Low Low High
Curated paths Low N/A Medium Low High

Decision: Start with content-based filtering (low cost, explainable, adequate accuracy) with plan to add collaborative signals if successful.

Phase 5: Signal Design - Success metrics: Completion rate (business), CTR on recommendations (proxy), user satisfaction survey - Leading indicators: Recommendation CTR (weekly), time-to-first-completion (monthly) - Kill signals: CTR <5%, completion rate unchanged after 3 months, satisfaction <3.5/5.0 - Instrumentation: Log all recommendation impressions and clicks, track completion by source (organic vs. recommended) - Timeline: Weekly metric reviews, pivot decision at 3 months

Phase 6: Commit - All questions answered clearly - Stakeholders aligned (Product, Engineering, Instructors) - Resources available (engineers allocated, data exists) - High priority (completion rate is OKR)

Outcome: Commit to content-based recommendation pilot with defined success criteria and evaluation timeline.

Reflection Questions

These questions guide your synthesis of the chapter concepts:

  1. Case Study Analysis: Choose one case study from this chapter. What was the earliest signal that the original framing was wrong? Could that signal have been detected before deployment?

  2. Pattern Recognition: Think of an AI project you've worked on or observed. Which pivot pattern (if any) does it match? If it succeeded, did it avoid the pattern's typical failure mode? If it failed, would the alternative framing have helped?

  3. Monday Morning Application: Take a current or hypothetical AI problem in your domain. Walk through the Monday Morning Checklist. Where do you encounter gaps in your understanding? What questions can't you answer yet?

  4. Trade-off Preferences: Across the case studies, different teams made different trade-off decisions (accuracy vs. latency, automation vs. control, engagement vs. satisfaction). What trade-off patterns do you notice in your organization? Are they explicit or implicit?

  5. Signal Blindness: In the fraud detection case, the team tracked false positive rate but missed false positive impact. What metrics do you track that might suffer from similar abstraction? What signals are you not measuring?

  6. Pivot Resistance: The content generation case showed reluctance to abandon full automation even when augmentation worked better. What psychological or organizational factors make pivoting difficult? How would you structure projects to make pivots easier?

  7. Pattern Bridges Transfer: Take Pattern 2 (Classification → Scoring) and apply it to a domain not covered in the case studies. What problem might benefit from this pivot? What would the continuous scoring enable that binary classification prevents?

  8. Checklist Customization: The Monday Morning Checklist provides a generic framework. What domain-specific questions would you add for your industry (healthcare, finance, education, etc.)? What constraints or stakeholders are always relevant in your context?

Final Portfolio: Complete Problem Framing Portfolio

This capstone assignment synthesizes all course concepts into a comprehensive problem framing portfolio demonstrating your ability to work systematically from ambiguity to actionable implementation plans.

Portfolio Requirements

Your portfolio must include:

1. Problem Selection and Context (500 words)

Select a real or realistic AI opportunity in a domain you understand well. Provide:

  • Business context: What organization, what challenge, what constraints exist
  • Current state: How is this problem addressed today (if at all)
  • Stakeholders: Who cares about solving this and why
  • Success criteria: What would "solved" look like

Quality indicators: - Sufficient complexity (not trivial, not unsolvable) - Clear business value (not AI for AI's sake) - Real constraints (not unconstrained thought experiment)

2. Complete Loop Analysis (1500 words)

Apply all five Loop phases systematically:

Outcome Definition: - Specific, measurable outcome with timeline - Clarification of what success means quantitatively - Identification of whose definition of success matters

Problem Deconstruction: - Atomic unit identification with justification - Explicit statement of assumptions with validation plan - Stakeholder mapping and constraint catalog - Current workflow documentation

Alternatives Menu: - Minimum 5 distinct alternatives spanning different archetypes - Include at least one non-ML alternative - Brief description of each approach (2-3 sentences) - Explanation of why each is viable

Trade-off Analysis: - Comparison matrix with minimum 5 criteria (cost, latency, accuracy, maintenance, organizational fit) - Qualitative or quantitative assessment for each alternative - Clear recommendation with justification - Acknowledgment of what you're giving up with your choice

Signal Framework: - 3+ success metrics across model, business, and user dimensions - Leading indicators that predict outcome achievement - Kill signals with explicit thresholds - Instrumentation plan and evaluation timeline

Quality indicators: - Each phase builds logically on previous phases - Decisions are evidence-based and transparent - Alternatives are genuinely different (not minor variations) - Trade-offs are honest (acknowledging weaknesses of chosen approach) - Signals are measurable and actionable

3. Anticipated Failure Modes and Pivot Plan (500 words)

Demonstrate proactive thinking about what could go wrong:

  • Most likely failure mode: What's the highest-probability way this framing could be wrong?
  • Pivot pattern match: Which pattern from this chapter (if any) matches your problem? What does that predict about failure modes?
  • Alternative framing: If your initial framing fails, what alternative framing would you pivot to?
  • Signal timeline: When would you expect to see signals of success or failure? What's your decision point for persist/pivot/stop?

Quality indicators: - Realistic failure modes (not strawman scenarios) - Clear connection to course patterns - Thoughtful alternative framing (not just "try harder") - Defined decision criteria (avoiding indefinite persistence)

4. Stakeholder Communication Plan (300 words)

How would you present this framing to secure buy-in?

  • Executive summary: 3-4 sentences capturing the opportunity, approach, and expected outcome
  • Key trade-offs: What are you optimizing for and what are you accepting?
  • Risk mitigation: What's the biggest risk and how are you managing it?
  • Decision points: When will you update stakeholders on progress and what criteria determine success?

Quality indicators: - Accessible to non-technical audience - Honest about limitations and risks - Clear ask (resources, timeline, decision authority) - Realistic expectations

Submission Format

  • Length: 2800-3300 words total (excluding references)
  • Structure: Use the four sections above as headings
  • Visuals: Include at least one diagram (trade-off matrix, workflow diagram, or alternatives comparison)
  • Citations: Reference course concepts explicitly (e.g., "Using The Loop framework from Chapter 3...")

Evaluation Rubric

Your portfolio will be evaluated on:

Systematic Application (40%)

  • Exemplary (36-40 points): Every Loop phase executed thoroughly with clear reasoning; decisions build logically; framework applied correctly throughout
  • Proficient (32-35 points): Loop phases mostly complete with minor gaps; generally sound reasoning; correct framework application
  • Developing (28-31 points): Some Loop phases incomplete or superficial; occasional logical gaps; framework applied inconsistently
  • Beginning (0-27 points): Missing critical Loop phases; unclear reasoning; framework misapplied or ignored

Depth of Analysis (30%)

  • Exemplary (27-30 points): Explores second-order effects, hidden assumptions, edge cases; considers multiple perspectives; demonstrates sophisticated understanding
  • Proficient (24-26 points): Addresses main considerations; identifies key assumptions; shows solid understanding
  • Developing (21-23 points): Surface-level analysis; misses important considerations; limited perspective-taking
  • Beginning (0-20 points): Minimal analysis; obvious gaps; single-perspective thinking

Evidence-Based Reasoning (20%)

  • Exemplary (18-20 points): Claims grounded in data, examples, or case study patterns; acknowledges uncertainty appropriately; transparent about assumptions
  • Proficient (16-17 points): Most claims supported; reasonable uncertainty handling; generally transparent
  • Developing (14-15 points): Some unsupported claims; treats assumptions as facts; limited transparency
  • Beginning (0-13 points): Mostly unsupported assertions; ignores uncertainty; opaque reasoning

Communication Clarity (10%)

  • Exemplary (9-10 points): Clear structure; compelling narrative; appropriate visuals; accessible to diverse audiences
  • Proficient (8 points): Organized; mostly clear; adequate visuals; generally accessible
  • Developing (7 points): Somewhat disorganized; clarity issues; minimal visuals; assumes too much context
  • Beginning (0-6 points): Disorganized; unclear writing; no visuals; inaccessible

Total: 100 points

Passing threshold: 70 points (demonstrates competence in problem framing for professional practice)

While not required for portfolio completion, peer feedback accelerates learning:

How Peer Review Works

  1. Pairing: After submitting your portfolio, you'll be matched with 2-3 peers
  2. Review Period: 1 week to read peer portfolios and provide written feedback
  3. Feedback Format: Structured questions guide your review (provided separately)
  4. Discussion: Optional live discussion session to deepen understanding

What to Focus On

When reviewing peer work:

  • Completeness: Are all Loop phases addressed?
  • Alternatives: Are alternatives genuinely different or minor variations?
  • Trade-offs: Are trade-offs realistic or overly optimistic?
  • Signals: Are signals measurable and actionable or vague aspirations?
  • Failure modes: Has the person anticipated realistic failure modes?

How to Give Useful Feedback

  • Be specific: "The trade-off analysis ignores maintenance burden" > "Trade-offs need work"
  • Be constructive: Suggest alternatives, don't just criticize
  • Be curious: Ask questions to understand reasoning rather than assuming it's wrong
  • Be humble: Your framing might have gaps too—peer review is collaborative learning

Benefits of Peer Review

  • Pattern recognition: See how others frame similar problems differently
  • Blind spot detection: Peers catch assumptions you didn't notice
  • Alternative approaches: Learn solution archetypes outside your expertise
  • Critical thinking: Evaluating others sharpens your self-evaluation

Course Conclusion: From Frameworks to Practice

You've completed a graduate-level course in AI problem framing—a skill that will compound in value throughout your career.

What You've Learned

Chapter 1 taught you to recognize the solution-first trap, embrace ambiguity, and activate System 2 thinking at critical moments.

Chapter 2 built your mental model of the AI solution space—13 archetypes spanning classic ML and GenAI, with trade-offs and application contexts.

Chapter 3 gave you The Loop, a systematic framework for moving from vague objectives to actionable plans through outcome clarity, deconstruction, alternatives, trade-offs, and signals.

Chapter 4 developed your ability to read signals from live systems—distinguishing meaningful patterns from noise, identifying leading indicators, and designing instrumentation.

Chapter 5 taught you when and how to pivot based on evidence, avoiding both premature abandonment and sunk-cost persistence.

Chapter 6 synthesized these concepts through case studies, pattern recognition, and the Monday Morning Checklist—moving from theory to professional practice.

How This Changes Your Work

Before this course, you might have approached AI problems reactively: someone asks for a solution, you build it, you deploy it, you hope it works.

After this course, you approach AI problems systematically:

  • You ask upstream questions before committing to solutions
  • You explore diverse alternatives rather than defaulting to the obvious approach
  • You make trade-offs explicit rather than discovering them post-deployment
  • You design signals to detect misalignment early
  • You pivot based on evidence rather than intuition or stubbornness

This systematic approach has three benefits:

  1. Fewer failures: You catch misaligned framings before investing months of engineering
  2. Faster pivots: You recognize failure modes early and adjust course deliberately
  3. Better stakeholder relationships: You communicate trade-offs transparently and manage expectations realistically

What to Learn Next

This course focused on problem framing—the upstream decisions that determine what to build. To translate framings into deployed systems, consider deepening expertise in:

For Classic ML Practitioners: - MLOps and deployment: Moving models from notebooks to production reliably - Causal inference: Moving beyond correlation to understand intervention effects - Fairness and interpretability: Ensuring models work equitably and transparently - Data engineering: Building pipelines that provide high-quality training data at scale

For GenAI Practitioners: - Prompt engineering and evaluation: Systematic approaches to optimizing LLM behavior - RAG architectures: Building retrieval-augmented systems that ground generation - Fine-tuning and alignment: Adapting models to domain-specific needs safely - LLM evaluation frameworks: Measuring quality beyond toy examples

For Everyone: - Organizational change: Implementing AI in real organizations with real politics - Product management: Translating technical capabilities into user value - Ethics and policy: Navigating the societal implications of AI systems - Domain expertise: Deepening knowledge in your application area (healthcare, finance, education, etc.)

Recommended Next Courses: - Production Machine Learning Systems (focuses on deployment and operations) - Causal Inference for Data Science (teaches intervention analysis) - Designing Human-AI Collaboration (explores organizational and UX challenges)

Continuing the Practice

Problem framing is a skill that improves with deliberate practice. To continue developing:

  1. Use the Monday Morning Checklist: Apply it to every new AI problem you encounter
  2. Study failures: When AI projects fail (yours or others'), diagnose the framing gaps
  3. Build a pattern library: Document pivot patterns you encounter in your domain
  4. Teach others: Explaining The Loop to colleagues deepens your understanding
  5. Stay current: AI capabilities evolve—new solution archetypes emerge that expand the alternatives menu

A Final Note on Humility

The Loop framework and Monday Morning Checklist are tools, not magic. They help you think systematically, but they don't guarantee success. You will still frame problems incorrectly. You will still miss signals. You will still build things that don't work.

The difference is that you'll recognize these failures earlier, diagnose them more accurately, and pivot more deliberately. You'll waste fewer resources on doomed approaches and more quickly find framings that deliver value.

Problem framing is not about perfection. It's about developing the habit of asking "Are we solving the right problem?" before asking "How do we solve it?" That habit—practiced consistently across projects and years—is what separates effective AI practitioners from those who build impressive solutions to irrelevant problems.

You now have the frameworks, patterns, and checklists to practice that habit. The rest is up to you.

Welcome to the upstream work of AI. The work that happens before the code, before the models, before the deployment. The work that determines whether everything that follows creates value or waste.

Go build something worth building.

Additional Resources

Case Study Sources

The case studies in this chapter are synthesized from real projects, with details anonymized and composited to protect confidentiality. For deeper investigation of similar patterns:

  • Churn prediction: "Why Customer Analytics Fails: And How to Fix It" (Harvard Business Review)
  • Fraud detection: "Machine Learning for Fraud Detection: Lessons from the Field" (Stripe Engineering Blog)
  • Recommendations: "Exploration vs. Exploitation in Recommender Systems" (RecSys Conference Papers)
  • Chatbots: "The ROI of Conversational AI" (Gartner Research)
  • Summarization: "What Makes a Good Summary? Perspectives from Legal Tech" (LegalTech News)

Pattern Recognition

  • Christensen, C. (1997). The Innovator's Dilemma - discusses pattern recognition across industry pivots
  • Gabriel, R. (2019). Patterns of Software - explores patterns in software architecture that parallel problem framing

Frameworks and Toolkits

  • Project retrospective templates: [add link to course resources]
  • Trade-off analysis worksheets: [add link to course resources]
  • Signal design canvas: [add link to course resources]

Congratulations on completing AI Problem Framing for AI Practitioners. You're now equipped to make better upstream decisions about AI projects. Go practice.