Rebuilding an AI Decision Tool with Constraint-Driven Arbitration

The author rebuilt their AI decision tool from a simple summarizer into a more robust, constraint-driven arbitrator. The key changes were separating the process into distinct stages of constraint extraction, research, independent advocacy, and final arbitration.

Details

The author realized that real decision-making involves three distinct cognitive operations: defining success criteria, building cases for options, and evaluating the options against the criteria. The original tool tried to do all of this in a single LLM call, leading to vague and inconsistent outputs. The redesigned pipeline separates these steps. First, the 'Constraint Extraction' stage defines a normalized framework of hard constraints, soft constraints, decision criteria, risk tolerance, and unknown critical inputs. This provides a shared reference point for all downstream stages. Next, the 'Research' stage uses a custom search API to gather grounded evidence, tagging each claim with its source type and evidence strength. This avoids the original tool's problem of citing unverified statistics. The 'Independent Advocates' stage then builds the strongest case for each option, explicitly referencing the constraint framework. Finally, the 'Arbitrator' stage evaluates the options against the defined criteria to reach a decision. This architectural change, with each stage's output becoming a hard input to the next, eliminated the hand-waving and inconsistency of the original tool.