Writing / Measurement & applied AI
An AI analyst can answer the wrong KPI perfectly
How to define the question before a fluent answer turns into a KPI.
“What was our conversion rate last month?” sounds like a straightforward question. An AI analyst can return a number, a chart and a calm explanation before anyone has agreed what “conversion” means.
The query may run successfully. The source may be real. The answer may still use the wrong metric, population or reporting window. That is the gap this article addresses: a fluent answer is not the same thing as a defined KPI.
Before asking an AI analyst for a number, write a short metric and question contract. Use it to define the decision, the measure, its boundaries and the evidence the answer must show. Then give the contract a governed home and a clear stop condition.
Start with the decision
A metric is useful because it helps someone choose an action. “Conversion rate” might be used to decide whether to change a landing page, move budget between channels or investigate a sales handoff. Those decisions can need different populations, time windows and levels of attribution.
Write the decision in one sentence before you choose a field. For example: “Decide whether the paid-social landing page needs a message or tracking change.” This makes the next questions visible. Which visitors belong in the comparison? Which completed action counts? What would be different if the answer moved?
Define the metric before naming the tool
A label is a shortcut for a definition. Record what is counted, the unit, the inclusions and the exclusions. A rate needs both a numerator and a denominator, plus the relationship between them. “Leads” could mean submitted forms, qualified records in a CRM or opportunities accepted by sales. They are related words with different decisions attached.
- Metric and unit
- Name the event or entity being counted and the unit returned: people, sessions, orders, dollars or a percentage. State the numerator and denominator when it is a rate.
- Inclusions and exclusions
- Record which statuses, test records, duplicates, cancellations, internal traffic or other cases are included. An exclusion is part of the meaning, not a footnote.
- Semantic layer
- A semantic layer is the governed description of fields, measures and relationships that a tool can use when translating a question. It can carry meaning into the query, but it cannot decide whether the business definition is the right one.
Google describes this path in its Looker documentation: the semantic model supplies definitions, joins, filters, aggregations and permissions, while the generated output still needs validation. Its guidance also calls out ambiguous labels, similar field names, unclear definitions and time zones as reasons a system can select the wrong field or aggregation[1][2].
Make population and grain explicit
Population is who or what can be counted. Grain is the level of one row or one observation, such as one person, one order or one session. A question about people can change when the table contains several events per person. A join can multiply rows before a measure is summed.
The scope behind the number
Name the boundary before the calculation.
Every answer should make its population, grain and route through the data inspectable.
01 / Population
Who is eligible?
State the records that can enter the measure, such as completed enquiry forms from new website visitors. Say what is left out and why.
02 / Grain
What is counted once?
Choose the unit before aggregating. If a person can submit several forms, decide whether the measure counts submissions or distinct people.
03 / Joins and filters
Which records are connected?
List the joins, filters and segments. Check that a one-to-many relationship does not duplicate the rows used by the measure.
04 / Attribution and time
Which path gets the credit?
Record the attribution boundary, reporting window, timezone, comparison period and freshness requirement. “Last month” is incomplete without those choices.
Databricks documents the same modelling risk in its metric-view guidance: measures depend on fields, filters and joins, and an incorrect many-to-one assumption can create fan-out and incorrect results[3]. The point is not that a platform is unsafe. The point is that the relationship belongs in the definition you review.
Illustrative structure, without a result: “How many leads did paid social generate last month?” becomes “How many completed, non-duplicate enquiry forms from the paid-social landing-page sessions were recorded in the CRM during the agreed local-time window, using the approved first-touch attribution rule?” If the CRM and analytics records disagree, the answer pauses for review.
Write the metric and question contract
The contract is a compact record that travels with the question. It is Meeno’s proposed synthesis, rather than a vendor standard. Keep the fields together so a reviewer can see what the number is meant to support and what would make the answer unsafe to accept.
- 01 / DecisionWhat will this number change?
- Name the decision, audience and action the result is meant to support.
- 02 / DefinitionWhat exactly counts?
- Give the metric, unit, numerator, denominator, inclusions and exclusions in plain language.
- 03 / Population and grainWho or what is one observation?
- State the eligible population and whether the count is by person, event, order, session or another unit.
- 04 / BoundaryWhich joins and filters apply?
- List joins, segments, suppression rules and the attribution boundary. Call out assumptions about one-to-many relationships.
- 05 / TimeWhen is it true?
- Record the start and end, timezone, comparison period and how fresh the source must be.
- 06 / Source and ownerWhere does it come from?
- Name the source of truth, access boundary, definition owner and the person who accepts a change.
- 07 / EvidenceWhat must the answer show?
- Specify the fields, filters, calculation, source rows or links, freshness and explanation a reviewer needs.
- 08 / Stop conditionWhen must it ask, stop or escalate?
- Require clarification for a missing definition, a conflicting source, stale data, an unexpected join or a consequential decision.
Use “unknown” deliberately. An absent timezone, owner or exclusion is a reason to ask a question, not permission to guess. A blank field and a confirmed zero are different states and should remain different in the contract.
Read the infographic as text
Before the prompt, define the question. A valid query can still answer the wrong metric.
- DecisionWhat will this number change?
- Name the decision, audience and action the result is meant to support.
- DefinitionWhat exactly counts?
- Give the metric, unit, numerator, denominator, inclusions and exclusions.
- ScopeWho or what is one observation?
- State the eligible population and grain: person, event, order, session or another unit.
- Boundary and timeWhich path and period apply?
- List joins, filters, attribution, timezone, comparison period and freshness.
- Evidence and ownerWhere does it come from?
- Name the source of truth, accountable owner and evidence the answer must return.
Valid SQL is not proof of valid meaning. Clarify, stop or assign human review when a field is missing, conflicts or is stale.
The metric and question contract is Meeno Zen’s proposed synthesis, based on the article’s documented sources. It is a review instrument, not a guarantee of accuracy.
Meeno Zen. Based on “An AI analyst can answer the wrong KPI perfectly”. Checked 11 September 2026.
Give the contract somewhere to live
A prompt can carry the contract for one conversation. A governed model, metric catalogue or report specification keeps it available for the next one. Store the definition beside the fields and relationships it describes, with an owner and a review date.
Looker uses LookML as the grounding source for field descriptions, join logic, filters, permissions and verified queries. Power BI Copilot can expose the measures, columns, filters, DAX and table used for an answer, while Microsoft also documents cases where the system chooses a similarly named field or date and warns that responses are not guaranteed to be correct[1][4][5].
Inspect the answer, not just the query
Review the complete path from the question to the decision. A query that executes without an error answers only whether the system produced executable instructions. It does not prove that the instructions match the contract.
Check the meaning
Read the metric definition, unit, numerator, denominator and exclusions beside the answer. Ask whether the words in the question map to the intended measure.
Check the selection
Inspect the tables, fields, joins, filters, segments, date field, timezone and attribution boundary. Look for a similarly named field or a population that changed at the join.
Check the calculation
Read the generated SQL, DAX or equivalent logic and compare its aggregation with the stated grain. Run a small sanity check against a known slice or an independently reconciled total.
Check the evidence and decision fit
Record the source, freshness, permissions, caveats and the person who reviewed it. A cited source can still be out of date, and a correct result for one decision can be irrelevant to another.
Microsoft’s guidance treats the answer, the fields used and the underlying data as separate things to inspect[5]. In a qualitative study of 22 professional analysts, researchers also found that AI-assisted analyses could look reasonable while containing interpretation or transformation errors. The study is exploratory, but it supports checking intermediate outputs and provenance rather than trusting presentation alone[6].
Make stopping part of the design
An AI analyst needs a useful way to say that the contract is incomplete. A refusal without a next question leaves the team stuck. A confident guess makes the gap invisible.
The review boundary
Clarify, stop or name the reviewer.
Route the answer according to what is missing and what the decision could change.
Missing meaning
Ask a clarifying question.
If the metric, population, grain or time window is absent, show the competing definitions and ask the owner to choose.
Conflicting evidence
Hold the result.
If two sources disagree or a join changes the population, return the conflict and the records needed to reconcile it.
Stale or incomplete data
Label the limit.
State the last refresh, missing periods and effect on the decision. Do not fill an unknown duration or value with an estimate that looks confirmed.
Consequential use
Assign human review.
For a budget, customer, employment or compliance decision, name the domain reviewer who accepts the definition, evidence and result.
NIST’s AI Risk Management Framework recommends context-specific, repeatable testing, documented methods and uncertainty, domain expertise and independent review before deployment and during operation[7]. The recommendation is proportionate: the more a number can change, the more visible its assumptions and review path should be.
Keep the contract alive
Definitions drift when fields, pipelines or attribution rules change. Put a review date and version beside the contract, and keep the old definition attached to historical reports.
For the next AI analytics question, start with the decision and complete the eight fields before opening the prompt box. Ask for the answer, the selected fields and filters, the calculation, the source and the freshness in the same response. If a field is unknown or two records conflict, make that the next action.
A useful AI analyst is one that can explain the number, expose its assumptions and stop when the question is not yet defined. The contract gives the team a shared place to make that behaviour ordinary.
Try the contract
Make the hidden choices visible.
Choose a familiar question to see the fields an AI analyst should carry into its answer. This example uses no account data and makes no claim about a result.
Contract preview
Conversion rate
- Decision
- Decide whether a landing-page message or tracking change is needed.
- Definition
- Completed enquiry forms divided by eligible sessions, with test and duplicate records excluded.
- Population and grain
- Website sessions and completed forms, counted at the session and form-event grain.
- Boundary and time
- Paid-social first-touch attribution within the agreed local-time reporting window.
- Evidence
- Return the numerator, denominator, filters, source freshness and definition owner.
- Stop condition
- Pause if CRM and analytics totals disagree or the attribution rule is missing.
Use the preview as a prompt for your own definition. A blank, unknown or conflicting field should produce a question for the owner, not a guessed value.
Sources7 references
Evidence & further reading
Numbered references in the article point to the documents below. Each title opens the original source.
- 01
Google Cloud
Conversational Analytics in Looker overview - 02
- 03
Databricks
Metric views: basic modeling - 04
Microsoft Learn
Power BI Copilot semantic-model guidance - 05
Microsoft Learn
Ask Copilot questions about your data - 06
Microsoft Research
How Do Analysts Understand and Verify AI-Assisted Data Analyses? - 07
NIST
AI RMF Core