Technical product judgment

Why ask for p95 or p99 latency as well as an average?

Averages can hide a poor experience for a meaningful minority. Define the population and measurement boundary, and inspect low-volume uncertainty rather than treating a percentile as self-explanatory.

Why can every service meet its local latency target while checkout is slow?

Sequential calls, browser work, queues, and retries add to the user journey. Set an end-to-end objective and trace the critical path rather than assuming local targets compose automatically.

What should '99.9% available' mean in a product requirement?

A defined fraction of successful eligible requests or time intervals over a stated window, measured at an explicit boundary. Clarify whether degraded or incorrect responses count as success.

How should a product team use an error-budget policy?

Agree in advance how measured unreliability changes rollout pace and reliability investment. Budget is a decision mechanism tied to user expectations, not permission to deliberately cause arbitrary harm.

How does data freshness differ from request latency?

Latency measures how long the response takes; freshness measures how current its information is. A fast dashboard can show yesterday's data, so specify both when decisions depend on recent events.

What product behavior should be specified when reads may lag writes?

Whether users must immediately see their own changes, how pending state is shown, and which actions require current data. Engineering can then choose consistency mechanisms around the actual user contract.

What should the UI promise when a backend accepts a long-running job?

Acceptance and a durable job ID, followed by progress and a final result or failure. Do not display 'complete' merely because submission succeeded; support reconnects and clear retry behavior.

Why is 'just retry the payment' an incomplete requirement?

A timeout may occur after the payment committed. Define operation identity, duplicate prevention, status lookup, and reconciliation so retries cannot silently charge twice.

What should a bulk-operation specification say about partial failure?

Whether the operation is atomic or item-by-item, how failed items are identified, and how retries avoid repeating completed effects. A single ambiguous error message leaves users unable to recover safely.

What should cancellation mean for a job with external side effects?

Specify which work stops, what has already committed, and whether compensation is possible. A cancelled UI state cannot promise that an email, payment, or external request was undone.

Why does a delete button require more than removing one database row?

Data may exist in indexes, caches, exports, backups, and downstream systems. Define user-visible timing, retention exceptions, restore behavior, and completion evidence with the responsible teams.

What should a PM define for a new sharing feature?

Who can grant access, which actions and resources are included, inheritance, expiration, revocation, and visibility of metadata. 'Team members can share' leaves important authorization rules unspecified.

Why distinguish viewing, editing, approving, and administering?

They carry different authority and risks. Combining them in one broad role can let a routine collaborator approve their own changes or alter access without the intended oversight.

What makes an audit-history requirement useful?

Specify the actions, actors, before/after context, retention, access, and integrity needs. An activity feed optimized for convenience may not provide trustworthy evidence for sensitive operations.

What product commitments belong in an API contract?

Supported requests, responses, error semantics, pagination, limits, versioning, and compatibility expectations. Integrators depend on behavior and timing as well as field names.

Why can changing a field's meaning break clients even if its type stays the same?

Clients may interpret a number as cents rather than dollars or a status as final rather than pending. Treat semantic changes as contract changes and plan migration and notice.

What must be known before retiring an integration endpoint?

Active and infrequent consumers, replacement readiness, migration guidance, support ownership, and the announced end policy. A quiet week of traffic does not prove every scheduled customer job migrated.

What should customers know about webhook delivery?

Authentication/signature verification, retries, ordering, duplicate delivery, retention, and recovery options. Design consumers to handle the documented guarantees rather than promising 'exactly once' without a defined boundary.

How can rate limits be made usable for API customers?

Document units and scope, return clear retry guidance, expose usage where appropriate, and distinguish burst from sustained limits. Limits should protect shared capacity without making normal workflows unpredictably fail.

Why is search latency alone an insufficient acceptance criterion?

Users need relevant, authorized, fresh results and a workable empty-result experience. Evaluate representative queries, ranking, coverage, filters, and permission behavior alongside speed.

What product tradeoff can a ranking objective conceal?

Optimizing clicks may promote sensational or low-quality results, while optimizing purchases may reduce discovery. Define long-term value and constraints, then inspect segment and supplier effects.

Why should recommendations have a fallback?

New users, sparse catalogs, missing features, or dependency failures may leave the model unsupported. Define a safe useful baseline and measure when it is used rather than returning arbitrary personalized output.

How should an AI feature's task be scoped for evaluation?

Specify inputs, expected outputs or outcomes, permitted tools, failure categories, and representative user conditions. A broad claim that the assistant is 'smart' cannot be tested or compared reliably.

What should an AI evaluation set contain beyond easy examples?

Representative routine tasks, costly failures, edge cases, ambiguous inputs, and relevant adversarial cases, with versioned scoring criteria. Keep development feedback distinct from a held-out release assessment.

Why should model-graded quality not be accepted without calibration?

The judge can be inconsistent, biased, or fooled by style and shared model errors. Compare against human-reviewed examples and known failures, track agreement, and reserve stronger checks for high-consequence outcomes.

What should a product promise about generated factual answers?

A bounded use case with grounding, evidence display where useful, uncertainty handling, and verification or escalation for costly errors. A confident tone or retrieval step does not guarantee factual correctness.

When can retrieval improve an AI product without changing model weights?

When needed information is external, frequently updated, or permission-dependent. The product still needs retrieval quality, source freshness, access enforcement, and evaluation of whether the model actually uses the evidence.

What question should precede a request to fine-tune a model?

Which repeatable error should training change, and is there suitable data and an evaluation to measure it? Compare prompt, retrieval, workflow, and model-selection fixes before committing training and maintenance cost.

How does giving an AI tool access change the product risk?

Errors can become external actions rather than incorrect text. Define authority, confirmation requirements, runtime limits, idempotency, and auditability for each tool, with sensitive authorization enforced outside the model.

What makes human review a real control rather than a label?

Reviewers have enough context, time, expertise, and authority to reject or correct the action. Measure workload and error escape; rubber-stamping high-volume output does not provide reliable oversight.

Why distinguish time to first output from time to a useful final result?

Streaming can make a response appear quickly while the task still takes a long time or fails. Measure both perceived responsiveness and completed task quality, including tool and verification steps.

Why measure cost per successful task instead of only cost per token?

Retries, long prompts, tool calls, failures, and human correction all affect delivered value. A cheaper model can cost more overall if it needs more attempts or causes more review work.

What should happen before a provider model change reaches all users?

Run the task-specific evaluation, check formats and tool behavior, assess quality/cost/latency, and stage rollout with version tracking and recovery. Provider benchmarks do not establish compatibility with the product's workflow.

What should be resolved before customer content is sent to an AI vendor?

Permitted use, retention, training use, access, location requirements, deletion, and incident handling under the actual service agreement. Coordinate with responsible privacy/security owners and implement the agreed data boundaries.

Why must a delivery plan identify external dependencies explicitly?

Vendor access, data readiness, approvals, and partner changes can determine the critical path. Assign owners and acceptance criteria, with alternatives when a dependency misses its date.

What should a technical feasibility spike deliver?

Evidence answering a bounded question, such as performance at realistic data size, plus limitations and a decision. Prototype code without a measured conclusion does not resolve the uncertainty.

Why request assumptions and ranges with an engineering estimate?

Unresolved integrations, migrations, and quality requirements can dominate uncertainty. A range with explicit drivers supports scope decisions better than a precise date unsupported by evidence.

What user-facing concerns belong in a migration plan?

Continuity, data correctness, compatibility, progress visibility, support, and recovery. Test representative old and new states; deployment success alone does not show customer data migrated correctly.

What does a feature flag enable, and what can it not undo?

It can control new use of a path without rebuilding, subject to propagation and implementation. It does not reverse data changes or completed side effects, so recovery may require more than turning it off.

What should a rollout gate measure?

Relevant user outcomes, errors, latency, data correctness, and segment-specific risks against predefined criteria. A healthy process or successful deployment command is not sufficient evidence to expand traffic.

What must be specified for offline editing?

Which data is available, how pending changes are represented, authentication expiry, conflict resolution, and retry behavior on reconnect. 'Sync later' leaves the hardest user-visible decisions undefined.

Two users edit the same record. What product decision is needed?

Whether to reject stale writes, merge particular fields, preserve versions, or use a documented winner rule. Silent last-write-wins can lose meaningful work even when the implementation behaves consistently.

How should accessibility enter acceptance criteria?

Specify relevant keyboard, focus, semantic, contrast, zoom, error, and assistive-technology behavior for the workflow. Include representative testing early rather than reducing it to one automated score at launch.

What telemetry should be planned with a new feature?

Events and traces linking attempts, completion, failure reasons, latency, and version, with privacy controls. Define ownership and interpretation so the team can distinguish adoption problems from system failures.

What does a PM contribute during a product incident?

Clarify user impact, critical workflows, acceptable degradation, and communication needs while technical leads coordinate mitigation. Avoid changing the recovery plan through parallel stakeholder requests.

How should recurring manual support work influence product priorities?

Measure volume, user harm, staff time, and root causes. A workflow that appears successful only because support repeatedly repairs it may have hidden delivery cost and reliability debt.

What makes an acceptance example more useful than 'works as expected'?

A concrete input and state, action, observable result, and failure or boundary case. Examples make disagreements visible and help product and engineering verify the same behavior.

What should be ready beyond the shipped code?

Monitoring, support guidance, ownership, capacity, rollout controls, migration steps, and recovery procedures appropriate to risk. The exact checklist should follow the feature's failure modes rather than grow without purpose.

How can a PM evaluate a technical tradeoff without dictating implementation?

Compare options by user-visible behavior, cost, delivery time, reliability, reversibility, and evidence. Ask engineering to expose assumptions and failure modes, then decide the product constraints collaboratively.