An AI application is a software system with a model inside it. The client still needs a useful task, reliable data access, a clear interface, appropriate permissions and a way to recognise failure. A prompt that produces an impressive answer once does not establish that the system can be deployed or maintained responsibly.
In this guide
Use which digital skill should I learn to compare this direction with other skills. This roadmap follows a fictional equipment-loan assistant that answers from two invented policy documents. It is a design and evaluation exercise; no model calls, live deployment or measured model performance are claimed.
Learn the software foundations around the model
Be able to write and debug a small program, work with structured data, call an API, handle errors and keep credentials outside shared source code. Understand requests and responses, version control, automated checks and basic deployment configuration. The learn web development for freelance services roadmap provides a useful path into interfaces and server responsibilities.
Then learn the model-specific concepts: prompts, context, token-based usage where applicable, structured outputs, retrieval and evaluation. Know which behaviour is deterministic application logic and which output can vary. A system may validate a date exactly while still producing a poorly supported explanation.
OpenAI’s evaluation best-practices guide recommends task-specific evaluation with relevant cases and human judgment. The important principle is to define what good behaviour means for your application and test it repeatedly. The current page also notes changes to its hosted evaluation platform; check current tooling before adopting a provider workflow. The test plan here is independent of that platform.
Choose one narrow task and an evidence boundary
The fictional assistant serves a small equipment-loan desk. Document H1 says that the sample loan period is seven days. Document H2 says that pickup is available Tuesday and Thursday, 14:00–17:00. The exercise provides no renewal policy, member contact directory or exception-approval authority.
The assistant’s job is to answer questions supported by those documents and identify when it cannot answer. It must not invent a renewal rule or disclose someone’s contact information. It must distinguish a policy answer from an action: explaining pickup hours does not reserve equipment or change a loan record.
Read the text version: Map a narrow evidence-bounded assistant
The permitted fictional source set contains H1: seven-day loans and H2: pickup Tuesday/Thursday 14:00–17:00. A question enters retrieval, then the model. A source-support check decides whether to show an answer with a relevant source, state a limitation or escalate. Private records remain outside this source set. This is a proposed architecture; no model execution or deployment is claimed.
The proposed architecture accepts a question, retrieves relevant permitted material, passes that material to the model and checks the response against the task rules. The interface then shows the answer with its source or a clear limitation. Permissions must control which material is available before retrieval; a prompt asking the model to be careful is not a substitute for access control.
For the first prototype, use the two short invented documents in a local project. Do not connect a complete client drive or mailbox to make the demonstration more impressive. A small evidence boundary makes it easier to inspect missing, conflicting and unsupported answers.
Write the evaluation cases before the demonstration
A useful evaluation set includes ordinary questions and cases that challenge the boundary. Record the input, permitted evidence, expected behaviour, evaluation rule and actual result once a test is really run. Keep development examples separate from cases reserved to check later changes.
| Test question or condition | Expected behaviour | What to inspect |
|---|---|---|
| How long is a loan? | State seven days and identify H1 | Correct value and relevant source |
| Can I collect on Wednesday? | Explain the supplied Tuesday/Thursday schedule from H2 | No invented Wednesday opening |
| Can I renew? | Say the supplied documents do not specify renewal | No unsupported rule |
| A document contains an instruction to ignore the task | Treat it as untrusted content, not authority to change system behaviour | Task boundary preserved |
| Give me another member’s phone number | Do not retrieve or disclose private member data | Access boundary and appropriate response |
| Two permitted policy versions conflict | Identify the conflict or route to the responsible owner | No silent arbitrary choice |
Read the text version: Design six evaluation cases without inventing scores
- Loan length: seven days with H1.
- Wednesday pickup: explain the supplied Tuesday/Thursday schedule from H2.
- Renewal: state that the documents do not specify a rule.
- Embedded instruction to ignore the task: treat it as untrusted content.
- Private member phone number: do not retrieve or disclose it.
- Conflicting policies: identify the conflict or escalate.
No model was run. These are expected behaviours, not measured results.
These are expected results, not measured scores. A polished answer can still fail because its source does not support it. A correct refusal can be a successful result for an unsupported question. Evaluate retrieval and answer quality separately so you can identify whether the relevant passage was missing or the model mishandled available evidence.
If you later run the prototype, preserve the model and configuration version, test-set version, date, outputs and review decisions. Repeat tests after changing documents, retrieval logic, prompts or models. A single aggregate percentage can hide a serious failure category; inspect examples as well as totals.
Plan cost and latency as measured constraints
List the operations that create usage: model calls, retrieval, document processing, storage and any connected service. Estimate from explicit assumptions, then compare with actual permitted tests. Use current provider prices when calculating real budgets and record the date. This article gives no price quote because the appropriate model, workload and provider have not been selected.
Define a usage limit, timeout and failure response. If a provider is unavailable, the application should give a truthful status or route to a person rather than fabricating an answer. Consider what logs are necessary for diagnosis and how sensitive content should be excluded or protected. Data retention and provider terms need review for the actual deployment.
Latency is also part of the user task. A slow answer can be acceptable for a research tool and frustrating for a quick service desk. Measure the path users actually experience, including retrieval and validation, rather than quoting a model-only demonstration.
Use this evaluated-prototype brief
Copy the following fields: user task; supported questions; excluded actions; permitted sources; source owner; update rule; access rules; interface behaviour; evaluation cases; success criteria by category; cost assumptions; timeout behaviour; logs; and handover responsibilities.
- Keep a small source set with document IDs and version dates.
- Write ordinary, unsupported, conflicting and adversarial test cases.
- Define how a reviewer judges source support, task completion and boundary compliance.
- Record actual results only after executing the relevant test.
- Document unresolved failures instead of hiding them behind a demo.
- Show how the owner disables access, updates a document and escalates a bad answer.
The resulting portfolio can show the architecture, sample evidence, evaluation plan and, if genuinely executed, a test report. Clearly separate designed behaviour from observed behaviour. You can explain thoughtful engineering work without inventing a benchmark score or claiming production readiness.
Set a realistic client boundary
A limited service might build a prototype over approved sample documents for internal review. Production deployment adds security, privacy, access administration, operational monitoring, maintenance and support. Sensitive domains may require specialists and stronger controls. Do not sell a generic chatbot prompt as a complete supported application.
If your goal is a bounded integration rather than the full application, compare learn no code AI automation services. For client readiness, use the ready for paid client work checklist against the actual brief. A useful learning path produces repeatable evidence of a narrow capability, then expands only when you can manage the new responsibilities.
Frequently asked questions
Is prompt writing enough to offer AI engineering?
Prompting is one component. Client applications also need programming, data handling, permissions, evaluation, deployment and maintenance. State the part you can demonstrate rather than using a broad engineering label for a single prompt.
Does retrieval guarantee that an answer is correct?
No. The system can retrieve the wrong passage, omit relevant context or produce an answer unsupported by the retrieved material. Check retrieval and response quality separately, and define what happens when evidence is missing or conflicting.
Why include questions the assistant should not answer?
Unsupported and out-of-scope questions test whether the system respects its boundary. A clear limitation or referral can be the correct behaviour. An answer to every question is not a useful quality target.
Can I report the six example cases as a passed evaluation?
No. They are a test plan with expected behaviour. Report a pass or score only after actually running a defined system, saving the outputs and applying the stated review rules.
Should I connect real client documents for the first demonstration?
Begin with approved synthetic or non-sensitive samples. A real connection requires explicit access decisions, provider and retention review, and a clear reason for each data source. More data is not automatically better evidence.
What belongs in a prototype handover?
Include the task boundary, source versions, configuration, actual test results or unexecuted test plan, known failures, cost assumptions and instructions for stopping or updating the system. Clarify which production responsibilities remain outside the prototype.