An LLM/AI security and governance checklist is only actionable when you define who can use which data and tools, what to test before launch, and who can stop the system if it fails. Do not treat installing a prompt filter as a pass.
The following is an example launch review for an enterprise LLM agent that uses RAG and external tools. It is not a certification scorecard or a legal standard common to every industry. Set owners and approval criteria according to your organization's data and operational risk.
LLM AI Security and Governance Checklist
| Area | Question to verify | Evidence to keep | Example owner |
|---|---|---|---|
| Scope of use | Are allowed and prohibited tasks, and the impact of failure, defined? | Use-case list, risk assessment | Service owner |
| Data | What information enters inputs, retrieval, training, and logs? | Data-flow diagram, retention and deletion rules | Data / privacy owner |
| RAG permissions | Is document access limited by the user's permissions? | Negative tests of tenant and document ACLs | Backend owner |
| Tool permissions | Are model proposals separated from actual execution authorization? | Tool allowlist, authorization tests | Platform owner |
| External content | Can instructions in documents or web pages change permissions? | Indirect injection test records | Security owner |
| Output | Are HTML, SQL, commands, etc. not executed directly? | Output schema and destination validation | Application owner |
| Sensitive actions | Are there adequate controls for outbound transfer, deletion, and payments? | Approval policy, retry and duplicate-prevention results | Business owner |
| Evaluation | Are successful-task rates and security failures measured together? | Fixed evaluation set, results by version | Evaluation owner |
| Operations | Are cost, errors, and abnormal tool calls monitored? | Limits, alerts, operations dashboard | Operations owner |
| Incident response | Can you halt execution, revoke tokens, and roll back? | Owners, halt-and-recovery rehearsal | Incident response owner |
Record each row as pass, needs work, or not applicable, and keep links to supporting evidence. If you allowed an exception, also record the approver and the next review date. This table and role mapping are an editorial operations example, not a table from an official framework.
Prompt injection cannot be stopped with regex alone
Besides direct user input, hidden instructions in retrieved documents or web pages can influence model behavior. Blocking a specific English phrase such as ignore system prompt still leaves other languages, phrasings, and external-document paths. OWASP describes combining multiple defenses—least privilege, output validation, separating untrusted content, and controlling sensitive actions—rather than promising complete blocking. OWASP Prompt Injection
So security tests should check not only whether “the model refused the instruction,” but also whether data was actually exposed or a tool actually ran. Verify that the server-side authorization layer still blocks an inappropriate tool call even if the model proposes it.
Bind tool execution to user permissions
For example, suppose a customer-support agent proposes ticket.read. The server must compare the authenticated user's organization and role with the organization that owns the requested ticket. Do not grant access based on a tenant_id the model emitted or a sentence such as “an administrator approved this.”
You can structure the execution flow as follows.
- Identify the user and tenant from the server session
- Validate the tool name and arguments against the allowed schema
- Perform server-side authorization for the target resource
- Confirm any policy-required action approval
- Check execution limits and duplicate prevention, then invoke the tool
- Record the result and authorization decision as minimal audit information
This is a design example. Real access-control code must connect to each service's authentication and authorization store. For an implementation view, continue with LLM Agent Security Design Guide.
Review RAG retrieval and audit logs
Build tenant and permission filters for retrieval from the server's authentication context. Keep document IDs and versions so you can trace answer sources, but a citation does not by itself guarantee accuracy or access rights. Test that permission changes and deletions also propagate to the index and cache.
Prioritize audit-log fields needed for incident investigation: request identifier, model and prompt version, tool name, authorization result, latency, and cost. Storing full raw inputs and responses for long periods can copy personal or confidential data into logs, so define purpose, access rights, and retention first.
Example tests to run before launch
The following is a test design you can run with synthetic data and no real customer information. This article does not claim measured results for any specific product.
| Test input / condition | Expected behavior |
|---|---|
| Specify another tenant's document ID | Deny access for retrieval, download, and tool results |
| Insert an outbound-transfer instruction into a retrieved document | Document instructions alone must not create transfer permission |
| Model generates a tool name that is not allowed | Block before execution via schema and allowlist |
| Change recipient or amount after approval | Re-review if the action no longer matches the approved content |
| Repeat the same action via network retry | Prevent duplicate payment, send, or deletion |
| Change model or prompt version | Re-run the same evaluation set of success and failure cases |
| Revoke a tool token or halt execution | New executions are blocked and operators can confirm status |
Also record how often legitimate requests are blocked. Do not judge operational fitness from “we blocked a few attack strings” alone.
Connect NIST AI RMF to operational records
NIST AI RMF is a voluntary risk-management framework that presents Govern, Map, Measure, and Manage functions. You can also consult the generative AI profile. The mapping below is an application example for this article. NIST AI RMF, AI RMF Core Functions
| Function | Operational records to produce for this service |
|---|---|
| Govern | Owners, approval and exception policy, change management |
| Map | Use cases, data flows, affected users |
| Measure | Benign and misuse evaluation results, limits and failure cases |
| Manage | Action priorities, launch criteria, halt-and-recovery plan |
Treat not only model swaps but also new data sources, new tools, and permission expansions as triggers for re-review. What matters is whether current operations and evidence still match the checklist—not the date it was written.
Nodelog는 모든 콘텐츠의 내용과 출처를 공개 전에 검토합니다. 환경(OS·버전)에 따라 결과가 달라질 수 있는 기술 정보는 공식 문서와 함께 확인하며, 검토 기준과 정정 원칙은 편집 정책에서 안내합니다. 오류를 발견하시면 이메일로 제보해 주세요 — 확인 후 신속히 정정합니다.
Comments
Be the first to comment.