A Prompt Is a Layered Program: Mapping Prompt Engineering Techniques to the Layer They Change

You move a working ticket-triage feature from OpenAI to Claude through LiteLLM, and the only thing you change is the model id. Every request succeeds. A week later, support agents report summaries that run past the 120 characters the UI was built for. The Pydantic model behind the output still says max_length=120.

Nothing in the prompt text changed. What changed is how one layer of the prompt was delivered. Anthropic's structured outputs reject maxLength with a 400, so LiteLLM 1.106.0 removes it from the JSON Schema and moves the limit into the field description as a note [4][6]. Gemini accepts the keyword and silently ignores it [5]. Both paths turn a hard constraint into a suggestion.

My position is that a prompt is a layered program, and each prompt engineering technique targets exactly one layer. Once you know which layer a technique touches, you know what will break when the model or provider underneath it changes.

Why One Tuned String Breaks on the Next Model

The common approach treats a prompt as a single string, tuned by trial and error against one model until the outputs look right. That tuning fits one model's quirks. PromptBridge measured the cost on HumanEval: the best prompt for GPT-5 scored 68.70% when reused on Llama-3.1-70B-Instruct, against 79.47% for Llama's own best prompt [1]. Moving to a stronger model does not avoid it either; a prompt engineered for GPT-4o reached 92.27% on o3, below the 98.37% that o3 achieves with a prompt optimized for it [1]. Pure formatting sensitivity has shrunk on current closed-weight models, which show little accuracy loss when asked for JSON, but open-weight models still pay that cost [7].

OpenAI's own guidance points the same way. It recommends pinning production applications to dated snapshots and building evals, since different models may need different prompts [2]. A string tuned for one snapshot is a program compiled for one target, with no record of which parts depended on that target.

The fix starts with naming the parts.

What a Layered Prompt Program Is

A prompt program is the set of inputs that define one model call, kept as separate layers, each of which travels through a different slot of the API request. There are six:

• L1 instructions: the rules for every request. OpenAI sends them as a developer or system message [2]; Anthropic takes system as a top-level request parameter, outside the message list [3].
• L2 tools: tool schemas in the tools field, with optional strict validation [13][4].
• L3 examples: demonstrations, sent as prior user and assistant turns or as tagged blocks [3].
• L4 context: retrieved documents. Anthropic and Google both recommend placing long documents above the query [3][10].
• L5 task: what this particular request asks for.
• L6 output contract: the schema the answer must satisfy, sent as response_format on OpenAI [13], output_config.format on Anthropic [4], and a schema subset on Gemini [5].

Request configuration such as temperature or reasoning effort is not a prompt layer, but it breaks on model upgrades in the same way, so the sample treats it alongside the layers.

In my opinion, the useful definition of a prompt engineering technique is the layer it edits. If you cannot say which part of the request a technique changes, you cannot say what will break when the model changes.

Which Layer Each Technique Changes

Mapping the well-known techniques to layers shows that most of them edit exactly one, and that two of the most cited ones are not prompt edits at all. The evidence column uses only studies and provider guidance from the past year, measured on current models.

The table also explains why some techniques age badly. A chain-of-thought trigger lives in L5, and ReasonLab found it worth about 4 points without a reasoning budget and nothing measurable once reasoning is enabled, while enabling reasoning itself was worth up to 12.74 points [9]. Google's Gemini 3 guide tells you to replace chain-of-thought prompt engineering with a high thinking level and simpler prompts [18]. Anthropic's current guide warns that emphatic "CRITICAL: You MUST" phrasing written for older models now overtriggers [3]. Instructions that once improved results became instructions you must delete on upgrade.

Close

How the Compiler Places Each Layer

The sample keeps the six layers in a frozen PromptProgram dataclass and compiles them per model. Read the diagram from top to bottom.

PromptProgram owns the layers and nothing else. It holds no provider name or request syntax, so the same program can target any model.

ModelProfile owns every provider difference the code knows about. It records whether the model can enforce a JSON schema and whether it accepts temperature. It also lists the schema keywords to strip, and its optional instruction_override replaces L1 for that model. profile_from_litellm() fills the capability flags from LiteLLM's bundled model map, using supports_response_schema() and get_supported_openai_params() [6].

compile_request() takes both and emits the keyword arguments for LiteLLM's completion(). Each layer goes to its own slot:

# src/promptlayers/compile.pydef compile_request(    program: PromptProgram, profile: ModelProfile, *, temperature: float | None = None) -> dict[str, Any]:    # L1: LiteLLM moves a "system" message to Anthropic's top-level system field.    instructions = profile.instruction_override or program.instructions    messages: list[dict[str, Any]] = [{"role": "system", "content": instructions}]    for ex in program.examples:  # L3: demonstrations as prior turns        messages.append({"role": "user", "content": ex.input})        messages.append({"role": "assistant", "content": ex.output})    docs = "".join(f"<document>\n{doc}\n</document>\n" for doc in program.context)    messages.append({"role": "user", "content": f"{docs}<task>\n{program.task}\n</task>"})  # L4, L5    request: dict[str, Any] = {"model": profile.model, "messages": messages}    if program.tools:  # L2        request["tools"] = list(program.tools)

The function uses the system role for L1 because LiteLLM's Anthropic transform moves system messages into Anthropic's top-level system field [6]. Context goes above the task inside one user message, which follows the placement advice in both Anthropic's and Google's guides [3][10]. L6 becomes a strict json_schema with additionalProperties: false on every object and the non-portable keywords removed. When the profile says a layer cannot be carried, the function raises UnsupportedLayerError instead of sending a weaker request.

run() sends the request through LiteLLMModel, which calls litellm.completion() with drop_params=False, a timeout and a retry count. It then validates the raw text against the full Pydantic model, including the constraints that never reached the provider. A failed validation is appended as feedback and retried within max_attempts; when the budget runs out, run() raises ContractError. A reply that contains tool calls raises ToolCallError, because this sample has no tool loop, and a tool call is not a malformed answer.

The full sample is in the tensorup-examples repo; uv run pytest -q in its folder runs 17 offline tests that cover each layer's placement and every failure mode below.

Design Decisions and Trade-offs

LiteLLM as the translation layer. LiteLLM accepts OpenAI-shaped requests and translates them per provider, including the system-message move and the schema filtering described above [6]. Per-provider SDK adapters give full control over each request at the cost of one adapter per provider to write and keep current. I chose LiteLLM because it gives agent framework SDKs one provider-agnostic model class: Strands Agents, for example, takes a LiteLLMModel, and the LiteLLM model id decides the provider [19]. It also exposes prompt caching for OpenAI, Anthropic and Gemini through the same interface, which matters for token cost [20]. Check the cache counters in the response's usage object, though: LiteLLM's docs warn that caching is skipped without an error when the input is below the provider's minimum, which is 1,024 tokens on OpenAI [20].

A profile you can override. LiteLLM's model map is a convenient starting point, but it is a third-party description of each provider. profile_from_litellm() therefore accepts overrides for any flag:

# src/promptlayers/profiles.py    params = litellm.get_supported_openai_params(model=model) or []    detected = {        "supports_json_schema": litellm.supports_response_schema(model=model),        "supports_temperature": "temperature" in params,        # Send only the portable schema subset everywhere; Pydantic enforces the rest.        "rejected_schema_keywords": NON_PORTABLE_KEYWORDS,    }    return ModelProfile(model=model, **{**detected, **overrides})  # type: ignore[arg-type]

The portable schema subset everywhere. The sample strips minimum, maximum, minLength and maxLength for every provider, including OpenAI. The cost is that OpenAI no longer enforces those bounds while decoding, so a violation costs one extra call through the retry loop. In return, the schema on the wire is identical for every provider, and one validator enforces the full contract on the client. Anthropic's Python SDK makes the same choice: it removes unsupported constraints and validates responses against the original schema [4].

Fail loud over drop_params. LiteLLM can silently remove parameters a provider does not support when drop_params=True [14]. That keeps requests from failing, and it also hides configuration that no longer has any effect. The sample keeps LiteLLM's default of raising and adds its own compile-time check, so the failure appears in CI rather than in production output.

Where It Breaks

Silently dropped parameters. Symptom: a model change that should have altered nothing makes outputs noticeably more varied. Cause: drop_params=True removed temperature, which OpenAI's guide for its latest models says must be removed whenever reasoning effort is not none [15][14]. Fix: leave drop_params off, record supports_temperature per model, and let compile_request() raise UnsupportedLayerError.

An output contract that became advisory. Symptom: values outside the declared range reach downstream code, with no errors logged. Cause: the provider or the translation layer removed the constraint from the schema; LiteLLM does this for Anthropic, and Gemini ignores minLength and maxLength [6][5]. Fix: treat the wire schema as a decoding hint and validate the raw output with the full Pydantic model on every response, with a bounded retry.

Instructions tuned for the previous model. Symptom: after an upgrade, the model overthinks, re-verifies its own answers or follows an emphatic rule too literally. Cause: L1 and L5 still carry chain-of-thought triggers and emphatic phrasing written for an older model [3][18]. Fix: pin the snapshot [2], keep a per-model instruction_override, and run the same eval set against the new snapshot before switching.

Checklist

• Store instructions, tools, examples, context, task and output schema as separate fields, never as one string.
• Pin every production model to a dated snapshot [2].
• Record each model's capabilities in one profile object and keep provider checks out of prompt code.
• Turn off drop_params and fail at compile time on unsupported parameters.
• Send a portable schema subset and validate every response with the full schema on the client.
• Remove chain-of-thought triggers from prompts that target models with reasoning enabled [9][18].
• Bound retries with an attempt budget and raise a named error when it runs out.
• Run the same eval set against a new model or provider before switching traffic.

My verdict: treat a prompt as source code with a build step. Write the layers once, keep every provider difference in one profile, and make the compiler fail loudly when a model cannot carry a layer. I learned this when a model upgrade and a provider switch broke multi-provider support that had looked finished, and the fix was structural: separating the layers, never rewording the prompt.

Which layer broke first the last time you switched models or providers?

References

1. Wang et al., PromptBridge: Cross-Model Prompt Transfer for Large Language Models, arXiv, 2025-12-01.
2. OpenAI, Text generation guide, accessed 2026-10-23.
3. Anthropic, Prompting best practices, accessed 2026-10-23.
4. Anthropic, Structured outputs, accessed 2026-10-23.
5. Google, Gemini API structured outputs, updated 2026-10-21.
6. LiteLLM, litellm/llms/anthropic/chat/transformation.py, v1.106.0, 2026-10-18.
7. Lee, D'Antoni and Berg-Kirkpatrick, The Format Tax, arXiv, 2026-04-04.
8. IH-Benchmark: A Conflict-Centered Benchmark for Instruction-Hierarchy Robustness in LLM Applications, arXiv, 2026-07-28.
9. ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice Question Answering, arXiv, v2 2026-09-22.
10. Google, Gemini API long context, updated 2026-10-15.
11. MiniMax, Interleaved Thinking Unlocks Reliable MiniMax-M2 Agentic Capability, 2025-11-03.
12. The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output, arXiv, 2026-04-28.
13. OpenAI, Structured model outputs, accessed 2026-10-23.
14. LiteLLM, drop_params, updated 2026-10-21.
15. OpenAI, Using GPT-6 (latest model guide), accessed 2026-10-23.
16. Althoubi, Reasoning Concentrates Errors, and Self-Consistency Never Notices, arXiv, 2026-09-25.
17. Yin, Ken and Stremmel, Denoising Iterative Self-Correction: Structured Verification Loops for Reliable LLM Reasoning, arXiv, 2026-06-19.
18. Google, Gemini 3 developer guide, updated 2026-10-09.
19. Strands Agents, strands-py/src/strands/models/litellm.py (LiteLLMModel), last changed 2026-10-16.
20. LiteLLM, Prompt caching, updated 2026-10-21.