We're seeing an issue where a single agent workflow can retain state on tool calls that it has performed (persist id 123) but as soon as we go multi-modal and pass in past conversation history the ids aren't persisted and are sometimes hallucinated (id becomes 456). Are we doing something wrong or are there persistent patterns for tool calls that aren't captured by just passing session history?

Thanks for reaching out, Arthur. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I'd like an example of this in practice, stage one of a workflow retrieve a user's driversLicenseNumber but we want to ensure that when we move to part 2 which is an agent to help change addresses / do other action - this driverLicenseNumber is required to look up the current address and then propose a change of address into the system. How would this be captured / addressed properly?

Thanks — I’ll show a concise, practical example for your two-stage workflow: Stage 1 collects

driversLicenseNumber

and stores it reliably, Stage 2 is an agent that uses that stored value to look up the current address and propose a change. I’ll focus on patterns you can adopt so the LLM does not invent or change the ID/value when you replay history. Key points up front • Persist the authoritative data (the driver’s license number and any tool-call metadata) into session state or your DB immediately when Stage 1 completes. • When invoking Stage 2, inject that persisted data as explicit structured context the agent can read (not only as raw chat history). • Return and persist structured tool outputs (include an explicit

id

field) and surface those in follow-up steps so the model uses the saved identifiers rather than inventing new ones. Example flow (conceptual + JSON-like payloads) 1) Stage 1 — Capture and persist the driver’s license • User: “My driver’s license is D1234567.” • Agent/tool collects value and returns structured result. Example tool return (Stage 1)

{
  "tool": "collectDriverLicense",
  "result": {
    "driversLicenseNumber": "D1234567"
  },
  "tool_call_id": "tool_call_123",
  "timestamp": "2026-03-02T12:00:00Z"
}

Action you must take immediately after the tool returns • Persist a record in session state / database like:

{
  "session_id": "sess_abc",
  "user_id": "user_789",
  "collected": {
    "driversLicenseNumber": "D1234567",
    "collected_by_tool_call_id": "tool_call_123",
    "collected_at": "2026-03-02T12:00:00Z"
  }
}

• Also persist any tool response body that matters (e.g., verification flags). Why: the persisted record is the single source of truth for later stages, so the LLM doesn’t have to “remember” or invent IDs. 2) Stage 2 — Lookup current address and propose change • Before calling the agent that performs address lookup/action, build an explicit structured context payload from persisted data and pass it into the agent as an authoritative block (not just chat history). Example structured context injection passed to the Stage 2 agent

{
  "context_block": {
    "purpose": "address_lookup_and_update",
    "authoritative_data": {
      "driversLicenseNumber": "D1234567",
      "collected_by_tool_call_id": "tool_call_123",
      "collected_at": "2026-03-02T12:00:00Z"
    }
  }
}

Prompting / system instruction guidance for Stage 2 • Give the agent a short system instruction that says: ◦ Use the

authoritative_data

block for any lookups. ◦ Do not invent or change

driversLicenseNumber

or

tool_call_id

. ◦ When you call the lookup tool, include the

collected_by_tool_call_id

as provenance in your request. Example Stage 2 tool call payload (address lookup)

{
  "tool": "lookupAddressByDLN",
  "input": {
    "driversLicenseNumber": "D1234567",
    "provenance_tool_call_id": "tool_call_123"
  }
}

Example tool response (Stage 2)

{
  "tool": "lookupAddressByDLN",
  "result": {
    "currentAddress": {
      "street": "123 Main St",
      "city": "Hometown",
      "zip": "12345"
    },
    "verified": true
  },
  "tool_call_id": "tool_call_456",
  "timestamp": "2026-03-02T12:02:00Z"
}

Action after Stage 2 tool returns • Persist the lookup result with provenance:

{
  "session_id": "sess_abc",
  "address_lookup": {
    "currentAddress": {
      "street": "123 Main St",
      "city": "Hometown",
      "zip": "12345"
    },
    "verified": true,
    "lookup_tool_call_id": "tool_call_456",
    "used_provenance_tool_call_id": "tool_call_123",
    "lookup_at": "2026-03-02T12:02:00Z"
  }
}

3) Propose change-of-address and perform action • When preparing the change proposal, include both the authoritative DLN and the lookup result you persisted: ◦ “Using driversLicenseNumber=D1234567 (collected_by=tool_call_123) we found current address X (lookup id tool_call_456). Do you want to change to Y?” • If the agent needs to call a change tool, include the persisted provenance fields in that tool request:

{
  "tool": "changeAddress",
  "input": {
    "driversLicenseNumber": "D1234567",
    "newAddress": {
      "street": "456 Oak Ave",
      "city": "Newtown",
      "zip": "67890"
    },
    "provenance": {
      "collected_by_tool_call_id": "tool_call_123",
      "address_lookup_tool_call_id": "tool_call_456"
    }
  }
}

• Persist the change result and its

tool_call_id

as well. Why this pattern prevents hallucinated IDs • The LLM sees explicit, authoritative fields you created (the persisted DLN and tool_call IDs). It reads and references exact values rather than guessing. If history is also included as chat, the structured authoritative block still wins because it’s treated as the system-supplied truth. • Always persist tool outputs and include their IDs as provenance in subsequent tool calls. That converts ephemeral runtime identifiers into durable application state. Quick checklist you can implement now 1. When a tool returns a DLN, immediately save

driversLicenseNumber

+

tool_call_id

to session state/DB. 2. Always feed Stage 2 an authoritative structured context block that contains that saved data. 3. Require agents to include the saved

tool_call_id

as provenance in subsequent tool calls. 4. Persist every subsequent tool response with its

tool_call_id

. 5. If you present prior actions in chat history, also include a short machine-readable provenance block so the agent won’t invent IDs. Would you like: • A concrete code example in your preferred language showing how to persist and inject the context? • A short template for the system instructions/prompts to guarantee the agent treats the injected block as authoritative? Tell me which you'd prefer and what language (or runtime) you’re using and I’ll provide the tailored example.

We are using Python - can you write out an example with an agent to show this? We want to persiste to session but ensure we don't write it to chat context like speech

Thanks — I’ll show a concise, practical Python example that demonstrates: • Stage 1: collect a

driversLicenseNumber

via a tool/agent and persist it in session state (not in chat speech). • Stage 2: call an agent that receives the persisted data as a structured, authoritative context block (not injected as chat), uses it to call a lookup tool, and then proposes an address change including provenance IDs. • Key best-practice points so the LLM won’t invent or change IDs. Notes up front • This example treats

session.userdata

(or your DB) as the single source of truth. We never put the raw DLN or tool-call IDs into chat/speech — we pass them only in structured context. • The code is an illustrative pattern you can adapt to your agent framework and HTTP/SDK calls. Example (Python, pseudo-SDK style) • Assumptions: you have functions/clients to call agents and tools, and a session store API (could be a DB or session service). Adjust names to your SDK.

import uuid
from datetime import datetime

# --- Helpers: session storage (simplified) ---
# Replace with your real session DB / persistent store.
SESSIONS = {}

def save_session(session_id, data):
    existing = SESSIONS.get(session_id, {})
    existing.update(data)
    SESSIONS[session_id] = existing

def load_session(session_id):
    return SESSIONS.get(session_id, {})

# --- Helpers: simulate calling a tool (returns structured result with tool_call_id) ---
def call_tool_collect_driver_license(user_input):
    # In real life this is your agent/tool execution that parses/collects the DLN
    tool_call_id = "tool_call_" + str(uuid.uuid4())
    result = {
        "tool": "collectDriverLicense",
        "result": {"driversLicenseNumber": user_input.strip()},
        "tool_call_id": tool_call_id,
        "timestamp": datetime.utcnow().isoformat() + "Z"
    }
    return result

def call_tool_lookup_address_by_dln(drivers_license_number, provenance_tool_call_id):
    # Real lookup would query authoritative system; return a structured response
    tool_call_id = "tool_call_" + str(uuid.uuid4())
    result = {
        "tool": "lookupAddressByDLN",
        "result": {
            "currentAddress": {
                "street": "123 Main St",
                "city": "Hometown",
                "zip": "12345"
            },
            "verified": True
        },
        "tool_call_id": tool_call_id,
        "used_provenance_tool_call_id": provenance_tool_call_id,
        "timestamp": datetime.utcnow().isoformat() + "Z"
    }
    return result

def call_tool_change_address(drivers_license_number, new_address, provenance):
    tool_call_id = "tool_call_" + str(uuid.uuid4())
    result = {
        "tool": "changeAddress",
        "result": {"status": "success", "appliedAddress": new_address},
        "tool_call_id": tool_call_id,
        "provenance": provenance,
        "timestamp": datetime.utcnow().isoformat() + "Z"
    }
    return result

# --- Stage 1: Collect DLN and persist to session (NOT written to chat) ---
def stage1_collect_dln(session_id, user_input):
    # user_input: e.g., "My driver's license is D1234567" (but we will extract exact DLN)
    tool_response = call_tool_collect_driver_license(user_input)
    dln = tool_response["result"]["driversLicenseNumber"]
    collected_tool_call_id = tool_response["tool_call_id"]
    collected_at = tool_response["timestamp"]

    # Persist authoritative data to session.userdata (or your DB)
    save_session(session_id, {
        "collected": {
            "driversLicenseNumber": dln,
            "collected_by_tool_call_id": collected_tool_call_id,
            "collected_at": collected_at
        }
    })

    # Return a user-facing confirmation (this should not leak internal IDs)
    return {
        "user_message": "Thanks — I’ve recorded your driver’s license number. We’ll use it to look up your address when you’re ready.",
        # Don't include tool_call_id or raw DLN in chat context if you want to avoid speech leakage
    }

# --- Stage 2: Use persisted DLN as structured context (no chat leakage) ---
def stage2_lookup_and_propose_change(session_id):
    session = load_session(session_id)
    collected = session.get("collected")
    if not collected:
        return {"error": "No collected driver license in session. Run Stage 1 first."}

    # Build authoritative structured context block to pass to the agent
    authoritative_context = {
        "purpose": "address_lookup_and_update",
        "authoritative_data": {
            "driversLicenseNumber": collected["driversLicenseNumber"],
            "collected_by_tool_call_id": collected["collected_by_tool_call_id"],
            "collected_at": collected["collected_at"]
        }
    }

    # Instruct your agent to use 'authoritative_context' for lookups.
    # Here we directly call the lookup tool with the DLN and provenance.
    lookup_response = call_tool_lookup_address_by_dln(
        drivers_license_number=authoritative_context["authoritative_data"]["driversLicenseNumber"],
        provenance_tool_call_id=authoritative_context["authoritative_data"]["collected_by_tool_call_id"]
    )

    # Persist lookup result with provenance
    save_session(session_id, {
        "address_lookup": {
            "currentAddress": lookup_response["result"]["currentAddress"],
            "verified": lookup_response["result"]["verified"],
            "lookup_tool_call_id": lookup_response["tool_call_id"],
            "used_provenance_tool_call_id": lookup_response["used_provenance_tool_call_id"],
            "lookup_at": lookup_response["timestamp"]
        }
    })

    # Prepare a user-facing proposal (do not include internal IDs/sensitive DLN in speech)
    current_address = lookup_response["result"]["currentAddress"]
    user_message = (
        f"I found your current address as {current_address['street']}, "
        f"{current_address['city']} {current_address['zip']}. "
        "Would you like to change it? If yes, please provide the new address."
    )
    return {"user_message": user_message}

# --- Stage 3: Apply change with provenance ---
def stage3_apply_change(session_id, new_address):
    session = load_session(session_id)
    collected = session.get("collected")
    lookup = session.get("address_lookup")
    if not collected or not lookup:
        return {"error": "Missing collected DLN or lookup result. Run previous stages first."}

    provenance = {
        "collected_by_tool_call_id": collected["collected_by_tool_call_id"],
        "address_lookup_tool_call_id": lookup["lookup_tool_call_id"]
    }

    # Call change tool with DLN and provenance (DO NOT put DLN into chat)
    change_response = call_tool_change_address(
        drivers_license_number=collected["driversLicenseNumber"],
        new_address=new_address,
        provenance=provenance
    )

    # Persist change result
    save_session(session_id, {"last_change": {
        "result": change_response["result"],
        "change_tool_call_id": change_response["tool_call_id"],
        "provenance": change_response["provenance"],
        "changed_at": change_response["timestamp"]
    }})

    # Inform user (no internal IDs)
    return {"user_message": "Your address change has been applied successfully."}


# --- Example run ---
if __name__ == "__main__":
    session_id = "sess_abc"

    # Stage 1: collect DLN
    print(stage1_collect_dln(session_id, "D1234567"))

    # Stage 2: lookup and propose
    print(stage2_lookup_and_propose_change(session_id))

    # Stage 3: user confirms change
    new_address = {"street": "456 Oak Ave", "city": "Newtown", "zip": "67890"}
    print(stage3_apply_change(session_id, new_address))

    # Inspect session state (for debug only; do not leak these contents to chat/speech)
    print("SESSION STATE (internal):", load_session(session_id))

Why this prevents hallucinated IDs and preserves provenance • The authoritative DLN and tool_call IDs are persisted immediately after the original tool call. They are not reintroduced into chat in raw form (speech). • Stage 2 and subsequent tool calls use the persisted values passed as structured context (or used directly in tool inputs). The agent reads an authoritative data block rather than inferring or reconstructing IDs from chat text. • Every tool call returns a new

tool_call_id

which you persist with its result; you then pass the prior IDs as provenance into future calls. This establishes a clear chain of custody so you can always trace which collected value was used by which lookup and which change. Best practices and considerations • Treat session storage as the source of truth and never rely on raw chat for runtime identifiers. • Keep sensitive data out of user-visible chat. Use internal structured context blocks for agents/tools only. • Persist each tool response with its

tool_call_id

and timestamp for traceability and auditing. • If you need the model to reference previous steps in natural language, include a short, non-sensitive summary in chat (e.g., “Previously you provided a license number and we found address X.”) but do not print IDs or full DLN in speech. • If you still see hallucinations, ensure your agent

But what if fields are named differently - if we call verify and get back driversLicenseId and the input is driversLicenseNumber would it have enough input to use this? It currently works when it is single stage so wondering if the lack of other tool call information affects this

Thanks — this is a great follow-up. You’re asking whether mismatched field names (for example a verification tool returns

driversLicenseId

while later tools expect

driversLicenseNumber

) will break the multi-stage flow, and whether missing other tool-call info can cause the LLM to hallucinate or fail to use the right value. Short answer • Field name differences do not have to break the flow if you normalize and persist authoritative values in your session state and pass a well-defined, structured context to downstream agents. Persist the value and any provenance under stable keys your application controls (for example

driversLicenseNumber

), and map tool outputs into those keys when you save them. If you do that, downstream tools/agents will have clean, consistent inputs and will not need to infer names from chat history. Key principles • Persist canonical data shape: immediately convert tool outputs to an application-level schema and store that. Let your app decide canonical field names (e.g.,

driversLicenseNumber

) regardless of what tools return. • Store provenance separately: include original tool fields and tool_call_ids in the persisted record for traceability (e.g.,

original_field_name: "driversLicenseId", original_value: "X", collected_by_tool_call_id: "tool_call_123"

). • Pass structured authoritative context only: when invoking downstream agents or tools, provide the canonical fields from session storage — do not rely on the LLM to map or infer different names from natural language history. • Avoid putting these raw values or provenance IDs in user-facing chat/speech to prevent leakage and keep the LLM from hallucinating them. Practical Python example (extension of prior pattern) Below is a concise example that shows: • A verify tool that returns

driversLicenseId

• Normalization into

driversLicenseNumber

when persisting to session • Passing canonical fields to a lookup tool and including provenance

# continuation of prior pseudo-SDK example
def call_tool_verify_driver_license(raw_input):
    # Simulates a verification tool that returns a different field name
    tool_call_id = "tool_call_" + str(uuid.uuid4())
    # Suppose the tool normalizes/creates an internal id called driversLicenseId
    return {
        "tool": "verifyDriverLicense",
        "result": {"driversLicenseId": "ID-98765", "valid": True},
        "tool_call_id": tool_call_id,
        "timestamp": datetime.utcnow().isoformat() + "Z"
    }

def stage1_verify_and_normalize(session_id, user_input):
    # Tool returns driversLicenseId (different name)
    verify_resp = call_tool_verify_driver_license(user_input)

    # Normalize to your canonical schema before persisting
    canonical_dln = None
    if "driversLicenseNumber" in verify_resp["result"]:
        canonical_dln = verify_resp["result"]["driversLicenseNumber"]
        original_field = "driversLicenseNumber"
        original_value = canonical_dln
    elif "driversLicenseId" in verify_resp["result"]:
        # Map driversLicenseId -> driversLicenseNumber at the application level
        canonical_dln = verify_resp["result"]["driversLicenseId"]
        original_field = "driversLicenseId"
        original_value = canonical_dln
    else:
        return {"error": "No recognizable DL field returned by verify tool."}

    # Persist canonical name + provenance
    save_session(session_id, {
        "collected": {
            "driversLicenseNumber": canonical_dln,
            "collected_at": verify_resp["timestamp"],
            "collected_by_tool_call_id": verify_resp["tool_call_id"],
            "original_field": original_field,
            "original_value": original_value
        }
    })

    # User-facing confirmation (no IDs)
    return {"user_message": "Your license was verified and stored."}

def stage2_lookup_using_canonical(session_id):
    session = load_session(session_id)
    collected = session.get("collected")
    if not collected:
        return {"error": "No collected info"}

    # Use canonical field for downstream lookup and include provenance
    canonical_dln = collected["driversLicenseNumber"]
    provenance_tool_call_id = collected["collected_by_tool_call_id"]

    lookup_resp = call_tool_lookup_address_by_dln(
        drivers_license_number=canonical_dln,
        provenance_tool_call_id=provenance_tool_call_id
    )

    # Persist lookup with its own tool_call_id and link back to the canonical record
    save_session(session_id, {
        "address_lookup": {
            "currentAddress": lookup_resp["result"]["currentAddress"],
            "lookup_tool_call_id": lookup_resp["tool_call_id"],
            "used_provenance_tool_call_id": provenance_tool_call_id,
            "lookup_at": lookup_resp["timestamp"]
        }
    })

    # Present human-friendly summary (no canonical id echoed)
    return {
        "user_message": (
            f"Found address: {lookup_resp['result']['currentAddress']['street']}, "
            f"{lookup_resp['result']['currentAddress']['city']} {lookup_resp['result']['currentAddress']['zip']}. "
            "Would you like to update it?"
        )
    }

Why this handles name mismatches and prevents hallucination • The app performs the mapping at the moment the tool returns data. The canonical key you persist (e.g.,

driversLicenseNumber

) is the one downstream logic uses. • Downstream agents receive only the canonical structured data (and provenance) instead of raw chat history. The LLM does not need to infer field mappings or invent IDs. • You still preserve the original returned field and tool_call_id for auditing and debugging, so you can trace "driversLicenseNumber" back to "driversLicenseId" if required. When might discrepancies still cause problems? • If you only pass chat history (no structured canonical context) to downstream agents, the LLM may see inconsistent names and guess or hallucinate mappings. • If tools or agents expect different canonical names and you don’t normalize before calling them, the call may fail or produce ambiguous outputs. Always normalize in your middleware before invoking downstream tools. Quick checklist to implement now • Normalize tool outputs to your canonical schema immediately on receipt. • Persist canonical fields + provenance (original field names and tool_call_ids). • Feed downstream agents/tools only the canonical structured context (not raw chat). • Keep original tool output stored for traceability, but do not expose it in user speech. If you’d like, I can: • Produce a full runnable Python sample that uses an in-memory session store and simulates each tool end-to-end, or • Show how to add a small mapping layer that automatically converts common tool field names to your canonical names based on a mapping table. Which would be more helpful for you?