Case 03 · Selected Work
Serving an AI assistant's traffic without holding a connection open
Re-architected an AI customer-support assistant's request path so that a conversation is queued and polled rather than held open on a synchronous connection — delivered and merged.
- Role
- Senior Web Developer
- Capabilities
- Applied AI · Architecture · Reliability
- Domain
- Production e-commerce platform with an AI support assistant
- Change
- Synchronous request path → queue with client-side polling
Context
A production e-commerce platform with an AI customer-support assistant serving its customers, alongside the platform's own established business workflows.
Problem and ambiguity
There is a structural mismatch at the centre of every LLM product: a language model turn routinely takes tens of seconds, and everything a web request passes through on its way to that model — gateways, firewalls, proxies — has a timeout budget designed for requests that take milliseconds. A synchronous design therefore has two failure modes at once. The obvious one is that a slow answer can exceed a limit somewhere in the path. The one people miss is that while it runs, it occupies that path, so the next customer waits behind it. Both look identical from the outside: the AI is slow. Neither is a model problem.
My role
I work on the company’s AI customer-support system — knowledge structure, response routing, tool integration, session handling, evaluation, safety controls and human escalation — and I designed and delivered the change to the request path described here. I also maintain the production e-commerce platform around it.
Discovery
The useful question was not “how do we make the model faster” but “what in this path expects a request to finish quickly, and what happens to everyone else while one does not?” Answering that changes the problem from model tuning to connection architecture.
Architecture and trade-offs
Move off synchronous handling. The request is enqueued; the browser polls every few seconds and renders the answer when it is ready. Nothing holds a connection open, so no timeout ceiling in the path governs the conversation, and customers are served concurrently rather than in line behind each other. The cost is a slightly more complex client and a visible waiting state — a trade I would make again, because the alternative fails silently and unfairly.
Implementation and integration
Queue workers on the platform’s existing framework, with client-side polling, and retry and error handling around the model calls. Two further controls came with it: idle sessions expire — a quiet conversation is prompted, and if there is no reply it closes and releases the capacity it was holding — and lighter requests are routed to smaller model tiers instead of every call going to the largest model, which is the simplest inference-cost control that does not degrade the answers that matter.
Production controls
Retry and error handling around the model calls; session lifecycle management; and an evaluation approach that is being designed in rather than bolted on — including correctness when a single reply needs several pieces of information retrieved and combined, and behaviour under adversarial and social-engineering probing. That evaluation is being built, not shipped hardening — I am describing a design, not claiming a result.
Impact
The re-architecture is complete and merged into the platform. No performance figures are published for this work, and none will be — I do not have a production result I can honestly claim.
Reusable learning
When an AI feature is described as slow, measure the path before you touch the model. In my experience the model is usually doing what it always did, and the architecture around it is what changed the user’s experience — and the fix is almost always to stop holding a connection open.
Evidence pending
Nothing from the client's assistant is published — not its knowledge structure, prompts, evaluation set, or any performance figures. The mechanisms described above are generic and are not the client's property; an independent implementation over my own content is a different system, and unrelated to that employer's business.