Artificial Intelligence
Why Human Judgment Is The Ultimate Bottleneck In AI Orchestration ft. Rajeshwari Sah
Share
Overview
TD Editor: You’ve had a fascinating career trajectory, starting with massive-scale payment systems at Visa and evolving into advanced AI orchestration at MAANG. How did your experience architecting high-throughput, fault-tolerant financial pipelines shape the way you approach building reliable, globally deployed AI systems today?
At Visa, beyond moving data, we also moved trust. From this perspective, every failure needed to be recoverable, and the system had to keep running regardless of what broke underneath. You don’t add reliability later in financial infrastructure, because it is supposed to be embedded in the architecture. Such a work approach taught me how to deploy AI systems, as well. When we built the multilingual sales framework across 20+ markets, I applied the same logic: each layer, that is, language, retrieval, inference, needs to be independently observable and recoverable. The question I always ask is: if this breaks at 2 am in a market I'm not monitoring, what does the system do on its own? Answering that well is what separates a proof of concept from a production system.
TD Editor: In your current role, you have architected a multilingual framework that supports over 20 global markets. You’ve specifically mentioned engineering 'localisation-aware RAG pipelines'. Can you explain how you ensure an AI isn’t just translating words accurately, but actually maintaining cultural consistency and linguistic precision across different regions?
Translation accuracy is the easier problem. The harder one is the intent preservation across different cultural contexts. If we look at the issue deeper, language is a surface layer of our localisation. Underneath it are assumptions about how people ask questions, what counts as a complete answer, and what tone feels appropriate. Those elements vary region by region, and a response that feels direct and helpful in one market can feel odd in another. This means the retrieval pipeline must also surface content validated within the user’s cultural context, and that the ranking model needs to weigh cultural fit alongside relevance. To evaluate the system and its performance, you need in-region feedback loops to catch what automated metrics miss. Hence, the main principle is that localisation is a data issue before it’s a modelling issue. If your training signals aren’t culturally relevant, no model sophistication will compensate.
TD Editor: During your time at Lifio.ai, you orchestrated AI agents to design clinical trial protocols. Healthcare is a notoriously strict industry. How did you balance the need for rapid trial execution with the absolute necessity of regulatory compliance and factual accuracy?
I push back on the “speed versus compliance” framing because, in reality, they’re not the opposite. Compliance failures are the delays: protocol amendments, IRB rejections, and FDA queries set timelines back by months. To accelerate, you need to build rigorously from the ground up.
We structured it so that we treated compliance as a generation constraint. The model couldn’t produce a protocol section without grounding it in retrieved regulatory source materials. We applied the principle: generate only within the space of what’s already validated. Factual accuracy followed the same logic. We fine-tuned on historical protocol data to build a strong prior for well-formed protocol language. Only after that did we add human expert review at high-stakes points, such as eligibility criteria and endpoint definitions, rather than trying to automate those decisions fully. The goal was never to remove the human but to make their review faster and more focused.
TD Editor: Let’s talk about pure scale. At Visa, you directed a massive data-localisation initiative to comply with RBI mandates, which required moving 5 TB of streaming transactional data every single day. What is the hardest part about executing an infrastructure shift of that magnitude ahead of strict regulatory deadlines?
The harder part turned out to be coordination under pressure across teams with varying definitions of “done”. Imagine you're migrating 5 TB of streaming transactional data daily, with engineering, compliance, legal, and business involved. Each of them has a different risk tolerance and a different interpretation of the deadline. Engineering may say migration is complete when data is flowing, while compliance says it is complete when the audit trail is verified. If you don't align those definitions from the onset, you discover at the end that everyone had their own interpretation of the “problem”.
Regulatory deadlines also brutally expose technical debt. Every hidden dependency, every old system nobody dared to touch, every messy data format - it all comes out the moment a hard deadline forces you to follow the data from start to finish. The Visa migration was as much a discovery process as an engineering one. We found things that surprised people who'd been there for many years. What helped us to overcome this challenge was treating program management with ownership and an honest distinction between what was done and what was “almost done”.
TD Editor: The tech industry is rapidly shifting from simple LLM chatbots to complex, multi-agent systems that reason and execute tasks autonomously. But as these systems get more complex, a massive challenge has emerged: how do we actually evaluate them? From an industry perspective, how do we measure quality, safety, and alignment when AI operates without constant human oversight?
The industry is still expanding in this direction. The main challenge is that traditional evaluation assumes a single model, a single output, and a single reviewer. However, in a multi-agent system, you have a long chain of decisions where each agent’s output becomes the next agent’s input. Because of this, an error at step two might not surface until step seven, and by then, it’s hard to trace back. The shift needed is from output evaluation to interaction evaluation. You have to instrument the handoffs between agents, the reasoning at each decision point, and the confidence signals passed along the chain. Think of it like tracking a package at every step of delivery, not just checking whether it arrived. Alignment gets tricky because a multi-agent system can produce misaligned outcomes even when no individual agent was misaligned. The problem arises from the coordination itself; evaluation must be systemic and defined at the level of the entire workflow.
Human oversight also doesn’t go away, but it gets restructured. The human's role shifts from reviewing every output to designing frameworks that catch failures early. It also shifts towards reviewing cases in which the system's own confidence signals indicate that something is wrong. That's where the field is heading: AI systems that know what they don't know, and escalate appropriately.
TD Editor: The recent rise of highly autonomous, agentic coding systems like Anthropic's Claude Code is fundamentally changing what it means to build software. We are moving toward a reality where AI can autonomously navigate complex codebases, rewrite code across an entire project and automatically test and launch the updates without a human in the loop. As someone who designs production multi-agent orchestration frameworks, how do you see the day-to-day role of the AI/ML engineer changing as the field shifts away from manually writing syntax and heavily toward 'continuous orchestration' and architectural design?
We’re at an inflexion point as significant as the shift from on-premise to cloud. The bottleneck now is the engineer who designs a system of agents that solves problems reliably and at scale. That means they need to set their mindset towards systems thinking, so less about “how do I implement this function” and more about “how do I structure this workflow so that failures are contained, outputs are auditable, and the system improves”. It's closer in spirit to distributed systems architecture than traditional ML modelling.
TD Editor: Building directly on that shift, the barrier to simply 'writing code' has effectively dropped to zero, yet the AI job market remains incredibly competitive and saturated. For a young computer scientist looking to break into the ML engineering field today, what specific skills, foundational knowledge, or architectural mindsets should they focus on to truly beat the crowd and provide value that an autonomous agent cannot?
Don’t compete at the layer being automated. Writing correct code is table stakes, and it will keep getting more automated. Instead, what isn’t being automated is human judgment: knowing which architecture fits the problem, asking the correct evaluation questions, and recognising when a system that looks like it's working is accumulating risk. The foundations I'd prioritise: how systems fail (distributed systems, fault tolerance, observability); how models behave rather than just how to call them; and thinking in workflows, not in individual functions. The engineers who define this next era won't be the ones who use AI tools fastest. They'll be the ones who understand enough to know when not to trust them.
Mon, Dec 1, 2025
Enjoyed what you've read so far? Great news - there's more to explore!
Stay up to date with the latest news, a vast collection of tech articles including introductory guides, product reviews, trends and more, thought-provoking interviews, hottest AI blogs and entertaining tech memes.
Plus, get access to branded insights such as informative white papers, intriguing case studies, in-depth reports, enlightening videos and exciting events and webinars from industry-leading global brands.
Dive into TechDogs' treasure trove today and Know Your World of technology!
Disclaimer - Reference to any specific product, software or entity does not constitute an endorsement or recommendation by TechDogs nor should any data or content published be relied upon. The views expressed by TechDogs' members and guests are their own and their appearance on our site does not imply an endorsement of them or any entity they represent. Views and opinions expressed by TechDogs' Authors are those of the Authors and do not necessarily reflect the view of TechDogs or any of its officials. While we aim to provide valuable and helpful information, some content on TechDogs' site may not have been thoroughly reviewed for every detail or aspect. We encourage users to verify any information independently where necessary.
Loading comments...
