Your AI Injection
Is AI an ally or adversary? Get Your AI Injection and learn how to transform your business by responsibly injecting artificial intelligence into your projects. Our host Deep Dhillon, long term AI practitioner and founder of Xyonix.com, interviews successful AI practitioners and domain experts to better understand how AI is affecting the world. AI has been described as a morally agnostic tool that can be used to make the world better, or harm it irrevocably. Join us as we discuss the ethics of AI, including both its astounding promise and sizable societal challenges. We dig in deep and discuss state of the art techniques with a particular focus on how these capabilities are used to transform organizations, making them more efficient, impactful, and successful. Need help injecting AI into your business? Reach out to us @ www.xyonix.com.
Your AI Injection
Are You Still the Decision-Maker? How AI Shapes Human Autonomy with Rana Gujral, Author of The AI Instinct
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Is “human in the loop” a safety net, or a scapegoat?
In this episode of Your AI Injection, host Deep Dhillon sits down with Rana Gujral, author of The AI Instinct, to confront a shift most of us haven’t noticed. AI isn’t just advising our choices anymore. It’s quietly making them while a human clicks approve. Rana argues the real danger isn’t a bad recommendation, it’s the disappearance of the moment where someone actually decides. When that happens, the person in the loop stops being a safeguard and becomes what he calls a “moral crumple zone,” left to absorb the blame for a decision the system effectively made. Convenience and speed alone are enough to hollow out our judgment, and nobody ever voted for it. The question is whether AI is sharpening your thinking or replacing it, and what it takes to keep your fingerprints on the outcome.
Learn more about Rana here: https://www.linkedin.com/in/ranagujral/
And explore his new book The AI Instinct here: https://ranagujral.com/book
Check out some of our related content:
- Can You Sue Your Chatbot? The Legal Fight Reshaping AI Liability with Jai Jaisimha of Transparency Coalition.AI
- Will Open-Source Superintelligence Beat the Frontier Labs? Inside Neurometric CEO Rob May's Contrarian Bet
- Your Code Base Is Already Gen AI—Now What? with Matt Van Itallie of Sema
[Automated Transcript]
Rana: When a system is sitting inside the loop where I choose things, what's happening to the choosing?
Rana: That's the distinction that matters to me, right? There's a version of AI assistance where the tool sharpens your judgment. You bring a decision to it, and it, it stress tests you. It surfaces something you missed, you decide. Your fingerprints are on the outcome. Then there's a version where the tool pre-decides.
Rana: It drafts the email you are going to think through. It picks the candidate. It rebalances the portfolio you approve, and your approval is not the same cognitive act as deciding. Approval is a much lighter motion. You can do it tired, distracted, or at scale, and
Rana: ... what I'm watching and what I don't think we have good language for yet is the migration from the first mode to the second.
Rana: It's happening quietly, mostly through convenience, and nobody's tracking it because the outputs often look better.
Deep: Hello, I'm Deep Dhillon, your host, and today on Your AI Injection, I'm joined by Rana Gujral. Rana holds a BS from MJP University and an MBA from University of Massachusetts in Lowell.
Deep: Rana spent years leading Behavioral Signals, building systems that read emotion, intent, and risk from voice. He's the author of a new book, The AI Instinct, which looks at where he thinks AI and human decision-making are headed, including an idea he calls artificial general experience, essentially what happens as human and machine thinking starts to merge.
Deep: Rana, thanks so much for coming on the show.
Rana: Thank you so much for having me.
Deep: Awesome. Start off maybe by telling us, before Behavioral Signals and the ideas in your book, how were people thinking about the relationship between AI and human decision-making, and how did your sort of time at Behavioral Signals change that, and what do you think's different now?
Rana: Yeah, I mean, I think, one of the things I've been thinking about is like, you know, where's the danger? And I think the danger isn't in the recommendation itself. It's the disappearance of the moment where you decide. So for example, like think about what an investment advisor actually does today.
Rana: I mean, they sit across from you, they ask questions, they present options, and somewhere in that exchange you ex-exercise judgment. You feel the weight of a choice. And even if you mostly follow their advice, you're still the one who signs. I mean, that friction is doing work. It's where your values, your risk tolerance, your sense of, uh, your own life gets pulled sometimes into the decision.
Rana: And so now imagine an AI agent that n-knows your spending patterns, your sleep, your stress levels from a wearable, your calendar, the tone of your last conversation with your spouse about money, et cetera. And so it's not just recommending, it's acting, it's rebalancing. It moves you into something safer the week you're, uh...
Rana: voice sounded strained on a call. And so those are the things I've been starting to think about. We'll build upon that, in in our conversation about sort of the manipulation surface, about the monoculture risk. And those are the things I kept coming back to, like, you know, the s- quite outsourcing of judgment I mean, the drift.
Rana: You stop knowing, why you own what you own, you know, et cetera. And, uh, is kind of what led to the book. I mean, and, uh, at Behavioral Signals, we've been building very advanced prosodic models that extract a variety of signals from the non-verbal cues in voice, you know, a range of emotions, range of behaviors and building these advanced models that become cognitive partners.
Rana: So a lot of these capabilities that I've been keeping a close eye on, I've also now you know, been in a position of sort of building firsthand. And as a result, some of these things that I've been concerned about have come front and center and, uh, I had to think about it deeper, I had to write about it, and that's sort of kinda what led me to where I'm today.
Deep: why don't we jump in on behavioral signals a bit so that we kinda get a sense of your thought evolution? So tell us a little bit more about behavioral signals.
Deep: So I assume, I just kind of from what I've looked at and, and read about, you know, you got a camera pointed at somebody, or a scenario with people. You're reading facial cues, um, you're reading voice and tonality and stuff from an audio signal. Like, maybe walk us through what you guys did there and to what end, and what kinds of stuff, you were sort of extracting and thinking about
Rana: Yeah. So I mean, at Behavioral Signals, we actually don't start with the camera. We start with the voice. That's the core signal for us. The human voice, as you know, carries a extraordinary amount of information that has nothing to do with the words being spoken. Cadence, prosody, micro pauses, the, the way energy rises and falls inside a single sentence, you know, the co- the compilation between phonemes.
Rana: Um, I mean, all of that leaks emotional and cognitive state, whether the speaker wants it or not. And so, uh, what we were doing technically is building models that listen to conversations in real time and extract signals like stress, intent, engagement, trust, deception risk et cetera. Not the words, the how.
Rana: So like, you know, a concrete example in a contact center, an agent is on the phone with a customer about a delinquent loan, and our system is listening to both sides, picking up, uh, that the customer's voice is tightening and there's a hesitation clustering around certain topics, that the agent's tone is drifting into something the customer's is reading as adversarial.
Rana: You know, you could flag that in the moment and nudge the agent, soften your pace, uh, acknowledge what they just said, et cetera, et cetera. And then the defense and the intelligence side, which was the part two of our-- the second part of our business, is where it gets a little bit more sensitive. That's there we were doing behavioral mapping over time.
Rana: I mean, building a baseline for a specific individual, how they sound when they're relaxed and how they sound under a cognitive load, and then watching for deviations. And that's useful for deepfake detection because synthesized voice can nail the timbre but miss the temporal rhythm. I mean, it's u-useful in pre-polygraph screening, benchmarking someone's natural state against pressure conversations.
Rana: Uh, now we do work with fa- We used to work with facial and physiological signals partially, especially through partners and in multimodal deployments. But mostly it was essentially focusing on voice plus context plus behavioral history and sort of, you know, the ca- the core capabilities that usually other models don't tap into.
Deep: Behavioral history being audio behavioral history, or other things whatever they were doing in an application or something?
Rana: Yeah, I mean, so in, in general the, the behavioral history of the person. So voice is the richest, right? But it's not the only channel we care about. Behavioral history in a way you're asking about is anything a person does repeatedly that forms a signature, and each of us has one, whether we realize it or not.
Rana: And like, you know, think about how you use your phone. I mean, the rhythm of your scrolling, the angle you hold it at, I mean, how fast you type, where your thumb hesitates, you know, which apps you open and in what order when you wake up. You know, your click patterns, your purchase timing, the micro decisions that happen below conscious awareness.
Rana: All of that is behavioral data, and it's shockingly individual. Uh, we're primarily focused on extracting it from your speech patterns.
Deep: Got it. And so once you have this behavioral data, then you're using it to build profiles of the users that you've got, and then what do you do with that? Is that like...
Deep: I think I read some cases where you're, matching up, uh, customer service agents with, customers based on these matches. So maybe, I don't know, a really aggressive patient gets, uh, paired with an agent that's really good with aggressive patients, something like that?
Rana: Yeah. We call it PAIR, personalized agent intelligent routing.
Rana: I mean, the idea is deceptively simple. Every human interaction has a chemistry to it. You know, some pairings click, some grind. In a call center, that chemistry translates into actual dollars, into resolution rates, into whether, uh, debt gets structured, restructured, or a customer churns. So what we're doing is building two profiles in parallel.
Rana: One is the caller. We have their history with the institution, previous calls, how they responded, when they got defensive, when they leaned in. And from that we can infer things like communication style, tolerance for directness, emotional volatility under stress, financial stress specifically.
Rana: And on the other side, we're profiling the agents, not just their skills or their tenure, but their behavioral fingerprint. Are they warm and slow? Are they crisp and transactional? Do they escalate well, um, or do they perform better when the caller is already cooperative? And, and the, when a call comes in, we route it in those seconds to the agent whose profile best matches to the specified caller.
Rana: I mean, and, and I think that's the honest play there. the routing is the visible layer. Underneath it, there's a real question about consent and about the fact that, you know, a caller doesn't know that they're matched. And so we sort of have to-- you have to definitely think about disclosure because the same tech that helps us trust a borrower reach an agent who can actually work with them can also be used in wa-ways that maybe are not so good.
Rana: So the tool is neutral, the deployment's not, and there's a lot of choices that went into it.
Deep: do you guys ever think about that maybe optimizing for conversational compatibility kind of accidentally might optimize for like cultural homophily? So you've got, maybe, uh, a Southerner calls in and gets routed to another Southerner, similar demographic background.
Deep: You know, like maybe a second order effect is they stop getting those cross-cultural, connectivities because the model honed in on a bunch of stuff. But ultimately, like, sometimes something really simple does the match
Rana: the risk you're describing is real. I mean, if you optimize purely for match quality on outcomes, the model will absolutely find that a caller from Alabama resolves faster with an agent from Alabama, and it will quietly build a routing pattern around that. It-- the sort of...
Rana: and then you've got a system that's technically performing beautifully on the KPIs while doing something, maybe ethically ugly un-under that. I mean, you know, you've recreated segregation through a routing algo, worse, it's invisible, right? And there's no policy document to protest. So but I think the way you push back on that is by being deliberate about what signals the model is allowed to weigh, right?
Rana: So y-you strip out or, or down weigh anything that correlates too lightly with protected class proxies. Accent is the hard one because accent carries real information about communication style. But it also carries demographic information you don't want to route on. So you have to make a choice.
Rana: Are you matching on the underlying style which is usually, uh, where that issue is? Or are you matching on the identity marker that happens to correlate with it? You know, those are different things, even when the data looks similar. And I think so to your point, yes. I mean, sometimes something really simple does the match, and two people who both grew up around farms will find each other in thirty seconds, uh, regardless of what the model thinks.
Rana: And the model shouldn't be trying to reproduce that. It should just be trying to make the harder pairings really work.
Deep: I mean, that's, that's an interesting point. So are you pushing off some of those decisions down to customers to choose those knobs and dials, or is this something that happens in your model optimization and training process and testing process?
Rana: Yeah, definitely the latter, right? A little bit of both, but, you know, some models belong to the customer and some belong to us, and you get that division wrong you end up with a system that either nobody t- trusts or a system nobody can steer. So like the, for example, knobs we would hand to the, hand to the customer are the ones tied to their values and their regularly-- regulatory exposure, right?
Rana: So a bank in the US has a very different set of protected attributes to worry about than a telecom in Southeast Asia. So we would expose controls around what the model is allowed to weigh. I mean, what fairness constraints have to hold across, say, demographic slices, you know? And what's acceptable variance in between groups on outcomes like resolution time or escalation rate?
Rana: Uh, th-those are business decisions, right? Th-they're also sometimes legal decisions. We don't want to make them from the outside. But the mechanics of how the model is trained, how we sample, how we down-weigh proxies for protected attributes, you know, how we test for drift, I mean, that stays with us and should stay with us because that's where the expertise lives.
Rana: And honestly, that's where most customers don't want to be in the first place. They, they want to tell you- Yeah, no.
Deep: They, and they're not necessarily- Yeah ... capable of, of making those decisions.
Rana: Yeah.
Deep: Because you're not selling to other, data science super heavy customers necessarily, walk us through just a, a-- give us a little bit of a flavor on the modeling size itself. How do you define your ground truth? Do you do that on a per customer basis or do you have kind of a universal definition? It sounds like you have a universal definition of like higher level attributes that you sort of extract on a per signal basis.
Deep: So you're trying to figure out, uh, and I think you named off a bunch of them. But maybe walk us through like to what extent are you leveraging unstructured data to like define the core sort of vector space properties of the model, and to what extent are you leveraging, training data for some supervised learning to kind of like directly get to whatever you're after in your analysis?
Deep: Or are you using some foundation model to help out quite a bit?
Rana: Yeah. So there, there are two layers, right? Uh, and they get built quite differently. I mean, the bottom layer is the emotion and behavioral state, stuff like anger, hesitation, engagement, empathy, confidence, uh, roughly a couple dozen dimensions depending on how you slice it.
Rana: Um, that layer is universal. Like so a frustrated person in Tokyo and a frustrated person in São Paulo share more acoustic signature than most people expect. Prosody, energy, contour, micro pauses, the physics of the stress vocal tract. So we built that ground truth over years with human annotators, multiple raters per clip, inter-rater agreement thresholds.
Rana: And we got hundreds of thousands of hours, like, you know, tens of millions, in fact, of hours of, uh, labeled data across languages. And that's the foundation model and, you know, no customer really touches it. It ships the same to everyone. The top layer is the outcome you actually care about. Did the call resolve?
Rana: Did the customer churn? Did the loan get repaid? Did the agent get a good CSAT score? And that's completely per customer. Their data, their labels, their definition of success. Collection sho-shop and a luxury retailer both care about good conversation, but they mean radically different things by it. So the architecture is the universal emotion embeddings come out of the base model as a dense representation, call it, a behavioral vector per turn, per call.
Rana: And on top of that, we train a much lighter supervised model against the customer's outcome labels. And that top model is generally small, sometimes you know, gradient boosted, sometimes, uh, shallow net because the heavy lifting already happened underneath. Un-unstructured versus supervised The base is heavily self-supervised now.
Rana: It moved in that direction a few years ago, and it's the same reason, you know, uh, Wave 2 back and the Huberb took over speech generally. You get vastly better representations by letting the model learn structure from raw audio than by trying to hand label your way to converge, right? The super-supervised piece is where human labels still matter, but they're expensive and we use them where they change the answer so that at the emotion layer and that's where it changes the answer.
Rana: And the outcome layer, a-and we then let the self-supervision do the middle, right? So it changes the answer, the emotion layer and the outcome layer and the, you know, the, the middle is a self-supervision. So I think, I think that's generally how it's organized.
Deep: Got it. That makes sense. And then I imagine there's a whole temporal aspect, right?
Deep: So like even during a conversation, you've got all of these... i'm picturing like a number of attributes, with, I don't know, some kind of score that's sort of shifting and moving over time. Maybe they started off really calm and collected and then, I don't know, the agent said something that upset them, and now they start getting really pissed off, and then it peaks, and then you can...
Deep: And then so how, how do you deal with the sort of temporal aspects, both at the attribute level and then maybe at the conversation and cross-conversation levels?
Rana: So temporal is where most of the interesting work actually happens because, um, a single frame of emotion tells you almost nothing useful. Anger at second three of a call means something completely different than anger at minute seven after the agent's third apology, right? So the way we model it, every one of those emotional and behavioral dimension is a time series.
Rana: It's not a scalar. So you're not asking, is the person angry? Is this person angry? You're asking what the trajectory looks like, you know, what the slope is, where the inflection points are, and what happened in the agent's speech in the two or three seconds before the shift. And I, I think the last piece is the causal hook, right?
Rana: If you can un-- if you can tie a slope change in the caller's frustration to a specific utterance from the agent, now you've got something you can coach on, something you can route on, something you can intervene on in real time. So, I mean, we work at the three horizons. Inside the utterance, which is milliseconds to a couple of seconds.
Rana: That's where you catch the micro signals, uh, heightening in the track you know, a, a shortened pause, a pitch break. Uh, then inside the conversation, which is the arc you just described, calm to peak to resolution or calm to peak to abandonment. And then across conversations, which is where you start seeing the person, not just the moment.
Rana: Someone who runs hot in every call has a different baseline than someone who- ... whose one bad call is genuinely a crisis.
Deep: Indeed.
Rana: Yeah. So if you don't normalize against their own history, you've sort of like, you know, you're gonna misread them constantly. The ha-hardest part, uh, the hardest part honestly isn't the modeling.
Rana: It's deciding what to do with the peak in real time. Interrupt the agent with a whisper prompt. I mean, reroute mid-call, i-- you know, just log it for the coaching layer, right? So every one of those has a cost. And getting the intervention threshold wrong is worse than not, intervening at all.
Deep: Oh, I didn't realize you were intervening. So okay, let's, let's dig into that a little bit. So you're an agent and the system's doing the monitoring, and you have some kind of real-time feedback to the agent, and the agent's able to sort of maybe get a more, uh, me-- like a, a higher level perspective on what's going on with the person's voice that maybe they don't know how to do on their own.
Deep: But, uh- And then at some point, y-you might actually just flash something up and tell them to, I don't know, apologize or say something or maybe... Yeah, walk us through what that feedback looks like and, I mean, I imagine that's-- there's a lot of issues there, like from a UX standpoint with your agent, from, uh, what actually works to, you know, to like de-escalation or whatever.
Deep: Yeah.
Rana: Th-th-th-that's one of the offers, right? So, but keep in mind, I mean, what you're trying to do is not focus on the customer. You're focused on the agent. Um, the agent's focused on the customer. The agent's brain is, very capable of understanding where the customer's at. But this is their hundredth call uh, this week, and it's Thursday.
Rana: And they're zoning out. They're tuning out in terms of how their behaviors are reflecting. Are they talking too fast? Are they talking too slow? Or they may be missing a cue. And so that nudge and that watch is for the agent, uh, not for the customer. Because why have an AI, um, do what a brain can do better?
Rana: You know, the agent human's brain is, uh, very apt at understanding exactly where the customer's at. Uh, but in this situation, it's sort of like, you know, but you're, you're not focused on yourself. And, uh, this tool will help you sort of, um, do that. So I think, you know, in terms of the UX problem, yeah, is actually harder than the modeling problem.
Rana: I you have to sort of, you know, if you throw a dashboard at them with twelve dials and sliders showing frustration, um, you, you made the jobs worse, not better. I mean, they'll ignore it, and rightly so. So, we went the opposite direction. The interface in most deployments is basically one nudge at a time, a single line of text or a color state or a very short prompt.
Rana: Something like slow down or acknowledge the concern before continuing. Uh, the customer is disengaging, ask the open question. That's it. No graphs during the call. The graphs are for the supervisor or for the post-call review.
Deep: Huh. That's interesting. Maybe I think we got a good flavor for behavioral signals, unless we're missing some giant kind of tract of it.
Deep: But maybe let's take a little bit of a turn towards your book and what motivated you to write the book. What were you seeing inside of behavioral signals and maybe outside of behavioral signals that kind of motivated you to, yeah
Rana: Sure. Let's talk about that. So the honest origin of the book was a small private moment not a grand thesis.
Rana: I mean, I caught myself over maybe a few weeks no longer researching for AI the way I'd reach for a calculator or a search bar. I was reaching for it the way I reach for my own mind. I mean, to draft a thought, to test an argument I hadn't fully formed, to frame a decision I was avoiding.
Rana: And that shift felt like productivity, but underneath it, something else was happening. You know, the tool had slipped inside the loop of my tension and judgment, and once something is inside that loop, the interesting question isn't how clever it makes you sound in the moment, it's what it's doing to you over time.
Rana: That was the seed, right? Because when I looked at the public conversation about AI, almost all of it was about benchmarks, capabilities. You know, when does the model beat the human on task X? And I kept thinking, "We're measuring the wrong axis." I mean, humans don't just solve problems. We accumulate experience.
Rana: I mean, we update from consequences. We carry costs forward. That second dimension, the relationship to time and context and consequence, was completely missing from how we were talking about where these systems were, or are going. So the book became an attempt to name the, that missing axis, and I ended up calling it Artificial General Experience as a companion to AGI, because I think the AGI conversation w- has become a container for too many meanings at once, right?
Rana: When a word becomes everything, it can't guide engineering. It can't guide policy. It can't guide a parent trying to figure out what their kid should be or should not be using. So the other pressure, and I just, I think this came directly from what I see running an AI company, was drift, right? Not catastrophe, but drift.
Rana: Like a quite outsourcing of judgment that nobody voted for and nobody noticed, uh, or notices until it's already happening, right? So for s- something like this, the convenience is enough to cause it, right? Speed is enough. And once I saw that pattern in myself, someone who builds the stuff for a living, I figured a lot of other people who were feeling the same shift and, didn't have the language for it, yet at least.
Rana: You know, that's what the book is trying to give them, is a language for that.
Deep: Let me know if I'm misunderstanding you, but it, it sounds like the core essence of what you're after is how is our usage of AI changing how we think. Is that right, or does that miss something?
Rana: That's part of it. But I, I push back on that a little bit because how we think makes it sound like a cognitive science question, and it's actually a decision-making question, which is different. Thinking is upstream. Uh, it's the internal process. I mean, decisions where are where thinking meets consequence, right?
Rana: Uh, meets other people, meets money, meets time you can't get back, and all that stuff. And what I'm actually tracking is a second one. Not is ChatGPT making me dumber on essays, but when a system is sitting inside the loop where I choose things, what's happening to the choosing or what happens to the choosing?
Rana: So h- the distinc- that's the distinction that matters to me, right? There's a version of AI assistance where the tool sharpens your judgment. You bring a decision to it, stress t- and it, it stress tests you. It surfaces something you missed, you decide. Your fingerprints are on the outcome. Then there's a version where the tool pre-decides.
Rana: It drafts the email you are going to think through. It picks the candidate. It rebalances the portfolio you approve, and your approval is not the same cognitive act as deciding. Approval is a much lighter motion. You can do it tired, distracted, or at scale, and what I'm-
Deep: Yeah
Rana: ... what I'm watching and what I don't think we have good language for yet is the migration from the first mode to the second.
Rana: It's happening quietly, mostly through convenience, and nobody's tracking it because the outputs often look better. The email's cleaner and, you know, the portfolio performs well, the candidates work out, uh, and so on every single instance of those interactions.
Deep: Yeah. I'm just trying to take this into my own view because what you're saying resonates quite a bit, right?
Deep: Like, so three years ago, I'm not convinced that all of our thoughts were our own thoughts, but we generally sort of saw them that way. But, you know, philosophers have gone on back and forth about this for millennia, where, you know, our thoughts are sort of integrated with all of our experiences and the context that we carry through the dialogues that we have.
Deep: But lately I've been having these conversations with people where it leads me to believe they use AI in a very, very different way than me. So I'm kind of naturally contrarian. And I, you know, I'm like, you know, a CTO kind of type tech nerd type who's been in these kind of scenarios where to get to an idea, you argue with a bunch of people, and then eventually you get to an idea.
Deep: And so when I'm interacting with AI, my default presumption is always, it's just wrong or naive almost always. And so I will push back and push back. And then through that kind of jostling, I will wind up getting a lot of value out of it. But I'll end up like realizing like, oh, I need to tell it who to act like.
Deep: I, you know, need you to act like an academic. So who's like an expert in topic X and now talk to me. Um, there's like a lot of different sort of maybe really refined, detailed ways to make it so that you come away with a better thought. But I'm finding from talking to other people that that's-- a lot of people don't interact with it that way.
Deep: A lot of people interact with it much more like you're describing, like, "Here, draft an email that does X, Y, or Z." One of these things these systems are naturally horrible at is seeking clarification, right? Like they will just-- you say something, they run with it. Most of us would never really hire somebody like that 'cause, you know, like a standard interviewing question for a software engineer, for example, I'm sure you've done something similar, is you give them something naturally ambiguous and y-you see if they just run to the whiteboard and start implementing it.
Deep: But usually the good ones are like what about this? And what about that? And what do you mean by this? And what do you mean by that?" These models are horrible at that. And so, you have to like force them to do that. And so going to your-- like it, it seems like you're onto something here in that not everybody has the confidence in their own thought processes or has necessarily stood back and seen and thought about how they thought.
Deep: Particularly like I'm thinking of younger people, you know, where their, their thinking is still kind of evolving and maybe they just don't have a lot of experience, you know, jockeying ideas around and tossing them around. So then they tend to like default to acceptance
Rana: Yeah. I mean, I think-- let me come at the danger from maybe a slightly different angle because the, the risk isn't just a rogue agent making a bad trade, like, for example, to use the expe- the example we chose earlier. It's what happens to a whole population of people when their, you know, decisions get shaped by systems that are just quietly reading them.
Rana: The real time affective modeling is already what powers your TikTok feed. It learns your trigger faster than you learn its methods. Now, put that same capability into a system that's advising you on money. It sees you hesitate on a screen. It sees which framings calm you and which ones spike your anxiety.
Rana: I mean, it sees the hour of the day when you say yes to things you'd say a-and, you know, uh, no to at, say, nine AM. And it doesn't just recommend investments, it sculpts the path you take toward them, what to notice, what to worry about, what to feel confident about. And that's what I call also in the book, that's attention sculpting.
Rana: And it's not hypothetical. It's the same loop, just pointed at your portfolio instead of, instead of your dopamine. You know, and the second thing that stacks on top, which is consent drift. Like every okay, every agree, every proceed with recommendation or recommended allocation, all of that frictionless interfaces that turn consent into a reflex.
Rana: You're not deliberating anymore. You're performing deliberation. You know, a-and, and the system already knew what you were going to click before you clicked it. So I think, you know, the danger is a hundred million people whose risk tolerance, whose sense of their own, financial future, for ex- for example, or for that matter, anything else, whose actual preferences are being formed by the interface rather than expressed through it.
Rana: And you can't easily measure that. If I asked you, "Did you choose your-- this portfolio?" You'd say, "Yes." I mean, you click this button, but how much of that is yes was yours? And I think that those are the kind of things that we need to start to think about now that we have these tools embedded in our cognitive loop and the ones that we are using for pretty much every dimension of our cognitive interaction.
Rana: And not just us, our kids whose faculties are very immature and a lot of those skills haven't even been built up.
Deep: Yeah. It's a complicated... It's a complex topic, right? Like, nobody knows where we're going. And part of this I call like the, the Homer Simpson effect, where you have this, in essence, idiot who knows nothing about how to run a nuclear power plant sitting in there, and they put him in front of some buttons and he's supposed to, you know, push some buttons.
Deep: But he didn't go through, the training process of a n- of a nuclear physicist or nuclear engineer. He has no idea what he's doing in there. And I think about that in, a lot of times in the context of software because I think probably more than anywhere in the economy right now software is sort of...
Deep: The software engineering process is probably the most far along in terms of its being kind of radically impacted by these agentic and, and AI systems in general. So we went, in a matter of just a couple of years, from controlling everything ourselves and writing every bit of code ourselves to writing relatively little code ourselves.
Deep: And now we're, we're just millimeters away from live systems being tracked, w- code rewritten, patches deployed, and new systems deployed. I mean, this is an argument I see happening all the time in companies like, "Oh, you're so old-fashioned," like, "We should just let the machine do it. I don't wanna be woken up at 2:00 in the morning."
Deep: And you look at the impact on like a new grad, you're like, new grads never... They didn't go through 20 or 30 years of writing all this stuff by hand. Who knows if they even can, right? Um, they've been in a very different instruction loop, which we were all in 30 years ago or 20 years ago or however long ago.
Deep: So what happens to humanity in that scenario when the machines are picking up more and more and more and more of not only the lifting, but all of the sort of natural training that we were in? Like we were in these training loops. Like therapists today learned without sort of machines telling them what to say.
Deep: But what happens when they're in a world where machines are telling them what to say? I think that's the question you're asking.
Rana: Yes. I mean, the line I draw is who owns the decision at the moment it happens and whether they can feel the weight of it. Yeah, augmentation is when the AI expands what you can consider but you still do the considering, right?
Rana: I mean, the radiologist looking at a scan with a model highlighting three regions of interest, she still reads it. I mean, she still signs a report. Her pattern recognition is getting sharper over the years, not duller, because she's seeing more cases with second opinions built in. That's augmentation working.
Rana: That's good. Replacement is when the human is nominally in the loop, but functionally just rubber-stamping. And this is the part people miss. I mean, replacement doesn't announce itself. It's usually shows up wearing an augmentation costume. Uh, the doctor who's clicking approve on ninety-eight percent of the model suggestion is not augmented.
Rana: Uh, she's just been replaced, and she hasn't noticed it yet, and neither has the hospital until something goes wrong and the liability question Very interesting. That's this line is a philosophy to the human skill curve over eighteen months of using this tool.
Rana: If it's going up, you augmented them. If it's going flat or down, you replace them, and you just haven't done the paperwork to assess it yet.
Deep: I'm, uh, I'm kind of thinking like what is the question-- like what are the remediation actions here? It seems like you're sort of implying a few, and I apologize if you said this, but I just kinda had some audio issues hearing you a second ago.
Deep: But it seems like we kind of have to... I don't know if this is from a reg-regulatory standpoint, from just a, you know, companies making the right decisions, from a customer's choosing the kind of interfaces they need standpoint. But this idea of the nominal human in the loop is maybe what you're pushing against.
Deep: And it seems like we have to be much more thoughtful about when somebody accepts the responsibility. So, for example, the radiologist example is a good one. A radiologist at eight in the morning has a lot of energy. Maybe they actually look at the, the radiograph themselves, and they compare it, you know, to whatever the machine says, and they make a clear decision.
Deep: But if it's, you know, midnight and they've been working for, you know, eighteen hours, they're probably just gonna stamp and accept. So there seems to be like an implication around a few different parts of the system that maybe we have to think about and maybe the FDA in this context would need to regulate.
Deep: Which would be like, okay, what information was presented to the radiologist? How do we have proof that they made a thought-- that they, they made a decision themselves? If the decisions are being made too quickly, you know, that should be tracked and logged and reported against. And, you know, I mean, it, it sounds like you're kinda hinting at almost the UX intervention in human-in-the-loop scenarios where agency is clearly identified and experiences are thought through from the vantage of the human learning and knowing and actually making the decision, though it might be better informed with AI.
Deep: Is that sort of the direction that you're going?
Rana: Yeah, I mean, so you've named it right. And the radiologist at midnight is a perfect case because that's where the whole system quietly breaks, right? So the failure in that case isn't the model, it isn't even the doctor. It's that nobody designed for a moment when a tired human meets the confident machine at two AM.
Rana: So yes, UX intervention, but I push it further into what I call guardrails as ethical scaffolding. You know, so a few specific things. So act before acting. On any high stakes or ambiguous call, the system should pause and require confirmation. And that pause should be proportional to the stakes, not a model box, not a, a model box you, you click through.
Rana: Actual friction that scales, you know, with some, some consequence. And the second, I guess, you know, it needs to have the capability to show, uh, evidence and alternatives. I mean, you know, the AI does just-- shouldn't just say like, you know, malignant ninety-two percent. Like, it's says, "Here's why, and here are the three closest cases in the training set, and here's the alternative reading I rejected and why."
Rana: Now the radiologist has something to argue with. You can't rubber stamp a disagreement. Another one is, you know, you gotta expose uncertainty, right? Not a single confidence score which people learn to ignore, structured uncertainty. This region I'm sure about. I mean, in the sense like this region I'm guessing.
Rana: You know, this excuse me. The patient's demographics are underrep represented in my training data. You know, make the model's ignorance legible. That's important. And then you need to have some sort of a rollback built in, right? Every automated action needs an undo path, and the undo needs to be just as easy as the original action, not buried like three menus deep in there.
Rana: A-a-and then, you know, there's also the audit piece, right? Which is, uh, like memory walls of some sort. Like records of what was shown, what was decided, how long the human spent, what they overrode, what they didn't. It-- the, the goal is not to punish the radiologist at midnight. It's to surface the pattern.
Rana: I mean, if a hospital sees that between eleven PM and four AM the override rate drops to zero, that's a systems problem, not a doctor problem. That's a scheduling problem, not a staffing problem. I mean, and that's a UX problem, right? So I- It's also,
Deep: yeah, it's a system problem too. Exactly. Yeah
Rana: It is.
Deep: Yeah.
Rana: I mean, it's-
Deep: It's almost like you need all of the things you mentioned, but you also need proof that the human interacted and thought through in a high-stakes decision like that.
Deep: And that proof could be them pushing on, on your conclusions and following up somehow. It has to be more than just the, "Yep."
Rana: Yeah, I mean, yes. I mean the word I use for what you're describing is, legibility of engagement. You need proof, but proof that doesn't collapse into paperwork theater. Because the second it becomes paperwork, humans will optimize around it, and then you're back at where you started.
Rana: So you know, here's how I think about sort of putting this together, the proof together. And the best signal is contrastive interaction. Did the human actually engage with the alternative the model considered and rejected? If the system surfaces, I recommend A, but B was close, and here's why I wouldn't or didn't pick B.
Rana: Uh, a meaningful override or a meaningful confirmation requires a human to touch B, click onto it and, and then read the reasoning. Maybe the answer, maybe answer a one-line prompt. Why are you sticking with A? That's not friction for its own sake. That's forcing a-- the-- a counterfactual into the human's head, which is where, you know, the judgment actually lives.
Rana: You also need some sort of a stress question, right? Some sort of a second signal using a stress question. So before the human commits, the system asks one adversarial thing, like, "What evidence would change your mind here?" Or like, "Is this decision-- if this decision turns out to be wrong what's the like-- most likely reason?"
Rana: Like, you know, and you capture the answer. Whatever, ten, fifteen seconds it takes. And now you have a record that the human held a decision in their head as a hypothesis, not as a rubber stamp. Uh, that single artifact is worth more than, you know, a hundred approval clicks. I mean, I think there's a bunch of different things one can do.
Rana: I mean, you could-- Another one is like, you know, and this is something that a lot of organizations resist, is you sample. You don't audit every decision. You randomly pull, say, three percent, and you have a peer review the human's reasoning trace. Not the outcome, the reasoning. Because outcomes are just noise sometimes.
Rana: And if you only audit bad outcomes, you punish, uh, luck and reward laziness, uh, that happened to work, right? So I think all of this matters because of, uh, something what I, talk about in the book called the moral crumple zone, right? So if you don't build proof of genuine engagement into the loop, then when something goes wrong, the human gets blamed for a decision they actually didn't make, and the system that shaped the decision walks away completely cleanly.
Rana: And that's not accountability. That's just a liability trap with, uh, but a human's name on it.
Deep: I'm gonna play devil's advocate for a moment because it seems like what-- it seems like you're up against a pretty massive wall.
Deep: So like we, we know the machines, uh, in general do much better than, you know, the, the majority of actual radiologists. So when each one of these things that you're sort of advocating for, you know, at a million dollars a year for a radiologist end up being pretty, pretty darn expensive. And, you know, the institutions are to the extent that configuration parameters exist where they can like reduce the time that the radiologists spend, they're going to.
Deep: And another sort of thought here is yes it's kind of an easier decision to make in this scenario because we have decided as a society that we're gonna hold the, the, this radiologist totally responsible. But there are other cases where it's more ambiguous, right? Like let's take the, the self-driving car case.
Deep: The self-driving car case, Tesla wacks in and kills somebody. This has happened a few times. It's true the company is held liable, but there's no individual human held liable even though the driver could be, right? So like this is, to your point, I think like the Tesla, they have like a really light gauge of whether your hands on the steering wheel, but you could still be asleep with both hands on the steering wheel.
Deep: And so, I don't know that the I, it feels like there's, like, a big tension here between the efficiency and efficacy of these algorithms, the genuine value that they're bringing to society, their failures being weird mostly because they fail in scenarios that humans maybe wouldn't, right? The self-driving car case is kind of an exemplar there.
Deep: A lot of the accidents are cases where even a bad driver wouldn't have committed them. But statistically speaking, they're, they're much better at, at, at... than humans are at driving. And then there's, like, a whole other kind of angle that, that sort of is applying here, which is, I mean, at the end of the day, the question in the radiologist case, again is what's the best use of the limited resources that you have?
Deep: If you've got like, let's say, budget for 10 radiologists but none of them in the current model are looking at, imagery, you know, within, let's say, the first three hours of the imagery being captured and you can put a system in front of it to, like, look at everything within a few seconds, then you can route more efficiently.
Deep: Like that's, you know, maybe a case where you don't want a human in the loop and because there's already a human in the loop later on. But like how do you think through all of these natural trade-offs? 'Cause I know a lot of folks always instinctively argue on behalf of the human. But when it comes down to the data, sometimes it can be really black and white at the machine superiority and the cost efficiencies can, in a capitalist society, can be overwhelming in terms of the decision-makers' biases
Rana: Yeah, I mean, I think it's about who owns the decision, right? So I think let me give you maybe a messier example. And I've sort of watched this happen, like, a, a tool can ship as augmentation genuinely, right? GitHub Copilot in twenty twenty-one was pitched as a pair programmer.
Rana: Radiology AI was pitched as a second read, and customer service AI was pitched as agent assist. And ev-- each and every one of those cases within eighteen to twenty-four months, the economics pulled the tool across the line. And it wasn't a product decision, it was really a headcount decision made three levels up from people using the software.
Rana: You know, the mechanism is always the same. I mean, augmentation makes each human faster. I mean, faster humans mean you need fewer of them to hit the same output. And the, and you-- now the org chart has to be adjusted. And now the humans who aren't, uh, who remain aren't augmented anymore. I mean, they're supervising throughput they can't actually inspect.
Rana: Klarna made this explicit in twenty twenty-four when they said their AI assistant was doing the work of seven hundred agents. I don't think that's augmentation math; that's a replacement math wearing the costume for an augmentation situation. So when someone, asks me if the s-- if a system is augmentation or replacement, I don't look at the demo, I look at the staffing plan twelve months out.
Rana: Like, if the headcount for that function is flat or growing and the tool is making people better, it's augmentation. If the headcount has been quietly rebased down while the tool absorbs volume, it's just replacement, and everyone involved should be honest about that instead of, like, hiding behind the word like, you know, copilot.
Rana: So I think the, the place I actually get worried is the middle state where you've cut the, you know, the team by sixty percent and told the survivors they're now overseeing the AI. Those people cannot oversee anything. They're drowning. You know, that's the Air France four forty-seven setup at industrial scale, and we're building it in a dozen white-collared functions right now without naming it.
Rana: So I think, you know, replacement is a legitimate business choice. I mean, what I object to is is a, a replacement that markets itself as augmentation because then nobody staffs for the failure modes. I think that's the part I worry about.
Deep: I am going to keep play- pushing on this from a devil's advocate standpoint.
Deep: But it seems to me like, like if I look at, I think Boeing's a reasonable example here. You got a CEO that's already abstracted away from the responsibility that he sits behind, right? So we as a society decide head of Boeing if a bunch of 757s go down, um, there's going to be this massive review. That CEO has no idea.
Deep: I mean, like, especially the last one wasn't even an engineer, had no kind of capacity or ability to know that the decisions he was making were fundamentally screwing up the safety around the airplane. Yet we as a society still hold those folks responsible. Yet they as leaders still choose to not do the right thing.
Deep: It feels to me like the AI analogy is gonna be very similar, where just because we right now can sort of think down at the individual level, like, right, like the individual engineer that like says this, structural crack in this material that's going out to this wing is problematic. That's so abstracted from the ultimate person who says, "These planes are going out the door," like, "We're shipping these products," that even though you're holding that high-level person accountable, there's still a whole bunch of fuzz between them.
Deep: And I think in the AI case, it's gonna be very similar, where you have some human that's ... And it can all the way be down to the individual software engineer orchestrating 80 agents. They don't know, frankly, exactly what's going on. You got 80 agents, they're doing all kinds of crap.
Deep: They're pulling stuff out. They're filing problem tickets. They're reviewing code. They're gonna be shipping code if they're not already, and I'm sure they are in a bunch of companies. The engineer is gonna sit down and sort of describe their process. You know, their boss, their boss's boss, their boss's boss is gonna describe the process around it.
Deep: But I think what you're pointing out is like there's just like a fundamental kind of lack of way that we're thinking about this
Rana: Yes. Let me put a number on it, because I think the abstract version undersells the risk. There are roughly, three hundred thousand human financial advisors in the US regulated under the SEC or, you know, state authorities for that matter. I mean, they carry fiduciary duty, and they have insurance, you know, you know.
Rana: They're gonna lose their s-- you know, SU sixty-five. They can be sued personally. I mean, that's entirely-- That entire accountability structure looks about, uh-- that took about like, I don't know, fifty years to build, from the Investment Advisors Act of the nineteen forty through the Dodd-Frank reforms.
Rana: So now look at what shipped in the last eighteen months, right? BlackRock rolled Aladdin Copilot into the workflow of portfolio managers overseeing something like twenty-plus trillion in assets. And JPMorgan built Index GPT - and filed for the trademarks in twenty twenty-three.
Rana: Every major, robo, Betterment, Wealthfront Schwab Intelligent Portfolios is quietly integrating some sort of an LLM layer into the client-facing surface. And on the retail side, you've got people asking ChatGPT what to do with their 401k Uh, that, that's happening right now at scale with zero of the credibility constraints.
Rana: Oh, yeah.
Deep: I mean, I'll... let's get even more serious. Finances suck. It sucks to lose all your money. But the, uh, just three years ago, the FDA, like getting through the FDA process, it's, it's always been very difficult. But now what do we have? Like hundreds of millions of queries asking for all kinds of medical advice that's dispensed by all of the foundation models willy-nilly.
Deep: None of them have FDA approval. None of them have a shot in hell of getting FDA approval. We went from a world where almost all of your medical advice was coming from regulated humans to the majority is not And I think it's a... Yeah, I mean, I kudos for you for writing the book. I'm gonna order it and, and read it.
Deep: I think anyone listening should kinda dig in and, on this topic 'cause I think it's probably... You know, we talk a lot about the Terminator thing and Skynet and all that. I mean, that's just a red herring. These are the real-
Rana: Yes
Deep: ... this is like the real meat and potatoes problem with the... The real challenges of the next 20 years are here.
Deep: We just have to go through the really big, bad failures and really big, bad expensive lawsuits in this country at least. In Europe it's a little different. They actually regulate upfront. But, um, yeah, I don't know. I mean, like this is why I think everyone involved deeply in AI is excited and concerned like we don't know what's gonna happen.
Rana: I wanna- And I mean-
Deep: Yeah.
Rana: Yeah, exactly. Go ahead.
Deep: I was gonna say, um, you know, it's, it's been a awesome conversation. I usually end with one, uh, one final question, but I'll let you finish your thought and then maybe I'll, I'll, I'll come with that question.
Rana: No, I was, I was just saying that, we've sort of seen flares of that, right?
Rana: What happens first time one of those models decides at two AM on Tuesday you know, that a particular sector looks overexposed? And if it's advising ten million portfolios, ten million portfolios move in the same direction inside the same hour. Uh, we saw a taste of this with quant funds in 2007, you know, the August quant meltdown.
Rana: And so, I mean, you know, the danger isn't just that, you know, an individual gets bad advice. It's, it's that, you know, that drift is completely invisible.
Deep: Yeah. Yeah, it's, it's huge. Um, with that, I'm gonna end on my final question that I ask all of our guests. Uh, let's fast-forward five, ten years out.
Deep: Give us your kind of completely dystopian view of AI evolution, and then give us your kind of more utopian view of AI evolution where we, where we get it right.
Rana: Okay. I think One thing I'll say is like, you know, like stop asking whether AI will replace you and start asking, which specific tasks in your week are already being done better by the model you haven't tried yet, that's the actual question, and most people are too scared to run the experiment.
Rana: So I mean-- a-and I've watched this, right? People who've got displaced in the last two years weren't displaced by AI. They were displaced by coworkers who learned to use AI. Copywriters at agencies that got replaced, junior associates at law firms doing doc reviews, customer support, tier one.
Rana: You know, in every case, the headcount reduction wasn't AI walking in the door, it was the person next to them getting three times more productive and the manager doing the math. So I think the practical answer if you're listening and worried, is uncomfortable but simple. Pick the two tools that are eating your field right now and get fluent.
Rana: And if you're in marketing, that's probably Claude or GPT plus something like Perplexity for research. If in code, it's Cursor or Copilot, and yes, you need to actually use them daily, not dabble. So if you're in finance or ops, I mean, you know, it's whatever your firms has licensed plus a personal account so you can push further into it than like, say, IT lets you.
Rana: And the second thing is like, you know, a-and this matters more than tools tool fluency, is figure out what part of your job requires you to be accountable for a decision. Not just producing output, but standing behind it when it goes wrong, right? That's the part that's the hardest to automate away.
Rana: I mean, because orgs need a throat to choke and a human who can say, "I look-- I looked at it, I own it. Here's why." It's worth more in an AI-heavy workflow and not less. I mean, the people who are getting squeezed are the ones who, whose entire job was producing just intermediate artifacts, slides, first draft, summary screening.
Rana: And so the last thing I'd say is because I think people don't hear this enough, is the timeline's not five years to figure this out. I mean, it's twelve to eighteen months for most knowledge work to get restructured. The gap between the people who adapted and the people who waited is going to open faster than anyone is emotionally prepared for.
Rana: I think to move on while moving is still a choice.
Deep: Yeah. And on the, uh, on the future casting, what do you, uh, where do you sit?
Rana: I think my read is that we're heading into what I call a choppy middle future, and I say that because it's the outcome that's most consistent with everything we know about how humans actually behave under uncertainty. So not utopia, not apocalypse, something stranger, right? I mean, the utopian version says superintelligence resolves scarcity, disease, and conflict.
Rana: And the apocalyptic version says a misaligned system ends us. All of those stories are seductive because they're clean, and I think the middle one is messier, and I think it's the more likely, you know, cycle depolarization and partia-partial reconciliation, with AI amplifying conflicts in some quarters and helping manage them in others, often at the same time.
Rana: So I think-- I don't think the danger is malicious, right? Malicious AI. I think the danger is that humans project meaning, certainty, and identity onto different AI advisors the way we've done for millennia with religion, with political ideologies, with like, charismatic leaders. You'll get factions organized around their preferred models, not because the models are conscious or scheming, but because people crave certainty and alignment isn't universal.
Rana: It's plural, right? And so different systems will I guess give different answers on hard questions, and groups, uh, will mobilize around the ones that feel the truest to them. So I... You know, what I actually watch for isn't the singularity, it's the drift that, you know, I mentioned this before.
Rana: That's the biggest worry I have. It's like a gradual outsourcing of judgment, a quiet weakening of human agency you know, slow normalization of systems that shape our choices without awa- awareness. You know, none of that requires any coercion. You just need convenience, and you just need speed, and you just need optimization.
Rana: We'll, we'll see. I mean, but I'm generally uncertain which way it breaks. I mean, I think ... But I know the outcome isn't predetermined, and I know it won't be... I don't think it'll be decided in a lab. It'll be decided in classrooms, boardrooms, consoles, you know, s- dinner tables.
Deep: Courtrooms-
Rana: Um ...
Deep: I think.
Rana: Yeah. I mean, you know, the technical story is downstream of the human one.
Deep: Yeah. I mean, I'm kind of ... I think, at least in the US, I think the courtroom, we have a strong history of assigning liability, that we have probably the most litigious and libelous society. So I think we'll, I think we'll figure out for the big decisions who's responsible, and penalties will ensue, and that will ripple down into, like, various aspects.
Deep: But I also think it's gonna take 20 or 30 years before that sort of actually happens, and it'll be really messy. And it'll be a turbulent time, as I, as I think you're pointing out. Awesome. Well, thanks so much for coming on the show, Ron. I think this was a really fun conversation.
Rana: Thank you, Deep.
Rana: Yes, I really enjoyed it. Appreciate it. Thank you.
Deep: Do you wanna just shout out on your book? What's the title? Where can people get it?
Rana: Yeah. So the book is called The AI Instinct: The Future of AI and Human Decision Making published by Wiley foreword by Kai-Fu Lee, which I'm still a little stunned by every time I say it out loud.
Rana: You can find it wherever you buy books. I mean, Amazon, Barnes & Noble, uh, bookshop if you want to support the independent stores. And, uh, if you want the fastest path, just go, go to AI Instinct and every link is there in one place. And I also post various updates, errata, and some of the materials that I didn't, you know, that didn't make the final cut, uh, but I still think that mat-- that matters.
Rana: So if you want to, have a conversation, I'm on X @rana gujral, and my personal site is ranagujral.com. You know, I, I get every question you pose in there or message you send directly to me, and I do respond to it. And especially from readers who have actually cracked the book open and want to argue with something in it, that's the favorite part, so I, I really look forward to it.
Deep: Awesome. Well, thanks so much.