AI Can't Fix a Broken System

In this episode of ENGINEERING CH∆NGE®, I take a systems-centered look at one of the biggest conversations happening across engineering organizations and society right now: AI.

While many organizations are focused on AI models, tools, productivity, and technical readiness, this episode explores a deeper question:

What happens when AI is introduced into systems that already struggle with communication gaps, lack of trust, inequitable processes, or flawed decision-making?

Drawing on examples from industry and recent research, we'll discuss:

• The disconnects AI adoption is exposing between leaders and employees
• How AI acts as an amplifier within organizational systems
• Why “human in the loop” must involve accountability, not just oversight
• How people-centered organizational systems shape AI outcomes
• Systems-level questions leaders should be asking before scaling AI adoption

I also introduce MESA® (Measure, Evaluate, Strategize, Act), a framework we use at The PEER Group to help organizations navigate change and continuous improvement.

If you’ve been thinking about AI primarily as a technology issue, this episode invites you to consider the organizational systems shaping what AI will ultimately produce.

Grab a latte and listen.

Request your free copy of the ebook Engineering for Society at engineeringchangepodcast.com.

If this conversation resonates with you, follow ENGINEERING CH∆NGE® and leave a five-star review to help more engineers and leaders join the conversation.

Transcript

Intro

Speaker 0:01

Welcome to ENGINEERING CH∆NGE®, the podcast designed to help REDEFINE engineering by RE-imaging who we see as engineers and what we see as engineering; DE-siloing our approach to academic programs, research, and problem solving; and FINE-tuning organizational conditions so people with different backgrounds and perspectives can contribute fully to outcomes that serve all of society. Each episode offers actionable takeaways you can use wherever you are in the process of ENGINEERING CH∆NGE®. I'm your host, Dr. Yvette E. Pearson. Hello,

Varied Perspectives on AI

Speaker 0:39

agents of change. Welcome to the ENGINEERING CH∆NGE® podcast. People are all over the place when it comes to their thoughts about AI. I know you've heard all of this before. There are folks who think AI is the future. And certainly it's the here and now. Others look at AI as a threat. A threat to their jobs, a threat to their very livelihoods. There are folks who think of AI as a wonderful productivity tool, helps them automate these mundane tasks that we have to do every day or helps us to get things done faster. And then there are others who think of AI as the beginning of the end of humanity as we know it. And because people are all over the place, and our organizations are made up of our people, then our organizations are all over the place. We see some organizations who have been working on AI adoption over the past decade, before many of us knew what this thing even was. And so they've been full speed ahead, way ahead of the curve. Then you see some that are cautiously experimenting, dipping their toe in the water just to kind of figure out what might work for them. There are others who are resisting altogether. I've been in meetings recently where conversations have taken the tone of "Thou shalt not use AI under any circumstances." And honestly, I find that many organizations have a combination of these things, or maybe even all of these things, going on at the same time, because again, our organizations are our people. And because our people are all over the place, our organizations are all over the place. Now, you might be thinking, AI is not a typical ENGINEERING CH∆NGE® topic. Have I tuned into the wrong podcast? No, you haven't. Clearly, I'm not an AI expert, so I'm not trying to turn ENGINEERING CH∆NGE® into an AI podcast. But bear with me as I kind of reframe the conversation a little bit. In most spaces where I've been part of conversations around AI readiness, AI adoption, or read about these topics, AI readiness is framed in terms of the tools. So what are the right tools for the right types of projects, for the specific organizations we're in? What are the use cases that are appropriate for the work that we do or for the tools that we use? How can we use AI to improve our productivity without losing quality? And what are the risks around AI use? What are those technical risks to intellectual property, to client confidentiality, and all of those things? And I was just thinking, you know, organizations are spending so much time on the technical aspects of AI readiness, which is very much needed. So I'm not saying that's not important, but the yes and is that we also need to stop and examine the systems that we're bringing AI into. So what happens then when we bring AI into a broken system? That's what I want to talk about today. So grab a latte and listen as we dive into episode 33 of ENGINEERING CH∆NGE® and discuss why AI can't fix a broken system.

Thank You for Six Years

Speaker 4:32

Before I dive in, I want to extend a huge thank you to you, our audience. You know, as I record this, I realize the release date for this episode lands on the sixth anniversary of ENGINEERING CH∆NGE® podcast, May 20th. And in fact, it ends up also being the ninth anniversary of my company, The PEER Group. And so when I when I think about where we started six years ago, you know, I was questioning whether I would get listeners beyond the 18 folks in my family group text. And now we have audiences in over 100 countries across the globe. And in fact, Buzz sprout places our podcast among the top 25% podcasts globally based on our download statistics. And that's because of you. Despite a long hiatus, multiple years, you, our audience, you've come back strong and we so appreciate it. So thank you, thank you, thank you. I would love for you to just drop us a fan mail. There's a link in the show notes where you can do so. Let us know what you're enjoying about ENGINEERING CH∆NGE® and what you'd like to hear more of in future seasons. I would absolutely love to hear from you. Now, I just want to acknowledge that although you hear my voice, there is a team behind ENGINEERING CH∆NGE®. So I want to stop and give a shout out to the team really quickly as well, because without them, there is no ENGINEERING CH∆NGE®. First of all, Quincy, our co-producer, who is the one who helps me get the content in line, who helps keep me in line. And I've said this on previous episodes. He's the one who twisted my arm to start the podcast in the first place. So I really appreciate his support day in, day out to get this done. Rachel is another team member we could not make it without. She's our audio engineer who handles all of the post-production. So the quality that you're hearing, it's Rachel's hands at work. Sean is our audio engineer who got us started six years ago. And so I want to shout out to him because he laid the foundation for what you're hearing now. DK has stepped in as a backup on post-production whenever needed. I haven't had to bug him in a while, but I feel like if I called him now and said, hey, I need you, he would be there. So I appreciate you, DK. And finally, but not least, Freddy, who has been a huge champion of the podcast and serves in an advisory capacity. And so when we're having conversations about what future seasons should look like and how they might roll out, Freddy's in those conversations, helping with those insights. So thank you so much to the team that makes ENGINEERING CH∆NGE® possible. And thank you again to our audience for an amazing six years. Looking forward to more to come.

Diffusion of Innovation

Speaker 7:44

Now, let's tackle this AI conversation from a systems perspective. We'll do so by covering three broad topics. We'll start by talking about how we navigate change generally. Then we'll look at AI as an amplifier, and we'll wrap up with a conversation about human in the loop, which is a big part of AI readiness and AI adoption. Let's get started with navigating change. For those of us who work in the organizational change space, there is a well-known theory called the theory of diffusion of innovation. And what it does is it explains how change, whether that change is a new policy, a new technology, a new ideology or practice or whatever it is, how it is adopted, how it is spread throughout an organization. And it describes it in terms of how the people engage with or uptake the change. At the front end, you have a small group of folks who are innovators and early adopters. These folks are the creators, the risk takers, the thought leaders, influencers, the folks who are ready to just dive in and see what happens. Most of us tend to fall in the kind of middle stage, which would be early and late majority. We are the folks we've kind of got to have to see at least a proof of concept first, but you can you can pull us along, you can convince us that, hey, we need to do it. So we'll come in and step in with you. We're just not comfortable being on that front end of it. And then you have laggards. These are folks who are on the back end, the folks that you not only have to have a proof of concept, but sometimes you got to kind of drag them along, kicking and screaming, because they're gonna resist the change. And sometimes you have folks who just completely resist altogether. Some will say, okay, everybody else is on board. I'll follow. But then there are going to be some people who say, I am not with this at all. And so there's this resistance there. And I'll pause to say this. Whenever we're embarking upon change or thinking about change that's needed, it's really important to listen to folks from all these groups. And I would say extremely important to listen to resistors and understand the nature of the resistance. Some folks have very good reason for resisting certain changes. And the questions that they're asking can help us have a more effective and more successful change process and outcomes from our change. So don't just kick them to the curb. Make sure you're listening. Now, again, there are going to be some people who say, you know, I don't care what you say, I'm never going to be on board with this. And your aim is for that group to be the few and far between so that they're not impeding your progress forward. I just had to give that plug there to really listen to the resistors, address what you can of the resistance, what makes sense to address from the resistance, and move forward from

Are leaders listening?

Speaker 11:20

there. There is a study that Harvard Business Review reported on a few months ago. It was a nationwide survey this organization did of about 1,400 folks. And the people were employees and leaders of companies across the U.S. And the survey was about the AI adoption practices within their organizations and how AI adoption and uptake has been navigated. Their study found that 76% of executives reported their employees are excited about AI adoption. Well, when they asked the employees, they got a very different story. Only 31% of the employees who responded to their survey were actually enthusiastic about AI adoption. And so when we think about this change that most of our organizations are in the middle of at some space making decisions about AI, this article demonstrates something huge about the disconnect between leaders and employees. The study also found a huge difference in the perceptions of the engagement of employees and conversations around AI adoption and whether employees felt informed and heard. So again, a huge disconnect between leaders and employees. And what this particular study found is that the gap widened as they went higher up in the organizational food chain. What this means is, and what this shows us, is that leaders aren't listening. In fact, leaders aren't even asking the questions. A lot of times leaders make decisions and do what they deem best for organizations. A lot of times they also miss the fact that people make up the organization. People are the organization. And we don't only see this with AI adoption. We could plug in a number of different things and changes that we face as organizations over time. And the bottom line is for our organizations to be effective and healthy, we have to be able to listen to our people and not only listen to them and say, okay, we heard you and we're gonna just go ahead and do what we're planning to do in the first place. No, we have to also be able to take that information and incorporate it into our decision making. So if you remember the REDEFINE framework from our introduction earlier in the season, that's about FINE-tuning that organizational environment so that people are heard, seen, valued, people have input, and the decisions that we make then can produce the best outcomes, not just within our organizations, but when we think about engineering and other STEM organizations, it's about the societal outcomes that we're able to produce. I talk about this a bit as well in the ebook, Engineering for Society. And if you haven't downloaded it yet, please go to engineeringchangepodcast.com and request your free copy. It's free. There's a link in the show notes so you don't have to remember the website. In the ebook, I talk about this in the context of the difference between design intent and operating conditions. So when we structure our organizations, when we design our organizations, we intend for them to function in a certain way. We expect as a result to see certain outcomes. But oftentimes our operating conditions, what's actually going on when the rubber hits the road, tell a different story. Organizations and organizational leaders intend to, and they often believe that they've created collaborative environments around AI adoption. When you look at the organizational conditions, like what's been described in this article in HBR, they tell a different story. So the real story is there are communication gaps. There are decisions being made at the upper levels of the organization without input from all levels of the organization. There's uneven access to information. So even when these decisions are made at the upper levels, it's not spreading throughout the organization. So people don't know what's going on. As a result, folks feel like they aren't valued. Why am I here? I'm not important. Nobody cares what I think. And again, all of this points to that disconnect between the rank and file employees and senior leadership. And it produces an environment filled with anxiety because folks are uncertain, they don't know what's going on, and distrust. Because if you're not sharing stuff with me that I need to know, or if you're not asking my thoughts on stuff that's gonna impact me and what I do and how I work and the outcomes of my work, I'm not gonna trust you very much. And AI can't fix

Measure, Evaluate, Strategize, Act®

Speaker 17:11

that. These are the types of challenges my team and I at The PEER Group help organizations solve using our MESA® process for change and continuous improvement. MESA®, M-E-S-A, stands for measure, evaluate, strategize, act. So capturing employees' experiences, and we work with institutions as well, so we've done with this with students. But capturing people's experiences and perspectives, these are things that we do at the measure stage to understand what's working, what's not working, what are folks feeling and thinking and living in the moment. With or without AI, we have to be intentional in how we design, evaluate, and improve our organizational systems. And again, going back to how we introduced the podcast six years ago, that's what ENGINEERING CH∆NGE® is about. Applying that intentionality, applying that iterative design process to organizational problem solving, people-centered organizational problem solving just the way we would do it, and with just the same kind of intensity and intentionality that we would do with any other engineering problem.

AI is an Amplifier

Speaker 18:29

Let's shift gears a bit and look at AI as an amplifier. And before doing so, I want to just briefly introduce a characteristic that we talk about in the systems space. So whether we're looking at an engineered system or an organizational system or any kind of system, we're always interested in the interactions and interdependencies of the different components of that system. And among the key characteristics are the feedback loops. There are two broad types of feedback loops. One is a balancing loop that regulates system behaviors. So if you think of the thermostat in your home or your office, it regulates the temperature. There's a set temperature. Whenever the space starts to get above or below that temperature, the system will kick in to bring it back to that set temperature. So it's a regulating function that's considered a balancing loop. On the other hand, we can also have reinforcing loops that amplify system behaviors. And an example of that would be adding traffic lanes to try to mitigate problems with traffic congestion. A lot of times what we see is when more lanes are added, we end up with the same level or maybe even worse traffic congestion for a number of reasons. And so what happens is the change amplifies what was going on in the first place. So it's a reinforcing loop. AI is an amplifier. If you bring AI into an organizational system where things are good, they're working well, AI can help make them better. On the other hand, if you bring AI into a flawed system, a broken system, it can make what's already bad worse. Several years ago, Amazon was in the news for their failure around using AI in their hiring processes. They used AI to evaluate applicants for software developer jobs. And they trained the system using 10 years worth of their data. So basically taking resumes, applications, and things like that from folks that they'd hired over the past 10 years and training AI to say these are the folks who do well in our organization. And surprise, at the end, when AI looked at the resumes of folks applying for these new software developer jobs, it was weeding out women. There was a clear bias against women. Stop and think about it for a moment. If you have a system where you have historically been biased against women, and as a result, you have hired more men, and then you train the AI on the data from your flawed system, it should be no surprise that what you get out of that is going to be flawed. AI can't fix a broken system, it's gonna amplify what's there. Now, Amazon stopped using that, and this has been many years ago, and in fact, in the article that I read, they said they never used it to evaluate candidates. I won't get into all of that, but the bottom line is this - garbage in, garbage out. That's something I remember learning very early on when I started learning programming or as folks call it today, coding. Back when I was in middle school and high school, it's garbage in, garbage out. The computer can't do any more or any less than you tell it to do. And people use the analogy to relate AI use to calculator use. If you put in flawed information into that calculator, it's going to give you garbage on the other end. You have to be able to interpret the results and know whether they make sense or not. So garbage in, garbage out. And then also we have to think about in the context of this hiring example and AI being an amplifier, what are our systems designed to produce? What are our systems designed to produce? And what are those performance signals that let us know whether our systems are setting us up to produce the outcomes we desire? And so in the ebook, I talk about this in terms of performance signals and how what we observe or measure depends on the resolution and the sensitivities of the instruments we use. So when I think about this from a civil engineering laboratory example, from higher ed space. You can have an environmental analysis lab that uses analytical balances that are very sensitive. They have draft shields to protect it from even the air in the room affecting the weights because they're designed to measure very small quantities to the thousandths with a THS on the end of milligrams. And then you can also have structures lab with equipment that handles materials that weigh hundreds of tons. And so what we're getting out of that is depending on what the instrument, what the system in our case is designed to detect, what outcomes are we aiming for? And are our "scales" designed to help us see what it is that helps or hinders us from realizing those outcomes? So if we go back to the MESA® framework, we talked about how we measure to see what's going on. This gets into the evaluate stage where we try to understand why, understand the root causes of what we see, whether it's stuff that works or doesn't work, and then think about where we want to be. So it's that deep dive into the why. And it's also that forward look into what are those outcomes we want? Where do we want to be? Where should we be? What outcomes are we trying to attain? With or without AI, meaningful change requires us to understand these things - root causes, and again project our targets and remember garbage in, garbage

Human-in-the-Loop

Speaker 25:13

out. So we started off with a general conversation about navigating change and what that looks like in the space of AI adoption and AI readiness. Then we kind of went into a discussion about AI as an amplifier and how it reinforces systems behaviors, good or bad. And now we're gonna take off on the third topic, which is something we nodded at in talking about AI as an amplifier, and that is the need to keep human beings in the loop. So you hear a lot of conversation about the dangers of anthropomorphizing AI. It's pretty widely understood that we can't have the computers making all the decisions for us. We have to have human beings making decisions. We need human beings to be able to take responsibility for the inputs to and outputs from our systems. If you've used AI, it's likely you've experienced hallucinations. I was interested in taking a deep dive on a topic and exploring a topic. And I said, hey, you know, instead of doing just a Google Scholar search, I wanted to have AI to do a little of the upfront work for me. So I put in a prompt that asked AI to identify five to ten peer-reviewed articles from reputable sources on this topic and provide me with complete citations and give me a three bullet point summary of each article. And so it gave me my list of five articles, full citations, authors, publications, volume numbers, everything. And so it gave me the three bullet points and the ones that looked like they aligned with what I was looking for, I started to go and search for the article so I could read the full article. And I checked one, I couldn't find it. I picked another one, I couldn't find it. I did this like three, four times, and I couldn't find any of these articles. And so I went back to AI and said, Hey, I can't find any of these articles. Where'd you get them from? And AI basically said, My bad, I made this stuff up. And so if you don't have a human involved, if you're not taking the time to say, you know, I'm gonna actually look for and read these articles to make sure, first of all, that they exist. And then secondly, that they say what this AI says, they say, that's a huge miss. It's one thing when it's, you know, just researching topics of interest, it can bring on a whole mass of other problems when you're using AI without that human reasoning, thinking, checking in the process. Depending on the AI application you're using, the problems can be quite severe. So you cannot rely on it. We cannot rely on AI. We have to involve our human intellect, and not only our human intellect, but human empathy.

Speaker 28:31

I remember talking to a director of a gender center a few years ago. This was in one of the states that was passing anti-DEI laws. And at this time, they were discussing the anti-DEI laws that would impact public institutions throughout that state. Understandably, this gender center director had students reaching out wondering you know, are they gonna have resources and services and support available for them? And when this one particular student reached out to this director initially, and this is before the law passed, the director responded and gave them a very human, empathetic response, expressing concern, assuring them that their team was there to support however they could. Fast forward, later in the process, there was a directive from the institution and also from system level leadership that said basically, nobody from your team is supposed to speak to anybody about anything related to DEI without it going through us. And so this same student reached out after the law had passed and had some specific questions and concerns. And so the director, following the instructions that they had been given, they sent the students' email up the food chain. And what came back was a canned response to send back to the student. The director was not satisfied with this, but they had to do this. This is what the higher-ups told them they had to do. They sent this response to the student. The student wrote back and said, Wow, I cannot believe the lack of empathy in this response. When I first reached out to you, you were so warm, you expressed such deep concern that it just basically floored the student that they got this dry, non-feeling response from this same person. And the student said in their message back to the director that I ran these two responses through AI, and AI even told me that the second response was AI generated and lacked human empathy. And so these are problems that AI can't fix. Communications require human touch. AI can help us maybe clean up language or flow, but AI cannot express human emotion and human empathy. And I know empathy is the latest bad E-word in some circles, but I stand 100% firmly in saying human empathy is very important to our organizations. And so that's something that AI cannot replicate because it's not human. And then framing this in a slightly different or from a slightly different angle, there was a more recent article on hiring and the use of AI in hiring than the one we talked about earlier with Amazon. This one was, I think, either a 2025 or 2026 article from Reuters. And it was a UK study conducted by an organization called Greenhouse. I'm not familiar with them, but Greenhouse uh interviewed about 3,000 job seekers in the UK. Nearly 50% of them said that they had encountered AI in their interview process in some sort of way. And one of the questions they were asked was whether they had ever walked out of an interview or they would walk out of an interview because of AI use in the process. And around 50% of the candidates who responded to their survey said that they have or would walk out of AI interviews. And some of the reasons they cited included encountering prerecorded interviews that have no human presence or interaction. Remember, we're talking about human in the loop here. Another reason is failure to disclose AI use and then having AI monitor interview processes. And I know for me, I hold a lot of virtual meetings. I believe we all do at this stage in time. And a lot of times people will have AI in the meetings as participants to take notes. And I'm just not entirely comfortable with AI monitoring and whether I could trust AI to represent what I said and the context in which I said it in an accurate way. I don't have trust of that. Certainly when the AI reports then back to someone else who initiated it, and I don't get to review it and clarify anything that it might have missed. So getting back to the this study from the UK, again, these folks are saying that they are not comfortable with AI in the interview process. And this is 50% of the 3,000 folks they surveyed. Over a quarter of them, I think it was around 27%, reported they felt they experienced age bias because of the AI. And nearly a fifth of them reported that they experienced racial or ethnic bias because of the AI. And this again goes back to what we were talking about with the amplifying or reinforcing loops. If your system is biased to begin with, AI is not gonna fix that. You've gotta fix the system. The lack of disclosure and the failure to disclose to people that you're using AI in the process before they even enter engage in the interviews would be ideal. But the the lack of disclosure altogether is to me quite shady. You know, people are in these interviews sharing personal information. Where's that going? And who has access to it and how is it being used? Again, this is something that goes beyond hiring processes. These are things that are important to address in AI adoption more broadly. And we need people to be thinking about the concerns people have over these things. Again, going back to our topic of humans being in the loop. The article that I was mentioning, the UK study by Greenhouse, the CEO and co-founder of Greenhouse, his name is Daniel, and I don't know if it's pronounced Chay, Chat it's C-H- A- I- T, if you want to look it up. But the article wrapped up with a quote by him that I think summarizes things perfectly. He said, "Most AI in hiring today is making a bad system worse... The process AI is being built on top of was already broken." And then later on, he adds something to the extent of rather than adding AI to a broken system, build a better one. And so that involves not just having humans in the loop, but making sure that humans aren't perpetuating the problems that have historically produced the undesirable outcomes we've been experiencing in the first place.

Speaker 36:20

And that's what this podcast is about. That's what this episode is about. How do we use professional development? How do we use process and policy improvement? How do we use systems and organizational change and continuous improvement processes to fix our systems before we introduce a tool into our systems that's going to amplify whatever is there? So if we think about MESA® again, we talked about measuring to understand what's going on, good or bad, evaluating to understand why we're seeing what we're seeing and to think about where we want to be. So this part deals with the next step, which is to strategize so that we're making people-centered, data-informed decisions. And we're using evidence-based strategies to address the root causes before we act, which is the A in MESA®, rather than just throwing things against the wall to see what sticks, or rather than taking the bandwagon approach, doing something just because someone else is doing it, or everyone else seems to be doing it. That's not what we want here.

System Check

Speaker 37:37

So I hope this has been helpful. Let's transition and do a quick system check. Over the next few weeks, try to answer these questions as you navigate your organization. Do people in your organization actually feel informed about how AI is being used? Or do leaders simply believe they do? And that goes back to that first article that we talked about from HBR. Are leaders listening? Are leaders even asking questions? What concerns about AI are being raised? And do people feel safe asking questions and challenging the decisions that are being made? What kinds of behaviors and outcomes are your systems rewarding right now? And what might AI unintentionally amplify? Are humans truly in the loop? Or are they slowly becoming rubber stamps for systems they no longer fully understand? If an AI-supported decision harms someone, who is accountable? That is really something to stop and pause and think about. And if AI amplified your current organizational systems tomorrow, would you trust the outcome? Or better yet, would you want AI to amplify your systems as they exist right now? Think about i

Speaker 39:19

t. These are the kinds of systems questions that we help organizations navigate through at The PEER Group using the MESA® framework again - Measure, Evaluate, Strategize, Act. Because meaningful change doesn't happen just by introducing new tools. They can help, but they can also hurt. What's key is intentionality in how we design, evaluate, and improve the systems that are shaping our engineering work and our societal outcomes.

Closing Thoughts

Speaker 39:54

If today's conversation resonated with you, please be sure to give us a five-star rating and review. It really does help this content reach others. So until next time, keep ENGINEERING CH∆NGE®.

Outro

Speaker 40:09

Thank you for listening. If this episode was useful, do me a favor, subscribe and leave a five-star rating and review. It helps this work reach others who are navigating change. To download resources or share ideas and questions for the show, visit engineeringchangepodcast.com. Until next time, remember the most meaningful change comes from being as intentional about our systems as we are about our solutions. That, my friends, is ENGINEERING CH∆NGE®.