[00:00:00] David Bergvinson: Well, to the next episode of Grounded Intelligence. This is episode seven. We’re talking about risk what could, in brackets, possibly go wrong. And with me to discuss this very interesting topic is Elliot Grant and Ricci Pith, and they’re going to introduce themselves a little bit, in a moment.
[00:00:19] David Bergvinson: Uh, but for now, just a high-level framing. You know, we think of technology as addressing problems, and indeed, artificial intelligence, especially as it relates to advisory services, can offer a huge benefit to farmers. But any technology comes with risks, uh, whether it’s a car, plane, or artificial intelligence. And so on today’s episode, we’re gonna talk about those risks, their implications, and how we could possibly get ahead of them to mitigate their impact on smallholder farmers and other stakeholders. So joining me now, uh, is Elliot Grant and Ricci Pith. Both are advisors for the Gates Foundation, helping them frame their thinking [00:01:00] strategy and execution of the services to empower farmers.
[00:01:03] David Bergvinson: And so, gentlemen, welcome to the show, and just invite you to offer a brief bio and how that fits into this topic.
[00:01:11] Elliott Grant: Oh.
[00:01:11] David Bergvinson: why don’t we start with you?
[00:01:12] Elliott Grant: Thanks, David. Appreciate, uh, the invitation to be here and talk about this important topic. Uh, as you mentioned, I’m Elliot Grant. I’m an advisor to the Gates Foundation. Uh, I also work, um, in Cambridge University as head of an innovation lab. Um, but I think more relevant to this conversation is, uh, I’ve spent the last 20 years working on ag tech and food tech startups, uh, building solutions to help improve the sustainability and traceability and safety of the food system.
[00:01:42] Elliott Grant: In fact, many of that ti- much of that time I worked with Rishi, and he’ll, I’m sure, talk about that as well. Uh, and for the last year, I’ve been here at Cambridge also researching AI and the impact on the food system, which is where I really started to think a lot about the risks that we are facing[00:02:00]
[00:02:00] David Bergvinson: Great. Yeah, we had some engaging conversations on this topic that, uh, resulted in it being added to our list of, uh, discussion papers that, uh, you and Rishi have authored. Rishi, over to you
[00:02:12] Rhishi Pethe: Uh, thanks David, for having me on this podcast. I’m excited to talk about some of the topics, uh, that, uh, we are gonna talk about. So in terms of background, I’ve spent about twenty-five years in tech, and a lot of it has been with legacy industries like food and agriculture, most of my effort is how do you deploy, build, and deploy technologies which create value. And over the last couple of years, I spent a lot of time on automation, AI, and supply chains, uh, especially for legacy industries.
[00:02:40] David Bergvinson: Excellent. Yeah, you guys both come with the credentials to speak to this important topic, so thank you for sharing your, your insights, knowledge, and perspectives with us on, on risk and how we get ahead of it. Um, so, you know, when we talk about, um, we… You know, there’s sort of the primary risks of a [00:03:00] technology, and then there’s secondary and tertiary risks, largely invisible until they happen. Um, and so, you know, I’d like to sort of, you know, talk about the high-level risks that we probably all are aware of, but then get into some of these more nuanced risks and how we can get ahead of them. So if we can achieve that during our discussion, that would be fantastic. Um, and so just to start off, know, what are those high-level risks, uh, that are framed in the paper?
[00:03:26] David Bergvinson: I think you categorize them into eight large categories. Um, maybe if you could just offer your perspectives on, uh, those eight categories as a s- as an entry point for our conversation. Elliot, do you wanna have a,
[00:03:39] Elliott Grant: Yeah. Maybe I’ll actually take a step back first, Dave, and talk about what’s the, sort of the context for this. If you look back at the history of agriculture, whether it’s Green Revolution or the, uh, invention of fixed hot nitrogen or, um, discovery of CRISPR, there– These things are tr- tremendous, right?
[00:03:57] Elliott Grant: We’ve improved productivity and sustainability, et [00:04:00] cetera. Um, and I think we were able to anticipate the first-order risks, as you said. The first-order risks are the ones we can see very clearly. But what happens is, over time, these second and third-order risks emerge, and they have long tail consequences.
[00:04:15] Elliott Grant: So in the case of fixed nitrogen, for example, there’s no doubt that having, um, artificial nitrogen was a huge boom. Um, but now we’re seeing eutrophication of water, nitrous oxide emissions, uh, toxification of soil. Those were not really anticipated when this invention first happened. And I think, you know, we can see these waves of innovations that happened over a hundred years.
[00:04:38] Elliott Grant: Now we’re facing the next great innovation, which is artificial intelligence. And so I think the question that frames up this co- topic is: What can we learn from those previous inventions and the long tail risks, the second, third-order tail risks, that we should try to anticipate this time around? So that’s sort of the, the context of why aren’t we just saying this is all gonna be great?
[00:04:59] Elliott Grant: ‘Cause the answer is, it usually isn’t. Um, and so to then sort of transition to this question of how do we even think about second-order risks, Rishi and I, um, worked on this paper that tried to think through several steps ahead and then categorize those risks into different buckets. Maybe I’ll, I’ll talk about a couple, and Rishi, feel free to jump in.
[00:05:20] Elliott Grant: I think the most obvious ones, and arguably this is a first order risk, is the model’s just wrong, right? It just gives bad advice. Uh, so if I’m using an AI agronomy tool and I show it a picture of a diseased leaf and it just tells me to spray the wrong kind of herbicide, I would call that a first-order risk.
[00:05:38] David Bergvinson: Yep
[00:05:38] Elliott Grant: whereas a, a second-order risk might be, um, that the, the models get worse slowly over time, right? We just sort of don’t notice it. Uh, and we become dependent upon the model, and we lose that sort of intuition that we have as humans. So I g- I use that as an example of how do we think of a first order and then think a little bit [00:06:00] deeper.
[00:06:00] Elliott Grant: Um, maybe I’ll pick the second one. So the first big bucket is, is the model is just wrong, right? There’s just risks in the model. The second one is this, as I touched on, sort of this ethical problem of, um, where does the knowledge come from? Are we extracting knowledge from experts, from indigenous people, from farmers, and then effectively packaging up and selling it back to them?
[00:06:23] Elliott Grant: So there’s a risk that we undermine the, the economics effectively of farming. Rishi, I wanna toss it over to you to pick up a couple of the other buckets.
[00:06:33] Rhishi Pethe: Yeah, thanks Elliot. Um, yeah, I think we could also use– I mean, I like the framing of the second or the third order effects, which are always difficult to figure out. And the attempt with this paper is to say, can we anticipate some of those things? And if we can, uh, can we design some mitigation criteria or procedures so that the risk of those happening are lower? Or, you know, if you could use sort of the known unknown matrix from Donald Rumsfeld, um, and, you know, the unknown [00:07:00] unknowns are gonna be very difficult, but like the known unknowns and the, unknown knowns, you know, can we anticipate those, right? So to build on what Elliot is saying, you could also, um, sort of part– um, partition these risks into two categories.
[00:07:11] Rhishi Pethe: So one is sort of system level risks and one which, uh, impacts an individual farmer, um, at, at different levels. So like for example, Elliot gave example of just spraying the wrong thing, right? That’s a farmer level risk. Uh, now, if that same advice is given to all the farmers because that’s what the model knows, like that creates a different system level risk. Or if everybody is asked to plant a particular type of, uh, hybrid or a particular type of seed, then because of a change in environmental conditions, um, you know, that creates a system level challenge, um, for the food ecosystem. there are different levels of risk. think the other thing which, um, is also challenging to figure out is, as Elliot mentioned, is certain risks don’t show up on day one.
[00:07:58] Rhishi Pethe: They show up like on, on day [00:08:00] ten or, you know, three years down the line. A model drifts or because of climate change, your model no longer is applicable, um, because the conditions on the ground have changed. Uh, so those are also something that we should think about as to how do those risks, how will those risks show up? Have some idea of how to detect when we start seeing signs of those risks, and have some idea of what kind of mitigation or what kind of steps we can take to reduce the chances of the, or the downside effects of those risks.
[00:08:29] David Bergvinson: Yeah, and, and, and some of our topics have actually talked about inclusion. So, you know, some of the risks, you know, could be, uh, language barriers, for example, and people being left behind unintentionally, but nonetheless left behind. you talked about environmental risks. Uh, there can be economic risks, obviously, following instructions that lead to, uh, destruction of a crop or loss in yield that, or l- lack of accessing a market. Um, you know, there’s, there’s several different topics here. What
[00:08:57] Elliott Grant: Actually, David, can I build on that one? Sorry to sort
[00:08:59] David Bergvinson: you
[00:08:59] Elliott Grant: in. I [00:09:00] think the economic one’s a super interesting one, which, which has these second order that we often see play out. Um, one of the big challenges in agriculture is how power tends to concentrate, right? Whether it’s in seed companies or equipment companies or off-takers.
[00:09:14] Elliott Grant: Uh, and here we’re talking about a technology that we wanna deploy very often to low resource farmers, right? Sub-Saharan Africa, Southeast Asia. Um, there’s a– I think that we should be talking about the risk of creating more concentration of power unintentionally. Um, really hard to imagine how these AI models will play out, but who’s gonna train them?
[00:09:36] Elliott Grant: Who’s gonna supply them? Who pays for them? These have those sort of economic second order effects.
[00:09:44] David Bergvinson: Yeah, indeed. Oh yeah, this, uh… I mean, it’s really hard to look in the crystal ball five years from now even, even a year from now, frankly, and how this will play out as far as, uh, as you point out, concentration of power. We’re already, you’re seeing that right [00:10:00] now, that play out in the frontier models and the companies that are developing them, uh, and how that is, uh, matching up against the open weight models or the open source models that are, are coming, uh, from other parts of the world.
[00:10:12] David Bergvinson: And so, yeah, it’s, uh, it’s going to be a very interesting journey, and I guess the paper’s framing is gonna help us think through how do we get ahead of some of these issues and put in place the policies, um, the rules of the road, if you will, to mitigate or at least to reduce the impact of these risks. Um, and frankly, even just track, uh, the emergence of these risks. I don’t think we really have good systems in place to do that even right now. Um, excellent. What… You know, in, in developing this paper, and it was great that you did look at the second, third order consequences of risks associated with advisory services that are enabled by AI, what do you think is likely the most nefarious, potentially, and, underestimated second or third order risk? [00:11:00] I know that, that-
[00:11:00] Elliott Grant: Rishi, you wanna take that one first? So I get more time to think.
[00:11:06] Rhishi Pethe: yeah. I mean, I think nefarious is probably not– Well, maybe in some cases it is true. Uh, but a lot of these are not because of bad intentions per se on somebody’s part. Some of them might be, uh, but it is linked. A lot of these show up because of how the system is desi- uh, designed, how the incentives show up, how information is shared, who has access to information, who doesn’t. so I think maybe we need to separate between like bad actors and just the system behaving naturally, right? And, and what can we do about it. Um, I, I think some of the, the ones which, uh, I worry about more are, um, just information asymmetries that can show up, um, because as an individual farmer, you have access to a certain set of information.
[00:11:53] Rhishi Pethe: Now, with the power of AI and the amount of information that a large corporation or a government for that matter, a state [00:12:00] actor can accumulate, and it just creates a lot of information asymmetries, um, which can, uh, put farmers, um, at a disadvantage because they don’t have access to certain information, whether it is risk of pricing.
[00:12:14] Rhishi Pethe: You know, if I am a, a seed company and I, I have access to all this information, I know which, which products are performing well and where, I can use that in my pricing decisions, um, for example. I, I think those are the types of things, um, uh, which can show up, and those are very difficult to, uh, track as well.
[00:12:33] Rhishi Pethe: That, that’s what makes it difficult. Um, the time when you realize what’s happening, uh, and when it starts happening, there might be a big gap, and you already have incurred a lot of harm to a bunch of farmers, uh, that you don’t want. Uh, so I think information asymmetry, um, i-is a challenge. I mean, I think we, we talked about, um, in the paper, Elliot mentioned something like a flash crash.
[00:12:56] Rhishi Pethe: I worry about that less, but even today without [00:13:00] AI, we see that happen sometimes, like in the tomatoes or onion markets in, in India, for example, right? The prices fluctuate massively because farmers got the impression that there’s going to be a shortage of a particular crop and everybody goes and plants that crop and then there’s a glut of supply. I, I think those kind of problems could get accelerated, how frequently they show up, uh, and at what, what speed they show up. So I think those are some of the sort of… I, I probably worry more about system-level risk than individual farmer risk. Obviously, we don’t want to do harm to any farmer, but those seem to be of a lower, um, magnitude, and maybe they’re manageable with other mechanisms.
[00:13:38] Rhishi Pethe: Uh, I think the system-level risk is what I worry about more.
[00:13:43] Elliott Grant: That, um, yeah, really good. And, and I agree completely with Rishi that this, you know, nefarious suggests intentional and, and maybe that I agree this is not what worries me most. I think there are two things I worry about and, um, adjacent to what Rishi mentioned. Um, [00:14:00] one, and I’ve experienced this one myself, I use Google Maps to literally go to work every day.
[00:14:05] Elliott Grant: Like I, I know the route, but I’ve become completely dependent on Google Maps. Um, and certainly if I’m doing anything, you know, unfamiliar. And I have a sort of a learned helplessness as a result. You know, I’ve, I’ve got this cognitive offloading that I’ve offloaded onto a tool, and it’s wonderful, but if that tool goes away or if that tool is wrong, I’m, I’m completely lost, literally and figuratively.
[00:14:26] Elliott Grant: So I think the analogy here would be the risk to indigenous knowledge, to, uh, sort of the inherent knowhow. And it– I don’t think it would take that long. It probably took me a few years to become useless at navigating. I do worry in a couple of generations or seasons we end up with this learned helplessness.
[00:14:44] Elliott Grant: So that’s one risk. Uh, uh, only insofar as if something goes wrong, we have no recourse. The second is a little bit more, maybe it is nefarious, but the second is we are potentially handing over a very fragile [00:15:00] system to an AI digital system, which opens a vulnerability to bad actors, right? So if, let’s say, this AI agents become wildly popular, ten years from now everybody is using them, without good cybersecurity, that creates a new risk for a very large number of people.
[00:15:19] Elliott Grant: So I think that sort of back bookends effectively. Maybe I’ll use Rishi’s framing as well. That’s the individual person losing their knowhow, and then we also introduce a systemic risk, uh, to the whole agricultural ecosystem
[00:15:33] David Bergvinson: Yeah, those are important points. And, uh, you know, what, what is it that we can do to get ahead of each of those, um, you know, potential risks? Is there, is it really a policy play? You know, how is society going to put the guardrails, um, to manage against these? Or is it just too complex to address?
[00:15:57] Elliott Grant: Um, maybe I’ll, I’ll take a crack at this and then Rishi, [00:16:00] clean me up. But I mean, and Rishi mentioned this in the opening there, the, the, the goal of this exercise is to try to anticipate these things because they can– they are not unforeseeable. Um, so if we can foresee them, then we can put in mitigations.
[00:16:15] Elliott Grant: Now, there will be the unknown unknowns that Rishi mentioned that will just come out of the black swan events. But our job should be as, you know, providers of this technology or funders of this technology, um, to try to anticipate some of these risks. And I think the consequences or the mitigations fall to a number of different types of actors.
[00:16:34] Elliott Grant: Clearly, the developer of the software has a certain responsibility, right? Making sure that the models are accurate or that the claims you make about it are accurate, that the uncertainties are, are shared. So that’s the– I would call that sort of a level one. You know, if you’re developing this tool, you’re responsible for avoiding some of these risks.
[00:16:52] Elliott Grant: I think the state has an important role to play. In the ca– agriculture is very often highly intertwined with the state, so state actors [00:17:00] have a role to play in terms of governance, liability, um, regula-regulation, benchmarking. There’s a lot the state can do. Um, the end users have some responsibility to be informed, um, although we really shouldn’t put that burden too much on these end users.
[00:17:16] Elliott Grant: Um, and then I think there’s probably th-third parties who can– who will play a role to come in. Let’s say they might be model evaluators who can provide effectively sort of a bill of health to say, “This is certified to be safe.” And that could be independent or it could be government. So, you know, and then there’s probably a bunch more, and this is what the paper sort of lays out, and if you read it, is here are all the risks, how do we mitigate them?
[00:17:40] Elliott Grant: And it isn’t gonna be a single organization that’s responsible.
[00:17:46] David Bergvinson: You know, Ricci?
[00:17:46] Rhishi Pethe: I think those are all valid points. Um, I, I think, you know, if we think about, um, who the end user is or who’s the, um, the biggest stakeholder is the farmer, I think I would probably put a little bit [00:18:00] more responsibility on them than what Elliot is suggesting because at the end of the day, um, you know, even today, like treat these things as if you’re talking to a friend.
[00:18:09] Rhishi Pethe: You know, your friend tells you, like you don’t believe everything they say, you still use your judgment. And I think a lot of these farmers or most of these farmers are very smart. They know their land. They’ve farmed there for a long time, so they have a lot of context which people don’t. So would still put some responsibility on them, like, you know, use this as if you’re talking to a or some stranger, um, uh, if you will, and, you know, treat the advice accordingly, right?
[00:18:33] Rhishi Pethe: So maybe that’s something. And that requires some level of educating the, the, um, uh, stakeholders saying, “Hey, this is a great tool, but still use your own judgment.” Um, this is not to, uh, condone any sort of activities or any, any responsibility from the developers or the state governments. Um, but I think we need to sort of talk about that, right?
[00:18:55] David Bergvinson: Yeah, it,
[00:18:55] Rhishi Pethe: think about sort of– Sorry, go ahead.
[00:18:57] David Bergvinson: well, it brings up in my mind, you know, two [00:19:00] things. One is, um, sort of a participatory approach, uh, to developing these, uh, services. So quite often farmers are left out in the development phase. Um, and so that, you know, that could actually potentially mitigate, uh, some of the risks you highlight. And the, the second is education. I don’t think we’ve educated farmers enough on these potential risks. So we, we’re good at highlighting potential benefits. We’re not so good at communicating potential risks, um, and having that dialogue with users and at least creating some awareness. We, we, you know, talked in our last episode around data and data sharing and how farmers really, uh, as you point out, it’s asymmetric in the benefits of that right now. And so how do we, you know, create awareness and educate farmers on these, uh, potential risks? Any thoughts around how we can do that a little more systematically or intent- intentionally?
[00:19:54] Elliott Grant: Um, and it’s, it’s good that we don’t agree on these things all the time with Rishi [00:20:00] and I. We have this good creative tension. Because maybe I wanna come back to the sort of the ability to educate people, and I’ll, I’ll use a different analogy, the self-driving car, right? So I had a self-driving Tesla when I lived in California, and I knew that this was only level four.
[00:20:12] Elliott Grant: It wasn’t level five. But man, after you’ve used it for about an hour, it’s almost impossible not to sort of lo- you know, stop paying attention. Even though I know the limitations, the cog- the, the desire to offload that cognitive burden is very high. So my worry, and, and Rishi, I take your point, like treat it like a friend, but if it’s billed as this super intelligence which knows everything about agriculture, um, it’s gonna be very difficult to not become to, you know, start to rely on it.
[00:20:40] Elliott Grant: So I do worry that we don’t wanna… My worry would be like, “Oh, we’ve educated these users. They, they know what they’re doing.” I just think it’s human nature to say, “Well, I’m gonna just default to this.” So maybe I’m coming on the side of I think there’s a bigger role for the government to, um, to play here
[00:20:59] Rhishi Pethe: Yeah. [00:21:00] And yeah, I, I agree with you. I think maybe how you design the systems, like you could design something in it. I’m not saying this is the only way to design it, but for example, when you talk about risks or probabilities, like it is very difficult for most people, including, you know, smart people on this podcast, to understand like, okay, this has an eighty percent chance.
[00:21:20] Rhishi Pethe: What does that mean, right? Like, and how do I make a decision based on that? maybe, y-you know, these tools, they can express the uncertainty as an action instead of saying like the example, uh, from that, um, from China where somebody sprayed twenty-five acres of their sesame field and, like, it killed everything. know, maybe those recommendations are, know, try this on a plot. A-And, and instead of saying there’s a ten percent chance of that, you know, this will destroy your field, ten percent are like, “Fine, great. It’s ninety percent great, I’m gonna go ahead and do it,” right? That’s not gonna work. So I, I think we need to sort of also be careful in the design of these systems where there are certain things which are [00:22:00] sort of irreversible actions.
[00:22:01] Rhishi Pethe: Like if you spray something, you cannot take it back, versus something else which is reversible. A-And tailor your recommendations through the AI systems based on understanding really the context of what is the real risk of taking that action. we– human beings, we are just really bad at evaluating uncertainties and risks, and that’s why we’re having this conversation,
[00:22:24] Rhishi Pethe: right?
[00:22:24] David Bergvinson: Yeah
[00:22:24] Elliott Grant: let me, build on this. Sorry, David, I’m gonna jump back in here. But I, I’m–
[00:22:28] Rhishi Pethe: Yeah
[00:22:28] Elliott Grant: don’t think it’s gonna stop at the, uh, at an AI agronomist. I think we’re gonna run straight into agentics, meaning that the model will take an action on your behalf, and we’re seeing this already happening in, in our world.
[00:22:41] Elliott Grant: Um, w- I don’t think pe- the, the world will say, “Here’s the advice, you know, take it or leave it. Use your judgment.” I think we’ll see, you know, this will automatically load your sprayer. It’ll automatically order the chemical, and that will take the, the human even further out of the loop. What, what do you think, Ruchi?
[00:22:56] Elliott Grant: Do you think that’s sort of the slide towards, um, an even greater[00:23:00]
[00:23:00] Rhishi Pethe: I mean, yeah, de-definitely. I, I think right now at this stage, it is very much advisory in nature. Like, hey, this is what the AI agent thinks you should do or the AI bot thinks it needs to do. In the future, we will offload some or a lot of, uh, those decisions. I think how we build those systems, um, will again sort of play a role, right?
[00:23:22] Rhishi Pethe: Like when you offload something, you know, start with something which are reversible tasks or something which is low risk and slowly build towards, you know, like don’t jump to ordering chemical and spraying your field immediately, right? yes, some people will do that. That will be unavoidable. Uh, but I, I think how we need to think about designing these systems.
[00:23:42] Rhishi Pethe: I’ll give an example from– So I, I use this app called Muse from Facebook. Um, and you know, it is so good. Like, if I want to add something to my calendar, I just talk to the thing. It, it is connected to my Google Calendar, and it just adds it, right? Like that’s a low-risk item. Like if it’s messes that up, [00:24:00] okay, I might miss a meeting or whatever, right?
[00:24:02] Rhishi Pethe: Like it’s not the end of the world. some slightly higher, uh, stakes item I’ve tried is like order a birthday cake for my daughter’s birthday, right? Like if that messes up, I’ll be in trouble. So I’m gonna like really look into like, did it actually do it or not? And so I think that’s how we need to think about these things.
[00:24:18] Rhishi Pethe: Like what is the action that you are offloading to an agent, which I agree with you, Aditya. Like I think more and more is gonna happen, um, and there’s a lot of benefits to it. but we’ll have to keep track of, you know, what are the risks of taking those actions? How many of them are reversible? slowly build towards it, right?
[00:24:38] Rhishi Pethe: Um, you know, don’t just jump to doing heart surgery using the AI agent. Like that’s not gonna go well.
[00:24:44] David Bergvinson: So, so as you did, you know, the research, did you see certain countries be very proactive and s- and offer a, you know, positive use case in policy or guardrail development for responsible use [00:25:00] of AI, especially in the agriculture sector? Is there anyone that’s really way out there ahead of everyone
[00:25:05] Elliott Grant: I think we’re– I think this is still very early days for the use of AI agents. I think this is probably an example where governments are still playing catch up with things that are gonna show up in the market. So, uh, in fact, part of our goal of writing this work was to give a– sort of bring everyone up to the same level of what are the things we should be thinking about.
[00:25:23] Elliott Grant: I know I didn’t do a comprehensive survey of all government policies. Maybe there are some countries that are ahead, but I mean, spending a lot of time in the US and now I’m in the UK, I don’t– I’m not aware of any policies around the use of AI and around agronomy advice. It’s sort of a caveat
[00:25:39] Rhishi Pethe: Yeah, I think right now it’s– Yeah, I think they’re more excited about how do we get more adoption. Um, like for example, in India, right? Like the ratio of your extension agent to farmers is very wacky, right? Like recommended is one to thousand or one to eight hundred, depending on, you know, what type of cropping system you’re in. There are like one to five thousand or one to ten [00:26:00] thousand, right? So you basically don’t have any human– shouldn’t say don’t have. It’s very limited on what you can access. And so the potential of these AI base, whether it’s Digital Green or whoever, right, to really bring down the cost of delivering this advice, uh, and the approach that a lot of these countries are taking is like, let’s play with this, let’s start getting the benefits, and we’ll figure out the risks as we go along, right?
[00:26:24] Rhishi Pethe: Which probably not a bad idea, but to Elliot’s point, the reason behind this paper is to say, “Yes, you are going down that path, but, you know, just pause for a moment and think about these things,” right?
[00:26:35] David Bergvinson: Right
[00:26:37] Rhishi Pethe: some mitigation criteria or some process solutions, along the way so that you don’t run into those risks.
[00:26:44] David Bergvinson: Well, the economics and scale of AI-enabled advisory is, is extremely attractive. You know, and governments, you know, may further reduce their investment in traditional extension research, for example.
[00:26:57] Elliott Grant: A classic moral hazard, right? Where,
[00:26:59] David Bergvinson: a moral ha- [00:27:00] Right.
[00:27:00] Elliott Grant: this to the AI and, and now we’re missing out on government-sponsored research
[00:27:06] David Bergvinson: Uh, and one thing, you know, I, I’ve come to appreciate, um, and it was in a recent conversation actually, we’ve used participatory approaches for a lot of technology development in agriculture, just given it- its nuance, complexity, and of stakeholders. Uh, like for crop breeding, for example, we often would use variety selection, uh, in a farmer’s field and getting the feedback from farmers themselves. we’re not, in my mind, doing a good job of that right now for AI advisory. Um, and it, and maybe we need to have to relearn that lesson again, but it, it’s shown us repeatedly that engaging end users in product development and design and delivery is so important for success. and, and maybe we need to, uh, have another discussion paper on that topic itself, uh, around participatory approaches of AI advisory services. thoughts on that, and could that mitigate some of the risks that you [00:28:00] highlight in the paper?
[00:28:02] Elliott Grant: Um, I mean, I think Rishi made the point well earlier, like getting user input early is critical. I think the question I haven’t quite figured out for myself is who will provide these services in the long run? There are several potential providers, right? The tr- traditional services in, in certainly in the Global North would be the seed companies or the chem companies, right?
[00:28:21] Elliott Grant: They have very clear vested interest in having this be bundled with the products, or maybe it’s the equipment companies. Um, Global South, it’s not obvious that that’s who it’s gonna come from. Maybe it’s gonna come from a state actor. Maybe it would come from the telco provider, right? We’ve seen a lot of interest from telco providers for whom agricultural users are a very attractive subset.
[00:28:44] Elliott Grant: So I think to answer the question about how do we make sure that the use- end user is engaged, I think we have to understand who is the provider of this and what’s their incentive. You know, and, and I think without knowing that, or maybe it’s a nonprofit or, or startups like Digital Green that do this. [00:29:00] Um, there’s so many different possible providers, it’s kind of hard to see how it’s gonna play out
[00:29:07] Rhishi Pethe: Yeah, I think given the, just the number of farmers that this needs to get to, it’s in billions, literally millions. I don’t think there’s gonna be like one way. It’s not like telcos are gonna do everything everywhere. Like, I think we’ll have different models depending on which country, which district even we are talking about, right? I, I, I think this is something that can come through the, like, benchmarking or evaluation efforts or maybe the state actors can set standards on you have to go through this process before you can deploy these solutions to our citizen farmers, and that would include evaluation. Have you, you know, part– Have farmers participated in this approach? What is your plan to get data continuously on an ongoing basis, uh, in terms of ground truth data? So that could, you know, solve or, or mitigate some of the problems that Elliot mentioned in the paper around sort of model drift or [00:30:00] model autophagy, you know, where it sort of, know, eats, drinks its own Kool-Aid, the model.
[00:30:05] Rhishi Pethe: Um, and so I think there are different approaches, um, and depending on the situation. But I do believe that the state actors have a role to play here in terms of setting some of the rules, because just given the number of the percentage of population in these countries which, who are farmers, state actors have a vested interest to make sure they do right with those people.
[00:30:28] Rhishi Pethe: Uh, whether they do it or not, that’s a different story. Whether for political reasons to get elected or to have a, have a constituency which supports whatever they’re trying to do, they need to do the right thing with the farmers. So that gives me some hope that some of these state actors will do the right thing these solutions are deployed.
[00:30:44] Elliott Grant: Uh,
[00:30:45] David Bergvinson: I-
[00:30:45] Elliott Grant: when you were talking it made me think of some- something else. Sorry, David, to cut in again, but which is there was some data that just came out from McKinsey that showed, I wanna say like 17% of farmers worldwide have used AI tools. Do you see– have you seen this one, Rishi?
[00:30:59] Rhishi Pethe: Yeah
[00:30:59] Elliott Grant: There [00:31:00] was a subcategory within that which was, are they using dedicated tools?
[00:31:03] Elliott Grant: Are they using sort of off-the-shelf generic tools? And the vast majority of farmers are using ChatGPT, Gemini, wanna applaud maybe, maybe, um, Kimi things. And I think this is something we’re not taking into account, that people are not necessarily gonna go to an, a gen– you know, an AI agronomist for this advice.
[00:31:22] Elliott Grant: They’re gonna, they’re gonna be asking their, what was it called? The muse, right? They’re gonna go to their muse. Well, these, these are not easy to regulate, right? So this– I do worry that the, the front door for agricultural advice is not gonna be the way we think it is. It’s gonna be s- s– the same tool you use to order your shopping or get directions
[00:31:41] Rhishi Pethe: Yeah.
[00:31:42] David Bergvinson: Yeah, that’s a good point
[00:31:43] Rhishi Pethe: think that is probably… I, I don’t know if it’s very different than people getting access to Google Search, right? Or, you know, they’re hearing something from Facebook. So yeah, I, I don’t know how to sort of approach that, but I agree with Elliot that I think the front door is not always gonna be– or [00:32:00] maybe in most cases it’s not gonna be like a, an app or something that is specifically designed for agriculture.
[00:32:05] Rhishi Pethe: It’s gonna be your Google or your Gemini or whatever, right? Like… yes, those will be very difficult to regulate, but those companies hopefully have the incentives to, you know, make sure they, they provide the right, at least basic level of information in terms of, um, you know, what, what an actor can do or a farmer can do
[00:32:25] David Bergvinson: there’s a wide spectrum of users, right? You have very progressive farmers who are using these frontier models for developing their own production systems and back end. And then you’re gonna have the other end of the spectrum where they’re a recipient of a ag advisory service that is delivered through either government or private, uh, extension or market service. Is there a need, um, to have a registry of risks as they are evolving and put sort of like a traffic light system on it where red is something that is eminent and going to be, you know, very impactful, [00:33:00] uh, and, you know, amber just keep an eye on it and, you know, green, you know, maybe just saying, “We’re watching it, but it seems to be going okay.”
[00:33:07] David Bergvinson: Is there, know, a need for that kind of system, or is that just… Uh, th- is it just too complicated? You, we just can’t create such a dashboard of, of tracking risks and helping governments get ahead of them?
[00:33:22] Elliott Grant: I always worry about putting in systems like that, that’s like very quickly it just loses, it just gets out of date, right? This is such a fast-moving space.
[00:33:29] David Bergvinson: Mm-hmm.
[00:33:30] Elliott Grant: It’s a good point, like how do we sort of maintain attention to this and keep it up to date? I don’t– I can’t think of a very elegant mechanism that I’ve seen elsewhere.
[00:33:39] Elliott Grant: Maybe the analogy is AI risk, right? Every year you have a conference somewhere in the world, and the good and the great show up, and they sort of say, “Where are we in our, you know, in our path?”
[00:33:53] Rhishi Pethe: Yeah,
[00:33:53] David Bergvinson: Okay
[00:33:53] Rhishi Pethe: I’m not a big fan of those re- red, yellow, green dashboards because to that extent people [00:34:00] start losing interest. also the incentive of somebody who is– It’s like airport security, right? Things never go back, they always get worse, uh, because the, the risk of saying, “I’m gonna take something off the table,” and then something bad happening, like nobody wants to take that risk, right?
[00:34:17] Rhishi Pethe: So if you say it’s green, incentive is to always keep it at red or yellow because like, let’s keep everybody on their toes and it’ll cover my ass, right? Um, so I, I’m not a big fan of those systems. I, I think with these type of conversations that we’re having, these papers, people sharing their experiences, and other nonprofits having more of these conversations and learnings that are shared. As long as we keep this in the conversation at some level on, on some regular frequency, I think that could be a good mechanism to, to, try to, you know, keep an eye on these risks and talk about what are the leading indicators which tell you that something,
[00:34:58] David Bergvinson: Great
[00:34:58] Rhishi Pethe: is drifting, right? Like, [00:35:00] um, and tracking those indicators versus having a red, yellow, green light signal
[00:35:04] Elliott Grant: Yep. I also am a fan of,
[00:35:05] Rhishi Pethe: could keep an eye on
[00:35:06] Elliott Grant: I’m a fan of economics, which means like how do you make, make a self-regulating system where there are costs to having it go wrong and there are, you know, you’re gonna benefit if it goes right. So I would be more in favor of a system that has carrots and sticks. Like there’s a, there’s punishment and some me- mechanism for punishing bad actors, uh, and then that should drive people into the better, you know, the market should self-regulate effectively
[00:35:32] David Bergvinson: Well, I invite, uh, those watching to, uh, go read the discussion paper in detail so that you can get all the nuances of this topic and, uh, and encourage you to actually contribute to it, uh, by posting comments either on this video or going to AgX AI where you can download the discussion paper and add your comments there. as we wrap up, what, is the biggest takeaway from the discussion paper from each of your perspective as, as a closing remark? So Elliot, to [00:36:00] you.
[00:36:01] Elliott Grant: Um, I think if we kept working on this paper, we would continue adding risks. This is– There is no end to this exercise. There will always be new risks to imagine. Um, and that’s a little troubling that this is not, um, a well-defined problem
[00:36:21] Rhishi Pethe: Yeah, I think my take is a l– maybe on the other side is, yes, it’s great to talk about risks, uh, and we should talk about them. a-a-and based on sort of what I see here in the beta, there just so much discussion about AI doom and, you know, P doom and all that, uh, that is scaring people. Uh, we have all this opposition to data centers, and there are very good reasons for that. I worry that if we talk about risk more than what is necessary, like we may be tamp down on the benefits that are available to that. So I, I see that as a risk, the risk of talking too much about risk, right? I, I worry about that a little bit [00:37:00] if we can get a little bit meta here. So, we, we just need to do it in a balanced way, in a data-driven way, as Elliot has laid out in his paper.
[00:37:07] Rhishi Pethe: Um, a-a-and, you know, s-stay rooted in facts and talk about the benefits as well. It’s a trade-off at the end of the day
[00:37:15] David Bergvinson: It is
[00:37:15] Rhishi Pethe: So we talk about both. Um, but yeah, the risk of talking too much about risk is what I worry about.
[00:37:22] David Bergvinson: Right. Okay. That’s a good point to end on. Uh, well, thank you, gentlemen, for your contribution both to the discussion paper and to th- this conversation. Again, I invite people to go to AgX AI to download the discussion paper on risk, uh, and also leave comments, uh, in this video as well, so we can get back to you. And also suggest other topics that may, uh, be relevant to this topic of risk. Uh, not to be, uh, creating a new risk around risk, but I do think a balanced approach, and thank you, gentlemen, for, for offering that in today’s g- discussion. so thank you, and, uh, onward. It’s a very exciting era. Uh, there’s no innovation that [00:38:00] comes without risk. Uh, it’s how do we manage them effectively so that the benefits of the technology are realized. So thank you both for your efforts.
[00:38:08] Elliott Grant: Thank you. Thanks for having us
[00:38:09] Rhishi Pethe: you for having us
