Week : Executive Playbook: Leading AI and Agents
[jupyter][google colab][reveal]
Abstract:
Capstone synthesis: connect strategy → decisions → data rights → operating model → metrics. Provide an executive playbook for adopting AI and agents with accountability, escalation, and resilience.
Synthesis: what leaders must control
This capstone is a control-point recap. Each item corresponds to an “executive lever” that limits failure at scale: define objectives, make uncertainty legible with thresholds, build trust infrastructure (rights, recourse, auditability), and operate with an explicit model of roles and escalation.
See Lawrence (2024) objectives p. 29, 36, 83-4, 148, 149, 179. See Lawrence (2024) topography, information p. 34-9, 43-8, 57, 62, 104, 115-16, 127, 140, 192, 196, 199, 291, 334, 354-5. See Lawrence (2024) trust p. 43, 79, 100. See Lawrence (2024) System Zero p. 242-7, 306, 309, 329, 350, 355, 359, 361, 363, 364. See Lawrence (2024) accountability p. 352, 363. See Lawrence (2024) intelligent accountability p. 363-4. See Lawrence (2024) automation p. 6, 24, 46-7, 77-8, 80-81, 83, 85-87, 363-6, 368-369.
Agents: capability with constraints
Treat documentation as a control surface. Keep “breadcrumbs” that connect WHY → WHAT → HOW → DO so that decisions can be challenged, reviewed, and unwound. If you can’t trace intent to action, you don’t have governance — you have hope.
30/60/90 day playbook
Myths to avoid (optional wrap)
Five AI Myths
- AI will be the first wave of automation that adapts to us.
- Hearsay data has significant value.
- The big tech companies have the landscape all ‘sewn up’
- ‘data scientists’ will come and solve all problems.
- The normal rules of business don’t apply to AI.
The five AI myths are patterns of thinking I’ve identified amoung those that are trying to take advantage of artificial intelligence to deploy new products.
The first myth is the “promise of AI” myth, that AI will be the first wave of machine-based automation that adapts to us, rather than us having to adapt to it. The reality is that we haven’t yet created machines that are as flexible as humans, the automation we are producing is still ‘fragile’, in that if it encounters unforeseen circumstances it breaks. This is a consequence of the way we design systems, flexible natural systems such as ourselves are evolved, not designed. And evolved systems have a first priority to ‘not fail’. What we think of us ‘common sense’ in the human is in reality a set of heuristics that prevent us doing stupid things in the name of achieving a goal. Our AI systems don’t exhibit this.
The second myth is that there is value in ‘hearsay data’. Hearsay data is data that people have heard exists, so they say it exists (See this blog onblog post on Data Readiness Levels. (Lawrence, 2017). The failure to understand the importance of data quality is resulting in unrealistic projects staffed by people with the wrong skill sets. Most decision makers don’t understand that implementation of a machine learning model is relatively trivial. But preparation of the data set and the data ecosystem around the model is extremely difficult. So the wrong investments are made, millions spent on recruiting machine learning PhDs and minimal spend on data infrastructure and systems for data auditing.
The third myth is that platform effects mean that there is no room for knew innovation in AI. Three factors will prevent the platforms dominating in the long term. Firstly, they are not agile, their approach to AI software development is grounded in the world that pre-dates wide availability of machine learning systems. Agile software development needs revisiting in the context of machine learning and this form of cultural change is difficult to achieve. In practice, these companies are larger than they need to be to deliver their services because they can afford to employ people to handle operational load. Newer agile companies will develop a better culture around data and machine learning. One that requires less operational overhead. This doesn’t just reduce costs, but it increases speed of movement and develops better understanding of the underlying systems. See this blog on blog post on The 3Ds of Machine Learning Systems Design..
The fourth myth is that soon there will be a wave of Data Scientists who will be equipped to enter companies and resolve their problems around data and AI. The mistake here is to assume that these graduates will have been trained in the necessary skills to do data science within a company. In fact, Universities will naturally focus on algorithms and models, because that material is teachable. Much more important is systems thinking and data wrangling. Processes to ensure that data is actionable. The weakness of senior decision makers, including CIOs and CTOs is that they don’t have a deep understanding of the technology, so they don’t perform critical thinking in this space. It’s a problem that can be deferred and solved by a mythical set of experts who will soon be arriving. In reality, domain expertise is key to successful data science, and bridging existing expertise with an understanding of the new landscape is far more important to delivering succesful systems.
The final myth is perhaps the most perniscious. It involves a suspension of normal business skepticism where AI is concerned. It may arise from the use of the term AI, which implies intelligence. If these systems were really ‘intelligent’, in the way a human is intelligent, plus if they had the skills of a computer, that really would be revolutionary. However, that’s not what’s happening, and won’t happen in the foreseeable future (i.e. on timelines that matter to business). In reality this is an evolution of existing technology, and it has the ususual challenges of adpotion that existing technologies have. The challenge for decision makers is how to assimilate the implications of this new technology within their business skill set. This means familiarisation, and doing courses etc isn’t good enough. Senior business leaders need to take time out working closely with the technology in their own environments to better calibrate their understanding of its strengths and weaknesses.
Recommendation: How to bust to these myths? The primary recommendation for businesses is that they start pilot projects which have executive sponsorship. They involve the CTO (or CIO or CDO), a technical ‘data science’ expert and a target domain area. Instead of feigning knowledge in this space, each admits their own ignorance of the other domains, and starts from scratch. Egos are left out of the room. The small pilot project is explored and delivered with the real challenges being noted. In this way each of the individuals will learn quickly where the pitfalls are.
One challenge is that for most projects the data will be too poor to even conduct the pilot. However, one data source that is consistently of good quality across companies is financial data. So a further suggestion is to initially focus on collaborating with the CFO and focus on financial forecasting (or similar). If the CFO, CTO and CEO gain a better understanding of the capabilities of data science, then the company can begin to turn around its systems and culture focussing on the important changes, making calibrated changes, rather than reacting to the sensationalism around AI.
Importantly, don’t go all in. Major companies are susceptible to what I call ‘(Grand Old) Duke of York Effect’, march 10,000 people to the top of the hill and march them down again. Command and control is not the right response to an uncertain and environment. Don’t think like regular troops, think like special forces, small groups with specialist expertise that are empowered to think independently and explore the landscape. Find which hill to march up, before committing significant resource.
Thanks!
For more information on these subjects and more you might want to check the following resources.
- company: Trent AI
- book: The Atomic Human
- twitter: @lawrennd
- podcast: The Talking Machines
- newspaper: Guardian Profile Page
- blog: http://inverseprobability.com