.TITLE Persuasive Language for Language Security: Making the case for software safety .ABSTRACT Cyber Grand Challenge was a DARPA program that proved the feasibility of real time conflict between hacking machines. It was also a live demo, a moment, and a symbol. If LangSec is as Buckminster Fuller instructed not to struggle against a problematic model but instead to build a new one, then symbols, demonstrations, and ways to communicate the power of a new paradigm for software construction must be employed. A collection of narratives and lessons on the subtle task of persuading power, authored by a hacker who spent four years in Washington attempting to change the world. .TEXT In 2013 I made a life altering choice. - I decided to leave the safe confines of my research lab where I happily went to work every day and talked to fuzzers and debuggers and assemblers, and worked in the company of other people who thought, like I did, that talking to software was a perfectly normal thing to do - and I accepted a role as a DARPA program manager. Now the term PM can be a bit misleading bit of language- more on that later. PMs at DARPA serve a limited term of about four years and the role is, quite simply, to conceive of and build a project in less than four years from scratch with the help of two tools: a travel agent and a research funding budget. When you're done, if you've done anything less than invent the Internet you get a B, and win or lose you're fired. There are a lot of things you could call that job but PM doesn't quite capture it. It was an experience equal parts terrifying, demanding and rewarding. Now I could tell you a lot about that experience, but I'm here today at LangSec 2018 to talk about just one facet of the job: communication. Because if there's anything the DARPA experience offers, it's the opportunity to talk directly to powerful people about technology. The Department of Defense, Congress, and the executive branch all have the opportunity to speak to DARPA's technology experts and can consult with them about technology, risk, and national security. Now in 2013 I was convinced, as I'm sure many are, that once I could talk to a few policy people I would have the whole thing sorted out by lunch and have time left over to change the world. It would be simple! Sure, security industry twitter is its own bubble shaped echo chamber that even reporters won't stick their heads inside that seems to be filled with people responding to press releases by inspecting the cipher suites supported on the server hosting them - "why should I take you seriously if you're still vulnerable to BEAST?" - but I was reasonable, and I was an expert, and I knew how to be polite. And there seems to be such a tremendous gap between where we are and what must be done to ensure the safety of connected civilization that I was possessed of the belief that I could bridge the gap. How hard could it be? for instance, if you want to build backdoors into crypto, first build an unpickable TSA compliant lock for your luggage just to prove the principle is sound. That argument should convince anyone! The conversation, as I'm sure you've guessed, is not so simple. There is a tendency, inside the security bubble, to think of policymakers as aliens, possessed of a foreign communication medium and possessing a flawed logic that simply cannot be reasoned with. Nothing can be further from the truth. Policymakers are brilliant, rational actors. They understand their world, and we need to understand it too. LangSec, as a technology movement, is interested not in the safety of software but in the safety of a world that depends on software. The goal is not to design and publish safe stacks, or to design and publish tools for creating safe stacks- the goal is to publish these stacks to a world that will take them up as defense and in so doing be granted immunity to an entire spectrum of silent malice. The goal is to build a safer world with technology. I for one, am a believer in software safety. I have become convinced, during my time in Washington, that the time will come when the world will turn to its technologists and demand verifiably safe software. I believe that day cannot be stopped, and that when it arrives LangSec will be one of the actors in the spotlight. And while you may take this as good news, I will note in passing that in some cultures, the following is considered a curse: "May you come to the attention of powerful people". When you do come to the attention of powerful people, it helps to be prepared. Today you are going to hear a series of deeply technical talks- building secure parsers, retrofitting security over dangerous parsers, detecting and exploiting dangerous parsers, keeping dangerous parsers out of SCADA networks. This morning I'd like you to invite you to launch a background process that you keep running all day: keep asking yourself why this matters to the world. If any work is to make the leap from novel to meaningful, it must be accompanied by persuasive arguments- arguments made not just with language, but with stories and shared moments. Having made the case to the nation that building a globally crowdsourced cyber conflict between AIs was both novel and meaningful, I hope that the things I learned may be of use to you as you make the case to replace today's shaky pile of scattered recognizers with machine-built, machine-verified foundations. I would like to tell you that my time in government was a series of deft communications steps and that in a few short briefings I convinced the world that software safety was a global imperative. The truth is that in four years I made many communications missteps, and that over time I made the most progress and gained the most value not from what I said but what I heard. So I will relate to you some of these conversations, and some of these failures, in the hopes that they will serve you well. One of the amazing things about being a PM at DARPA is that you never know who's going to walk through your door. People with an active role in running the country will occasionally drop by, or be shown around, and suddenly you're having a conversation that could make a difference. After the first time this happened to me, I decided I would put a conversation starter on my door- I had just the thing. I had found a web site named "World of VNC" - I'm sure you've heard of it - and what it contains is thousands of screenshots of open VNC terminals that are connected to the Internet without a password. It's easy to connect, and so someone had. And I had clicked through all these unprotected systems until I found a SCADA system that controlled an animal farm, with hundreds of automated feeding systems and cages. You could feed pigs, or let them loose somewhere, it was all mouse clicking - quite remarkable. And so I printed it out and put it on the door, in full color, and used it to start conversations. And what I found was that this image was a Rorschach test. When computer security people saw it, they would put their head in their hands and shake their heads and say "I know, something terrible is going to happen and we can't stop it". But when someone from outside the field would look at the image, I would tell them what it was they would look again and stare through it like it was made of glass, and then they would shrug. One day, someone incredibly powerful stopped by my office, and I explained the image, and this leader laughed and said "That doesn't worry me at all! Why would anyone want to do that?" I was so baffled that I couldn't respond, and they walked away. With time, I've come to understand what was happening here. The difference is that we possess in our minds several constructs that are unique to our field. The problem with becoming a domain expert is that as you gain expertise it becomes intuition, and as you go deeper into the development of your expertise you forget what you have learned. The thinking becomes automatic, and we become blind to the distance we have covered from the understanding of everyone else. So here are some of the leaps you've made listening to this story without even thinking about it. First, you understand the scale of the Net, to the extent any human can imagine something so vast. You understand that where there is one such service, there is an uncountable universe of similar potential disasters. You probably remember in the 2000s when the first Windows worms showed up on ATMs, and imagine them today showing up in pig farms, helium plants, beer breweries, mars rovers and refrigerators all at the same moment. Second, you understand the incredible power and accessibility of scripting. You know that it is a night's work for a computer literate person to cobble together something that can connect and click randomly on unprotected services. You've probably thought through the first set of technical challenges right now, before I've finished this sentence, and are already counting the cloud nodes needed to make it scale. *you think in code*! While we've been told our whole lives that this makes us valuable, it's worth mentioning that it also deprives us of a certain communication intuition. Finally, you understand that no specific malice or intent would be required to kick off thousands of disasters like this, each a unique snowflake of chaos. Those of us who've been on the receiving end of worm attacks understand, intuitively, the destruction wreaked by unintended consequences at scale. Random action in control systems would fail to cause disaster thousands of times, but when you take scale and multiply it by random action the result is disaster. Someone could easily write such a script, or hundreds of possible variations on this theme, just for kicks. In fact, a few months ago, someone did - an anonymous author laid out the few lines of python required to attach Metasploit to Shodan, enabling anyone who can download a GitHub python project to start gaining remote control over any scannable unpatched device. They called this project "autosploit". Now, this shouldn't surprise anyone, right? It was implied by the image on my wall, we had all the evidence needed to prove such a thing was possible. One weekend of scripting work doesn't change that. Now, I'm going to read a tweet from Katie Moussouris on the subject: "The danger of shaking one's fist at the release of a simple-to-create tool like this is more overregulation of these tools, like the original intrusion software language added to Wassenaar that we spent years undoing. Stop overreacting. Policy wonks & legislators are listening." Katie of course is one of the few deeply technical compsec experts who has decided to spend her career attempting to bridge the gap between the alien worlds of policy and technology. So when Katie says "legislators are listening", she's not speaking speculatively, she's speaking about her circle. She means that the incredibly obvious act of tying a scanning service to a pentesting service could result in real legislative action. How is it that something so obvious to us creates disbelief followed by reactive panic? Simple. If you show someone that the door to the pig farm controls is unlocked and say that it's an example of disastrous unpreparedness, they will laugh at you and tell you that no one wants to attack a pig farm. And that's because they're thinking about a door, and a lock, and a conscious and planned act of malice. Policy makers spend their days thinking very seriously about actors, threats, and acts of malice, and they cannot take anyone seriously who thinks that a planned attack on livestock fits in somewhere in the threat hierarchy. They are trapped in the confines of the analogies used to explain our world to them- and whose fault is that? This is our communication failure. What is the real world equivalent of a cloud node and a scripting language? That's simple - it's a free supply of teleporting robots. With a few lines of python you can knock on every door on the planet and put "admin/admin" in the lock. The folks in Washington are still grappling with a world where every citizen has access to an unlimited supply of teleporting robots. We here have long ago stopped questioning the existence of the robots- we summon them daily by typing "import sys". This was the beginning of my understanding that the policy thinking within the conflict domain of computer security is dominated by thinking about planned, intentional malice rather than considerations of the collective risk of randomized disaster. This is a real divide; you will find it again and again. What it creates are Rorshach Tests: both sides stare at the inkblot, one side sees a butterfly and the other side sees a butterfly that has been stepped on. Let me give you an example. V2V is a technology that allows cars to communicate, digitally exchanging position and vector information; when two cars are on a collision course they can compute the collision without seeing each other and either alert the driver or autonomously take avoidance action. This digital exchange of vectors will take place through constant vehicle-to-vehicle exchange of vector information signed and secured by the X.509 certificate industry standard. Now as I look around the room, and I gauge your reaction, what I see is people who see a butterfly that has been stepped on. Because what we all heard is that the cars of the future are going to be part of a peer-to-peer digital mesh network that transmits, receives, and parses X.509 certificates. We don't think a lot about the safety history of cars, but we think quite a bit about the safety history of X.509 certs, and what we know scares us. Now if you believe that parsing complex, nested binary formats is dangerous, that the uncomputable complexity of ASN.1 Basic Encoding Rules creates unnecessary danger, that peer-to-peer communications without central inspection or forwarding is a basic requirement for self-replicating code, and you intuitively understand that the worst teleporting robots can clone themselves, then you can in one breath summon to mind the sum of all fears: a widely used implementation, memory corruption, a worm, and a nation of vehicles afflicted with a control systems virus. What the other side heard was that thousands of automotive collisions could be wiped out by a single technology based on a tried-and-true industry standard, defended by best practices that work every day to keep the Internet safe. Our concerns are theoretical; theirs are governed by real lives being lost right now. And if you speak only to the perceived technological weakness, the other side of the table will immediately want to know who perceives this weakness, what their effort to act is, what their motivation to act is, and what deters them. When you possess a threat model dominated by intentional malice, you think about people, not bugs. And if you push on this divide, you end up having what is essentially a deadlocked conversation. I have had this same conversation with senior technical people and senior policy people again and again. It goes something like this. I would go to visit a very senior technical person, someone who has worked in the field of computer security for decades, and they would listen to my optimism, think a bit, lean back and say, in hushed and somber tones: "Something terrible is going to happen and they won't listen to us until it does." And so I would nod my head, and absorb this, and go talk to a senior policy person. They would listen to my pessimism, making a face like they had heard these arguments before, and shake their head to correct me. And then they would say: "These cyber disasters you are worried about cannot happen, because if it was that easy someone would have done it already". Now I've had these same conversations so may times, with so many brilliant people, that I have come to the conclusion that both sides are correct and that our perceived disagreement is simply a matter of timing. Don't think about that too hard. The point here is that we must build a bridge to the other side, and that our tools to date have been composed primarily of language. Language is a dangerous and imperfect tool; we need something better. The control system vendors tell us that our concerns are null and void; we say yes, exactly, we insert a null byte into the certificate and overwrite a void pointer. Ok, it's not really that bad. Except sometimes it is! Cyber Grand Challenge was the world's first all-machine hacking tournament, a competition between autonomous systems. Now I gave it that name because it was the only possible name. In the Defense Department "Cyber" is the fifth conflict domain. If your travels take you to the right places you will always see the five words printed on large metal signs hanging from the ceiling: Land Sea Air Space Cyber. It's an ordered list. These are the places the safety of the country is defended. Cyber is a domain; it is a contest to control the logic and memory of networked computers. Everyone knows this. Machines that could live and fight in such a domain were always going to be part of a Grand Challenge in the Cyber domain. Now when I was walking around the DARPA Robotics Challenge in street clothes I had the chance to talk to a lot of smart people; one of them was a young roboticist from MIT who was taking a debugging break and we just started talking about the competition network. So I brought up hacking contests and I asked this young man what the word Cyber meant. He told me that cyber was a word used exclusively by people in government to let everyone know that they didn't understand how computers worked. I think maybe he was on to something. I think this definition is still universally accepted in the hacker community. The word "Cyber" is yet another Rorschach test: everyone looks at it and sees something different. Our field is filled with these. They exist at every level of dynamic range; if you find two exploit authors arguing about ROP versus Return to LibC you can always rush in and tell them that you use the Geometry of Innocent Flesh on the Bone. Too obscure, right? At DARPA the rules about language are simple; they're named Heilmeyer's Catechism. Rule Number One goes like this: "What are you trying to do? Articulate your objectives using absolutely no jargon." Absolutely no jargon. Jargon is the fundamental building block of our field. When we do our work well we get to create new jargon; we call this novelty. Here are some examples of real conversations I had in government. After a talk during which I was challenged to explain the difference between dynamic and symbolic execution, I was taken aside and counseled to stop using the term "concrete input" because construction references would confuse people. I was informed that a network monitoring approach was so effective that it continually discovered zero-day malware. To this day I don't know what that means. While I caught these instances, the ones that haunt me are the ones that slipped by me unnoticed; conversations filled with Rorschach blots where words spoken by one party constructed a completely different meaning in the mind of the recipients. In a culture where everyone's an expert, nobody can ask for clarification. These are little stories about the imperfections of language, yet my assertion is that language is imperfect and dangerous. The danger lies in the summoning power of words. Reagan said that "if you're explaining, you're losing" and truer words were never spoken. The power of that statement is that no one ever needs to explain it, which is telling. You'll find that in places where power is aggregated and used, there is an enormous focus on the economy and precision of language. Your public words will be handed back to you filled not with objections to what you are saying but rather with objections to what people will think you said. A historical wasteland of blowback craters has taught the immune systems of government never to write a long letter when an empty one will do. This is an important lesson, and it would have served the hacker community well. Unfortunately, at some point, people who wanted their work to sound expensive started describing some very boring programming and QA work using the rhetoric of war. You can open a newspaper now and read about arms dealers who traffic in "fully-weaponized nation-state zero-day exploits". If you're running a public survey entitled "Things we should ban", that phrase is one of the few things ordinary people will pick over dihydrogen monoxide (nasty stuff by the way, it can suffocate you). If we'd been listening to Katie I think maybe these words could have been abandoned sooner, but it's far too late. We've summoned the myth of a dangerous, single-use good into a reality in which all tools are dual use. We could, of course, have published a cyberwarfare secret decoder ring. A Proof of Concept, see, means "it works on my machine". Fully weaponized means "it works on your machine", "nation-state" means overpriced, and so on. Instead we're left attempting to claw our way back to the actual technical reality that underlies our field, crippled by language that has summoned myths into the room. I had the unfortunate opportunity to do exactly this in 2015. I was party to a few conversations about some new arms control language being adopted by the US and its impact on research funding. As I experimented with ways to talk about this subject, I settled by trial and error on an approach where I would relate a story from my career. So I'll relate this same story for you now today. Around 2004 the global internet was struck by a worm that contained a remote exploit – a Von Neumann UDP probe, if you will – that came to be known as SASSER because of the name of the process it knocked over. I worked for a pentesting team that developed automation the day my company was struck by Sasser. My company was a global company with a Class A network and tens of thousands of globally distributed computers. The worm crippled us in hours. My team and I hatched a plan to scan our entire class A using our toolset, automatically throw the same exploit as the worm, get a shell on computers in our company, and install the patch. We got authorization to execute this plan, executed it, patched thousands of computers all around the world, and ended the worm outbreak on our network without a single administrator-installed patch, put a massive organization back in business, and got home in time for dinner. To do this, we built a framework which handled and used remote exploits and launched them against thousands of computers all over the globe. This framework engaged in global automated exploitation for the purpose of authorized access. I literally got to hack the planet- a fact I only realized years later. So, that's the story. I would tell this story, and the folks on the other side of the table would pause a bit, then they would say "so you attacked your own network"? And I would say, "no, we defended our network with a tool called an automated exploitation framework". And then they would look again at the proposed ban with new eyes. Language may be imperfect and dangerous, but it's at its best when it's telling a story. Over time, when speaking to power I became a student of stories composed of language that expressed only core truths. I began to describe computer attacks as "programs introduced inside programs", because "weird machine" tended to lose my audience. And when I listened to Meredith tell the world that if software was secure then its exploit was simply proof of vulnerability, I scribbled those three words on a piece of paper and wrote them into the founding Rules of Cyber Grand Challenge. I hope today to return credit where credit is due, with interest. Now I could explain this language choice easily: an exploit, free of context, is a word that will make people think it's an attack; banning exploits and their containers is simply banning the act of attacking. A proof, free of context, is scientific evidence, and banning proofs is an attack on science. Of course, I would never explain this, because if you're explaining, you're losing. Let's stop losing. No one should attack science. If we describe our complex, dual-use landscape of information/code with single-use language, we will summon mythical monsters and see them used against us. The solution is clear. Rewrite the cyberwarfare secret decoder ring, using absolutely no jargon. There are no attack or defense tools. Describe powerful things with simple, precise language that expresses their core truth. No one in power should ever be put in a position where they act against something whose true nature was hidden from them. This is our communications problem, our failure, and we can solve it. And of course, many are trying. Andrew Myers wrote The Wooden Firehouse, a brief allegory for the current state of software safety. He compares the case for secure software to the case for fire resistant building codes, and there of course he is on to something. In the case of fire prevention we have an example of a functioning safety triangle, with three points: Producers of materials, who compete to comply with codes, Consumers of materials, who are liable for compliance with codes, and independent assessors, like NIST, who do research, study the science of fire, and publish information about how buildings burn. To this day NIST continues to learn more about the science of fire dynamics, fire forensics, and fire protection. Note that all three points of the triangle are independent: producers, consumers, and assessors. This is just one functioning triangle. With cars we have another, with producers competing on a scale of one to five stars, consumers purchasing the level of risk they are comfortable with, and independent assessors awarding stars and publishing videos of vehicles hitting concrete walls. With software we currently have no such triangle. Assessment cannot live, as it does now, with the producers or the consumers of software. With software producers we have conflict of interest; with software consumers we have an inability to collectivize resources. With all parties we have an inability to marshal uniform testing for vulnerability. This would change, of course, if we had an independent software safety lab that had access to automated hacking machines. Many of you know of Sarah & Mudge Zatko's founding of the Cyber Independent Testing Labs- Cyber-ITL. Their vision- to publish open, accessible risk ratings backed by a standardized methodology- may someday provide the sunlight necessary to differentiate risk levels in software that parses inputs. They're well on their way. While they started out with DARPA funding, today the CITL is a financially independent non-profit dedicated to open software risk rating. In a post-CGC world, the day is near at hand when automated hacking technology can rate software based on its vulnerability to a real, thinking, automated hacker. Imagine informed consumers walking through the IoT aisle, looking at safety ratings. That is a world in which LangSec could shine. I was often asked: what is the purpose of automating hacking when we can simply build software correctly instead? There were many reasons to pursue integrated automation, but to this question my answer is simple: automated exploitation machines act as illumination. If we think of security as resistance to the pressure of an adversarial mind, then hacking automation provides uniform, repeatable application of pressure. Strong performance against such pressure creates evidence. Evidence-based arguments change the world. The world sees what happens when cars are slammed into concrete. Seeing is believing, and the world today does not believe in secure software. Consider for a moment the consensus of disbelief you are up against. I will read two quotes to you now: one from the media world, one from the hacker world. First, Dave Aitel, posting to his Daily Dave list, in a post that was the beginning of the revolt against the phenomenon of stunt hacking: "your blackhat talk was not accepted!" "Cars are the pinnacle of junk hacking, because they are meant to be in your garage. Obviously there is no security on car computers. Nor *will there ever be*. enough with being amazed when people hack their junk" Second, David Pogue, a technology reporter, writing for Scientific American in the wake of Wired's coverage of Charlie Miller's vehicle hacking demo: "Why Car Hacking Is Nearly Impossible. Scary hackers can take over our cars! Our lives are at risk! No, they're not." Now at first blush these sound like diverging opinions: it's trivially easy to hack cars and nothing will change, it's nearly impossible to hack a car and nothing will change. Taken in aggregate, this is consensus: nothing will change. No one believes that it is feasible or achievable to build a system known to the world to be secure. No one believes in secure software. Except maybe you. I spent most of my career at DARPA telling various audiences about a bet my team was taking. Our bet was that one day, automated systems were going to download new binary software, reverse engineer it, write exploits, connect over networks and use those exploits, then detect the opposition's exploits, write patches, and defend a network live and in real time. I never heard a word of agreement. Many people have spent a career learning to do one of these things well; the idea of an integrated system managing it all simply did not compute. While all I heard was challenge, each challenge was different. There were a million reasons why not. Taken in aggregate, the message was clear: "prove it". On Thursday August 4th, 2016, seven automated systems fought each other. While I'd like to tell you that there was a consensus on how to build such a thing, an emergent monoculture, the truth is quite different- each entry was a deeply flawed early prototype. Even in the parts of their function that are well established, like bug hunting, each system got very different results. There is no monoculture yet – we're still learning how to do this. But fight they did, in interesting ways. Two systems, Galactica and Deep Red, got into an interesting battle. Galactica exploited some software that Deep Red was running; it ran some shellcode, ASCII-encoded, on Deep Red's stack. In response, Deep Red binary patched its service with a non-exploitable stack. In response, Galactica wrote a new exploit that used ROP, encoded in binary. In response, Deep Red wrote a network filter that dropped a specific part of Galactica's binary payload. The attack was over, and Deep Red's patched, IPS-defended service kept on servicing legitimate requests until it was retired. Quite a contest. It would make sense if it were part of a college CTF writeup. Instead, this was a "flash breach" that took place in the space of 20 minutes, authored and managed entirely by machines. Today the conversation about automated security systems isn't an existential conversation. It's a conversation about when. The winner of Cyber Grand Challenge was installed in the Smithsonian for a summer. While Mayhem rested near the entrance, the Museum of American History was visited by about two million people. A hacking AI, glowing orange, wearing a DEF CON black badge that it had won in battle. Behind Mayhem is a bench, and I spent one day that summer sitting on that bench, watching people watch Mayhem. The American people had paid to develop this technology and this was an opportunity for me to hear what they thought of their investment. One high school class entered the museum and a young man left his pack of friends and began reading the carefully phrased signs that accompanied the exhibit. He watched the video, he looked at Mayhem, he inspected the black badge, the process repeated. It was a quiet five minutes. Finally his friends caught up, and he turned to them. "Guys!" he yelled. "This thing is an AI! And it can hack anything!" That young man was not captivated by the imperfect, dangerous tool of language. He was drawn in by a story, and transformed by a moment. CGC was, of course, a contest and a technology demonstration. It was also a story about the future, and a moment that changed minds. V2V is a messaging standard. Langsec could release a reference parser for the V2V Basic Safety Message, and this wouldn't quite be a story. Langsec could release a permissively licensed, open source reference parser and challenge an independent third party, like CITL, to hack it – and that tale might reverberate within the security industry Twitter bubble. Or perhaps, a reasoning system like Mayhem could be challenged publicly to test a Langsec-developed V2V implementation and every other V2V parsing implementation currently available. You could face the world in that moment and tell a clear story to policy makers and the public using absolutely no jargon: Hello, this is LangSec. The whole world, even the cars we drive, runs flawed code written by people. Flawed code can be reprogrammed. We have a technology that writes code for people; it doesn't introduce flaws. We have proof this works and we'll show you. It's a mistake to hear a cacophony of opposition and believe that it cannot be overcome. The world is often loudly signaling its need to be convinced. It's a mistake to believe that the change that has not yet come will never come. On 9/11 the world changed; on 10/26 the PATRIOT Act was passed. The Patriot Act was not written in 46 days. It was simply the instrument that was ready when the moment arrived. Software safety will have its moment. You have these few days to prepare. With that, let's get started.