Handicapping the US Supreme Court

Video thumbnail (Frame 0) Video thumbnail (Frame 1085) Video thumbnail (Frame 2293) Video thumbnail (Frame 2930) Video thumbnail (Frame 7711) Video thumbnail (Frame 9533) Video thumbnail (Frame 12880) Video thumbnail (Frame 14051) Video thumbnail (Frame 15699) Video thumbnail (Frame 16477) Video thumbnail (Frame 17334) Video thumbnail (Frame 19920) Video thumbnail (Frame 22109) Video thumbnail (Frame 23527) Video thumbnail (Frame 26061) Video thumbnail (Frame 30479) Video thumbnail (Frame 31405)
Video in TIB AV-Portal: Handicapping the US Supreme Court

Formal Metadata

Handicapping the US Supreme Court
Can we get rich from forceful browsing?
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Foofus - Handicapping the US Supreme Court: Can We Get Rich by Forceful Browsing? https://www.defcon.org/images/defcon-19/dc-19-presentations/Foofus/DEFCON-19-Foofus-Forceful-Browsing-WP.pdf Using only script-kiddie skills, it may be possible to handicap the outcome of decisions of national importance. This talk presents a walk-though of a project to make more accurate predictions of US Supreme Court case outcomes. That could be a useful thing, if you had something at stake. Conventional techniques for predicting outcomes rely on legal expertise and knowledge of the policy issues at stake in a case and the justices' voting records. Forget all that: we're going to see what we can do with perl and XML transcripts of oral arguments. It's only 20 minutes of your life, but it might equip you to astound your lawyer friends, or make some canny investments. For nearly fifteen years, Foofus has worked in network security, spending most of that time leading the charming and intelligent foofus.net team of penetration testers. Prior research has dealt with software security and trust relationships between systems in large networked environments. In more recent times, Foofus has been enjoying law school, and in particular, finding ways to apply hacker techniques to legal studies.

Related Material

Software Hacker (term) Multiplication sign Physical law Gradient Bit Function (mathematics) Information security Window Metropolitan area network
Point (geometry) Purchasing Slide rule Group action Civil engineering Multiplication sign Physical law Mathematical analysis Bit Proof theory Word Term (mathematics) Personal digital assistant Order (biology) Video game Procedural programming Cycle (graph theory) Quicksort
Point (geometry) Suite (music) Presentation of a group Dependent and independent variables Digital electronics Key (cryptography) Multiplication sign Decision theory Physical law 1 (number) Parameter (computer programming) Declarative programming Word Personal digital assistant Phase transition Energy level output Procedural programming Hill differential equation Writing Resultant Window Relief Row (database)
Slide rule Suite (music) Capillary action View (database) Flash memory Thermal expansion Parameter (computer programming) Web browser Theory Bookmark (World Wide Web) Personal digital assistant Information Predictability Algorithm Information Decision theory View (database) Projective plane Planning Cursor (computers) Statute Statistics Process (computing) Personal digital assistant Complex system Right angle Quicksort
Parsing Concurrency (computer science) Information Computer file Decision theory Structural load Decision theory GUI widget Electronic mailing list Parameter (computer programming) Parameter (computer programming) Voting Word Voting Personal digital assistant Personal digital assistant Information Quicksort Proxy server Gradient descent
Parsing Dependent and independent variables Multiplication sign Mathematical analysis Parameter (computer programming) Bit Parameter (computer programming) Line (geometry) Counting Formal concept analysis Rule of inference 2 (number) Database normalization Inference Proof theory Database normalization Word Personal digital assistant Personal digital assistant Right angle Pattern language Quicksort Extension (kinesiology) Row (database)
Information Range (statistics) Mathematical analysis Sampling (statistics) Division (mathematics) Parameter (computer programming) Leak Voting Word Goodness of fit Voting Cross-correlation Hooking Average Personal digital assistant Order (biology) Inference Pattern language Right angle Information Quicksort Position operator Spectrum (functional analysis) Row (database)
Suspension (chemistry) Point (geometry) Suite (music) Group action Direction (geometry) Decision theory Multiplication sign Parameter (computer programming) Event horizon Number Sound effect Heegaard splitting Computer configuration Term (mathematics) Personal digital assistant Energy level Game theory Position operator Social class Rule of inference Information Decision theory Block (periodic table) Structural load Physical law Civil engineering Mathematical analysis Planning Bit Group action Substitute good Word Personal digital assistant Video game Pattern language Quicksort Videoconferencing Data structure Window
Point (geometry) Area Email Open source Software developer Software developer Open source Code Solid geometry Formal language Area Wave packet Neuroinformatik Substitute good Software Quicksort Information security Hacker (term) Information security
ladies and gentlemen a man who needs no introduction hi I'm foofa's and I'm going to talk today about some research that I did over the last few months and there's a bit of work that has to be done to set the stage so some of you know me this is my third actually time talking at Def Con previously I've talked about windows networking stuff and some software security stuff over the last few years I've been enjoying law school and so I thought well how can I use the fact that I have at least mediocre hacking skills to my advantage in law school so that I can loaf through classes and get better grades rather than have to learn all this stuff and so this is the output of some of that research so I'm going to spend a little
bit of time kind of a one slide introduction to federal civil procedure so just so we have some terms set I'm going to talk about some tools that I made for harvesting data and then look at a little bit of analysis and then talk about what good it might be so in other words what was the point of this exercise so I'll say up front that this is by far not a work of great genius it is I would consider it kind of a proof of concept and the question is can we without knowing anything about the law and without knowing anything about how stuff works can we start making some interesting deductions about what the Supreme Court is going to do so that's that was the question I set out to try to solve and so unfortunately in order to get to that solution we're going to have to do a little bit of law
background so so this is it civil procedure in 11 slide so the life cycle of a so by the way we're only going to talk about civil cases criminal cases are interesting but we're not going to talk about them because nobody invests its stocks the stock makes stock purchases based on criminal action actions well they're not supposed to so the life cycle of a federal civil case it starts off when someone gets mad and
wants to sue someone else so that's called pleadings some guy will file a complaint and the other person will have to file an answer to that complaint so in other words you wronged me hell no i didn't and then the battle is joined then we begin a phase called discovery which is designed to keep the key the case out of court and you don't have to worry about all the little words there that's mostly to keep me from losing losing my place the vertical words are the interesting ones so discovery you'll see you have depositions interrogatories request for admissions stuff like that basically this is the parties are going to exchange information so that each can size up what the others case looks like and if you are a defendant and you can see that the plaintiff has a really strong case you might want to settle and then that takes it off of the federal courts docket and you don't have to worry about it on the other hand if you think that the thing has no merit whatsoever you can maybe persuade the plaintiff of that so okay we've done discovery next we're going to do some motions trying to get evidence either included or excluded trying to get if we can manage it summary judgment which is basically okay yeah everything the plaintiff says is true but it doesn't really amount to a hill of beans so what get it out of here dismiss the suit then assuming all that fails we'll have to have a trial where we'll present evidence and hear arguments and then you'll get a verdict and that will include a judgment of you know Co get some damages or injunctive relief or declaration or something like that and then whoever didn't like the judgment can appeal it so when we appeal and you can appeal actually from any final judgment by the court so if your case is dismissed at the motion stage you can appeal that in fact a lot of the cases that the Supreme Court takes come to the Supreme Court from summary judgment alright so now we're at the appeal stage and we no longer say plaintiff and defendant we say petitioner and respondent petitioner is the person who says hey the court got it wrong respondent as the one who says no the court was totally right and and so you usually would in a federal court you've heard your trial or you know whatever procedure at the at the district court level you're going to appeal to the circuit court level then whoever doesn't like what the circuit court did they can appeal it and try to get it into the Supreme Court and this is called a petition for certiorari certiorari is the writ that the supreme court issues saying yeah we're taking the case and it takes four justices to agree so in other words the supreme court gets you know i don't know a gazillion petitions a year and they're going to select you know 80 or so to to hear they hear only a you know portion of what they do and they vote on the cases and if for justice has agree that this is one that's worth hearing they will hear it so now we're at the Supreme Court so the first thing that happens is the parties file their briefs in other words this your honor is based on just the law and all the precedent that we can see is why I win after the briefs are filed they'll be read by the justices law clerks and then by the justices presumably and then we'll have oral argument and then the justices will deliberate and then we'll get a result which is somebody wins or somebody loses or maybe it's totally ambiguous who won or lost or maybe they just decide not to decide those are all kind of conceivable outcomes and what I'm interested in is
the time between oral argument and the time when a decision gets rendered because that's a point at which after which nothing new is going to be added to the case all of the arguments have been made all of you know whatever whatever is going to be considered has been entered into the record and nothing is going to change between then and the time when a decision is rendered so okay we have this window of time when were the case is kind of in suspended animation and it's possible that it's even been decided in fact the Wisconsin Supreme Court right after they hear oral arguments they go and and sit in the room and they vote and the case is decided right there the world doesn't hear about that for a couple months probably because somebody has to write a well reasoned opinion and explain why this happens but but the case is decided right then and there so we have this time when the case might be decided maybe it's not maybe it is we don't really know what the Supreme Court does does they're pretty secretive about that but we know the one thing we know is there will be no more input whatever stuff they're going to consider it's in there and we just have this time when we're in limbo not knowing what happens in the case and that's what I wanted to see if we could do something about so
there are a couple of resources that are available to us and the best and my favorite is a website called OA org or is what the bailiff cries out when in summoning people to come and listen to the Supreme Court oral arguments and you can get a information about the Supreme Court docket since we've had a Supreme Court so from day one all the way up you can read about all of the different cases who were the justices who were the advocates a little summary of them but what's also fascinating is you can listen to oral argument there you can go and there's a little flash applet which I'm sure is totally secure and then you can listen to the oral argument it actually has a little kind of comic book style i'll show it to you on the next slide we can look at the pictures of the justices who ever speaking their picture will light up and you can see the text of what they're saying and so you can experience as though you were there at one first street so that's one resource the other thing is we already have several ways of predicting case outcomes one is the scholarly way right we're going to study the questions that the court is being asked to answer we're going to research the policy issues that underlie those questions we're going to understand the ideologies of the Justice the justices who will be voting on this and we're going to make a prediction based on that so this is like sort of mainstream theory of predicting what the court is going to do this is what you know Dahlia with wick or you know whoever is on NPR these days is doing next you have there's this really funny thing called the fantasy scotus project where it's sort of like fantasy baseball it's trying to harness the collective wisdom of you know legal pundits around you can enter and sort of bet on outcomes and it turns out that fantasy scotus is not a bad predictor they do a decent job and then there are some people who have tried to do sort of algorithmic looks at this and make make other you know complex systems for predicting what the outcome of a case will be and I say the hell with all that what can we do with absolutely no knowledge whatsoever and you know completely unprofessional tools so this
is what the org little let's listen to an oral argument thing looks like so you can see that justice Sotomayor is speaking right now and it's kind of has a little cursor that follows along and this is by the way the suit here is Ashcroft vs al kid the suit where it was announced by the court that the federal government can detain you under the material witness statute even if it's totally a sham so you know it's not a great case as far as I can spin anyway this is this is what's called the expanded view which means you can see the pictures of the people who are speaking and you can see the text and so I thought well that's pretty interesting you can see the text somehow that text is coming to my browser and have to research how that's happening so my plan was let's go ahead and grab all
of the docket information by docket information I mean what's the list of cases and where the justices participating in them and when were they heard so we're going to grab all of that and then we're going to grab the information about the individual cases how did each justice vote did they write a decision in the case in other words what you know the majority opinion of the court or a descent or a concurrence did they write something that you know expressed an opinion but then it turns out if you watch what that flash applet is doing through a proxy it's really just pulls down an XML transcript of the oral argument which is nicely annotated and well suited to parsing by Pearl and so you can grab the oral argument transcript as well that's kind of fun so I built two tools for doing those and these are on your def con DVD i believe i submitted them anyway i haven't actually looked at the DVD but they should be on there in case you want to try this at home get docket in case brief and and these will they're basically just information harvesting tools they'll pull this stuff down for
you so what we're going to do with that is parse it and sort it so what I did was for each case I recorded the disposition and who are the advocates and you know sort of the the facts that I could get out of there that were easy facts and then I also for each justice I made a file that contained you know the cases that they've heard the cases that they voted with the majority of the cases that they voted with the minority cases where they've written an opinion etc etc and so that was all good fun and
so I just said well screw it let's just get all of it you know let's get all of the complete record for every justice who's currently on this preme court it turns out that's about 250 megs of data which is not that much these days and I thought well let's just count the questions that they asked at oral argument so it oral argument what happens is you stand up and you say you know may it please the court and you start into your argument about 30 seconds in the justices start peppering you with questions and you never really get to say what you came to say but but you can see in the transcript who's asking what questions and I thought maybe we can spot some patterns that correlate between who's asking questions and how they're going to vote all right
well there's some problems the first is what's a question you know maybe a justice says something like excuse me you're standing on my foot or you know I don't know what this you know em that's something like that and I just decided for the purpose of this analysis anytime a justice speaks to the extent that they were recorded in the transcriptional a line they're saying who is speaking and there's a some words that's a question any any utterance really the second question is what is the outcome of a case oftentimes there will be several questions presented to the court they'll rule for the petitioner on some and for the respondent on others and so it's pretty ambiguous who won that's a that's a tricky thing to determine and so I decided for the purposes of this sort of proof of concept analysis I was just going to take whatever oya dorg said was the winner I declare that's the win or two and then they're probably right most of the time anyway the third thing is some some data normalization issues so we saw that screenshot with justice Sotomayor she sometimes appears in the XML as sonia sotomayor sonia underbar Sotomayor sometimes as justice underbar Sotomayor sometimes as justice underbar sonia underbar Sotomayor and there are other tricks like that in there so I had to do a little bit of just fudging with the data to get it so that I was really counting correctly and then there's something horribly wrong with the year 2008 up on a org I don't know what happened they must have had like sleepy interns or something like that but it's the data is horribly mangled for that and I actually couldn't get much useful out of that year so great we've collected a pile of data can we draw any inferences from it for some justices no I
there are three justices or four justices sorry for whom we really can't say anything meaningful first Justice Thomas has not spoken at oral argument for about six years over five years anyway and so this makes analysis of his questions fruitless so he does not leak any information at all however if you can guess how Justice Scalia will be voting you might have a good clue about justice thomas the one that you really wish would would leak some information as Justice Kennedy he's generally considered the swing vote on divisive issues and he exhibits no obvious patterns so that's very frustrating it would be great if we could have spotted something there because we could really make some headway but he's a near as I can tell a completely random questioner just popping in out of nowhere with no obvious agenda and you can't tell what he's going to do and then we have Kagan and Sotomayor who have not been on the court long enough to establish much of a record so you know they might they might be leaking information we don't have enough of an actuarial tale to actually say yet but there's some hope in other words they deviate radically from from the average of the court at least in this short sample range so maybe they will be doing something interesting down the road okay so we did that and we can say that for these for not much can be said but 45 the remaining five they
exhibit what i would call antagonistic questioning so in other words if you if you count up the questions that they asked whatever side they're going to vote for that's the side they're going to ask fewer questions of and this is generally true and it sort of makes sense right if someone stands up in front of you and is espousing a position that you disagree with you might tend to kind of get into it with them and if somebody's just saying stuff you totally agree with you might kind of let them off the hook and not not bother them so much so here they are in order of strength of correlation Justice Scalia is you know sort of pegging the meter over at you know if he doesn't agree with you he's just going to be on your case at oral argument and then all the way to the at the other end of the spectrum Justice Roberts is you know raising the needle somewhat but but isn't isn't a strong strongly predictive questioner so the next question is okay so who could use this you know who might this be good for well the first answer
is law school nerds who are trying to write papers for their Supreme Court seminar class but the end pundits maybe this might be you know I don't think this is any substitute for actually you know knowing stuff but but when you're totally on the fence about what's going on this might be an interesting thing to supplement with so that's one thing to to you know one community who could use it but also litigants who are in suspense after their oral arguments take place some of these people own companies who have been sued and and they might like to insulate themselves from the upcoming ruling and if they could know or have some hint about what it is they could take a position that might help them a little bit also can we think of any cities where you could make a bet on the outcome of an upcoming important event so you could just you know you could just literally wage your money on it see what the over/under is on on the upcoming case okay stuck it but and this is where I'm really headed is his actual investors so suppose you had prior knowledge that the walmart class-action suit would not be allowed to go forward you know so Walmart stock is maybe going to take a bump rather than a hit when that's announced or maybe the suit to to strike down the California law barring the direct marketing of violent video games to two minors if you knew what the outcome of that was going to be you might be able to make some fairly strategic investments because these involve large publicly traded companies and that's sort of the inspirational thing about them so this isn't a foolproof method by a long shot this is
really only the most superficial kind of analysis that we could be doing and everyone that I tell about this will hit me with like a bunch of other things hey have you thought about doing this or that thing and the answer is what I thought about it maybe a little I didn't have time this is very interesting but a couple of points even before you know thinking about you know doing more sophisticated analysis even at this level the court just tends to rule in favor of petitioners it's about a 55-45 split there and that kind of stands to reason to because why would you grant certiorari to someone who is petitioning you why would you agree to hear the case of someone who you didn't think had a point you know so if you're if the Supreme Court is going to take a case they probably think it is at least some merit to it the other thing that's interesting is there are very strong groupings of outcomes the court favors certain split patterns so in criminal cases you see a huge number of unanimous decisions and and you still even in civil cases we'll see 90 or 80 a lot of the time and then the other sort of Big E is is 54 you see a lot of those splits nowadays but the biggest challenge for us as people who would like to get rich off of this technique is the fact that we don't know in advance when a decision is going to be released so we know when oral arguments are held and we know that the decision will come sometime after that but we can't say with any kind of precision when that's going to be and that is frustrating because if you want to buy a stock option the the load that you pay on that option is tied directly to the duration of the option so in other words if you could predict it in a narrow time window you could do a lot better on your option you'd spend a lot less on the option itself so talking with people who are smarter than I am about finance the guidance that I got is that this is not the sort of thing that you even if you had really good information even if this were you know a pretty ironclad and method of predicting outcomes you probably wouldn't want to do something like start a hedge fund that does only this because the number of cases that you're and therefore the number of trades that you'd be making is pretty small it's maybe you know five or ten per term and that's not how those funds like to work the better plan would be to get a small cadre of investors you know four or five of them who have some money that they're not too risk-averse about and make some small investments based on that you can get a fair amount of leverage with options you know more than just buying the stock in fact that's why when you see an insider trader case the way the SEC caught those people is they purchased large blocks of options it's never that they bought the stock itself because they can get so much more mileage out of an option vehicle all right so kind of starting to wrap up here again this is no substitute for
actually knowing stuff I don't recommend you know disregarding traditional methods of insight about the Supreme Court but like I said if you were at a point where you're just trying to make a guess and you'd sort of exhausted your other sources of knowledge this might be an interesting supplement to it and it's kind of fun to be able to make you know somewhat sophisticated judgments with absolutely no legal training whatsoever and that's that's pretty much what I came to say so like I said this isn't a major computer security issue or anything like that is a footnote but it's kind of a fun one and the other thing that I would like to mention is I
don't know some people in the room will know who foofa's done that is we are hiring if you are a software developer with hacking interest we would love to talk to you so drop an email and we'd love to see about your resume I am going to be over at least briefly in the QA area after this but no one is actually yanking me off the stage now or are you okay so i will be over in the Q&A area shortly thank you very much and have a good rest of the cop let's make a brief announcement