DEFCON Networking Team: How we learned to stop worrying and deploy a network for ~12 hackers in 3 days.

Video thumbnail (Frame 0) Video thumbnail (Frame 9702) Video thumbnail (Frame 15519) Video thumbnail (Frame 29922) Video thumbnail (Frame 31747) Video thumbnail (Frame 34500) Video thumbnail (Frame 36460) Video thumbnail (Frame 37836) Video thumbnail (Frame 50613) Video thumbnail (Frame 53984) Video thumbnail (Frame 65489) Video thumbnail (Frame 76994)
Video in TIB AV-Portal: DEFCON Networking Team: How we learned to stop worrying and deploy a network for ~12 hackers in 3 days.

Formal Metadata

DEFCON Networking Team: How we learned to stop worrying and deploy a network for ~12 hackers in 3 days.
Alternative Title
Building The DEF CON Network: Making A Sandbox For 10,000 Hackers
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
We will cover on how the DEF CON network team builds a network from scratch, in three days with very little budget. How this network evolved, what worked for us, and what didn't work over the last ten years. This network started as an idea, and after acquiring some kick butt hardware, has allowed us to support several thousand users concurrently. In addition I will cover the new WPA2 enterprise deployment, what worked, and what didn't, and how the DEF CON team is has mad the Rio network rock! David M. N. Bryan has 10 years of computer security experience, including pentesting, consulting, engineering, and administration. As an active participant in the information security community, he volunteers at DEF CON, where he designs and implements the firewall and network for what is said to be the most hostile network environment in the world. This network allows speakers, press, vendors, and others to gain access to the Internet, without being hacked. In his spare time he runs the local DEF CON group, DC612, is the president of Twincities Makers group, and participates in the Minneapolis OWASP chapter. Luiz Eduardo is the Director of SpiderLabs Latin America Countries. With almost 20 years of experience, throughout his career he has worked with possibly all types of networking technologies on the enterprise and service provider sectors, as well as the security involved in these technologies. Luiz is the founder of the y0u Sh0t the Sheriff security conference held in Brazil and has worked on the wireless infrastructure of Blackhat, DEF CON, Computer Chaos Congress and Shmoocon. As a public speaker, he has given presentations on diverse infosec topics at worldwide on conferences such as DEF CON, FIRST, H2HC, HitB Malaysia, Layerone, ShmooCon, BlueHat, ThotCon, Toorcon and others. Luiz holds the following certifications: CWNE, CISSP, GISP, GCIH and CEH.
Point (geometry) Server (computing) Building Service (economics) Ripping Modal logic Multiplication sign Electronic program guide 1 (number) Mereology Disk read-and-write head Perspective (visual) Mathematics Operator (mathematics) Analogy Computer network Videoconferencing Software testing Metropolitan area network Fiber (mathematics) Mobile Web Addition Dependent and independent variables Host Identity Protocol Information Weight Feedback Graph (mathematics) Content (media) Electronic mailing list Computer network Bit Software maintenance Connected space Process (computing) Computer network Website Right angle Hacker (term) Table (information) Sinc function
Point (geometry) Trail Multiplication sign Drop (liquid) Open set Streaming media Disk read-and-write head Login Wave packet Twitter Front and back ends Computer network Series (mathematics) Information security Metropolitan area network Physical system Area Enterprise architecture Structural load Computer network Connected space Data management Virtual LAN Web 2.0 Process (computing) Event horizon Computer network Connectivity (graph theory) Speech synthesis Right angle Wireless LAN Spacetime
Group action Logistic distribution Multiplication sign Execution unit Range (statistics) Mereology Proper map Different (Kate Ryan album) Computer configuration Oval Computer network Core dump Videoconferencing Hill differential equation Fiber (mathematics) Area Email Coordinate system Infinity Staff (military) Bit Lattice (order) Connected space Band matrix Degree (graph theory) Digital photography Data management Process (computing) Compilation album Website Right angle Fiber (mathematics) Arithmetic progression Spacetime Asynchronous Transfer Mode Point (geometry) Purchasing Dataflow Implementation Game controller Patch (Unix) Mobile Video Digital object identifier Staff (military) Number Twitter Population density Band matrix Intrusion detection system Touch typing Computer hardware Energy level Software testing Booting Wireless LAN Standard deviation Multilateration Cartesian coordinate system Population density Personal digital assistant Data acquisition Computer network Videoconferencing Wireless LAN
Area Slide rule Distribution (mathematics) Mapping Forcing (mathematics) Multiplication sign Menu (computing) Digital object identifier Disk read-and-write head Regulärer Ausdruck <Textverarbeitung> Perspective (visual) Frame problem Different (Kate Ryan album) Computer network Right angle Data structure Local ring Spacetime
Point (geometry) Uniform resource locator Software Bit rate Logic Bit Right angle Drop (liquid) Identity management Transmitter Number
Decision theory Open set Disk read-and-write head Information privacy Perspective (visual) Neuroinformatik Front and back ends Videoconferencing Cuboid Data conversion Fiber (mathematics) Physical system Bit Thermal expansion 3 (number) Connected space Band matrix Data mining Virtual LAN Internetworking Computer cluster Website Right angle Moving average Video game console Sinc function Electric current Spacetime Point (geometry) Laptop Game controller Server (computing) Firewall (computing) Virtual machine Device driver Drop (liquid) Average Goodness of fit Internetworking Energy level Codierung <Programmierung> Router (computing) Address space Noise (electronics) Just-in-Time-Compiler Graph (mathematics) Polygon mesh Copyright infringement Debugger Planning Database Line (geometry) System call Peer-to-peer CAN bus Radius Normed vector space Computer network Service-oriented architecture Communications protocol
Dynamical system Transportation theory (mathematics) Multiplication sign Range (statistics) Workstation <Musikinstrument> Design by contract Client (computing) Mereology Core dump Computer network Diagram Damping Resource allocation Library (computing) Exception handling Physical system Area Enterprise architecture Theory of relativity Mapping File format Shared memory Data storage device Maxima and minima Instance (computer science) Dynamic Host Configuration Protocol Band matrix Type theory Dynamic Host Configuration Protocol Virtual LAN Data management In-System-Programmierung Process (computing) output Configuration space Right angle Quicksort Spacetime Point (geometry) Game controller Server (computing) Service (economics) Firewall (computing) Online help Event horizon Power (physics) Number Latent heat Internetworking Energy level Software testing Address space Authentication Time zone Key (cryptography) Interface (computing) Planning Line (geometry) Cartesian coordinate system Wind tunnel Film editing Radius Computer network American Physical Society Wireless LAN Communications protocol Local ring
Presentation of a group Existential quantification Multiplication sign Port scanner Open set Mereology Disk read-and-write head Perspective (visual) Information technology consulting Neuroinformatik Sign (mathematics) Different (Kate Ryan album) Computer network Cuboid Data conversion Physical system Block (periodic table) Web page Feedback Electronic mailing list Bit Instance (computer science) Lattice (order) Connected space Type theory Virtual LAN Data management Repository (publishing) Telecommunication Website Configuration space Right angle Quicksort Metric system Spacetime Point (geometry) Trail Game controller Service (economics) Firewall (computing) Online help Rule of inference Power (physics) Goodness of fit Term (mathematics) Computer hardware Energy level Interface (computing) Graph (mathematics) Planning Line (geometry) Graphical user interface Loop (music) Integrated development environment Computer network Speech synthesis Game theory
basically we're going to talk about is who we are a little bit about what we do
ourselves some of our goals for the network some of our challenges high-level discussion of what we've got going on in the network here and just have a basically a QA we want to get feedback from the audience to really understand what you would like us to do where you want us to go things you'd like us to see for likely to have us do
does anybody heard about this it's a new meme we're starting it's called pen test John enables SSH can't get in all right
so let's start off with Heather who's going to be introducing walkie hi I'm Heather I'm also representing Lockheed today some of you if you've gone to the DEFCON networking site and seeing some of the updates and the graphs he's the mastermind behind creating that content and keeping that up to date Lockheed has been with Def Con since I believe dc-3 and he's been running the network he is the primary lead for the network team and I am the second he also built the mobile agenda not sure if any of you use that this year but it was on both it was up both this year and last year pretty pretty handy you know quick reference guide and I've been with Def Con sense dc9 so this is about my 10th year and I do a lot of the tactical operations on the ground all right so next would be Mac hip a Mac I handle a large part of the wired infrastructure and servers and services for the network make sure that the baseline connectivity is there uplinks you know wired Jack's for the speakers goons vendors contests etc etc all right I mean I'm video man I've been doing this since Def Con 6 basically it's been really fun and challenging seeing things grow over the years you know we went from an infrastructure at the Alexis Park where literally we get up on roofs and run cable you know in hundred-degree weather or run cables across the parking lot and you know seeing this network mature and see it grow into what it is today has been really fun really great and hopefully it provides value to the conference attendees for sure it's definitely provided value from the perspective that you know updates Def Con org how many people use that site few of you okay I mean that's the info booth they set that up they come in and do that and we provide the network for that if that network wasn't there that wouldn't be there all right next Louise I'm Louise I've been doing the wireless network here for I think this is my sixth year so have done one year at the Alexis Park another year or all the years at the Riviera and now we're here I'm Derek I'm the guy who runs DC TV I've been with Def Con since Dethklok ha out of curiosity how many of your drunk hungover asses were sitting in bed watching DC TV this morning I just want to sit at the edge of awesome sorry for the outage that sells my fault nice he's changed a lot over the years especially ones where we didn't have it and it's been a long time since we had all the pure analog ballons and old analog cameras switching over to what we have now we're now running mpeg-4 over IP over a 4,000 foot fiber trunk all the way the cable TV head end and the other total end of the building so that was a bit of a challenge to set we had a lot of last-minute changes to the infrastructure but it looks like it worked out pretty good special thanks to the sauna knowledge folks they helped pull it together yes roast Sparky my name is Michael or Sparky gets them essentially the DC tech for the knock everything you see taped to the floor my blood sweat and tears went into that fine fine work yeah resident bartender yeah that's about it i've been with Def Con since Def Con aid without Sparky there would be no network rip see who is down at the end yes my name is Richie um my responsibilities basically are very similar to sparkys we run as a team we were around we make sure everything was working the last minute do repairs if necessary tack down tabled puddin switches anything against name we make sure it happens and Eric who is a new addition to our team hello and I married and what I've been doing all along this week is to try to ensure you have the wildest coverage and and reachable for for almost all the time so he was ten pounds to the wall for the whole pretty much pretty much that's a lot of weight for a little man yeah I mean so basically Louise and Eric are in charge of the wireless infrastructure and you know we arrived on site on Sunday and started setting up the infrastructure and if you arrived on saturday as well but Louise and Eric actually go through and and started reassigning all the access points we we did a little bit of an upgrade we'll talk about a little bit later and they're there that team the last person on the list is tease who is actually wasn't able to join us he he just got a new job which is good but unfortunately couldn't take the time off but what he does is a lot of the the content updates so yeah we've got the mobile edition Emmy def con org that he helped put together he's helped with a lot of our maintenance of our wiki that we're using what else what other duties is it has you put together I think that's general cable bitch yeah general bitch alright
so what is the network right some of our goals is reliable connectivity you know if you're going to come to a network or come to a conference and have a network connection it needs to be somewhat reliable secure ish because we don't know if someone has a zero-day for the wireless network right we say secure ish because now we have WPA with enterprise management on it basically allows you to log into that how many people here actually created a login before they came all right how did you find out about it Twitter was that primary all right then that's great because that's something that we've kind of used as a team if we know something we try and tweet it out like hey go do this and then usually someone retweets up for us which is good okay say we also pay attention to the Twitter stream a lot you know so if you know if some people are complaining about particular problems in an area or something that's one of the ways we've been trying you know that's our poor man's ticketing system so the other thing does anybody remember at the Alexis Park I think it was the first or second year basically Ghent who is no longer with the team do too well anyway he brought along a switch from what it is his his work like that was it a 4 4000 or 6000 series switch and we went oh we've got you know one hundred and some-odd ports let's plug them all in so we plugged all the ports in made all the ports in the conference center active and what you have happen is people would come along and plug in and go oh I can get access here great and then people be like oh do you have a hub or a switch oh sure they plug that in and all of a sudden you have this train of people going down the hall it was kind of funny I mean cool but fun except the fire marsh yeah yeah so the farmer shal i don't know if anybody remembers that but we had problems with the fire marshal at the Alexis Park and so we said okay we can't do this people needed to be able to be mobile and be able to get access so at that point we started really deploying a wireless network you know the wireless network at that time was in no way secure at all I was all just open wireless access points well and so basically all right let's work towards something that that people get any access to anywhere the other thing we have a boatload of segmentation we have speakers press all the speaking tracks so like this podium has its own VLAN has its own network each of the five tracks that we have out there have their own VLAN we decided to change that up last year it was one VLAN for all the speaking tracks and then someone decided to do some man in the middle on the speaker network and we're like yeah maybe we should change that so we did you know it's it's constantly a growing process of learning process and you know that that's really what this is about but right now I think we're at 199 VLANs that we've configured on the back end so I mean that's pretty cool what is it 140 year for wireless 140 or for wireless and 59 are for wired drops and we had 69 actual drops into the spaces so only 10 of those drops shared VLAN segmentation with anybody else and those are primarily the contest and unless we get a specific requirement that it needs to be on its own network we kind of say all right maybe some people can share it reduces the load that we have it go ahead I don't know what our back-end switches can hold off the top my head it's like 300 VLANs I think is what we max out at but
alright so some of our challenges I mean we don't have an infinite budget right we actually I mean when we started out doing this we said all right what can we get off of ebay what do we have we can borrow and that worked out well for quite a while unfortunately we can't go borrow reboot controllers because sometimes they're in use well most the time in their use so what was that def con 1313 that we actually went out and purchased the wireless controller the Aruba wireless controller primarily because it would have some IDF and built into it you know intrusion detection be able to detect rogue aps I think that was the year that there was some contest that was doing something in the in the contest area like a king of the hill access point or something and we would see it and we'd das it we'd say alright no it's not on the network gone you know five minutes later access point would disappear the next day it would show up again we're like come on guys and finally add Sunday afternoon I think they came back into the NOC like yeah we know we tried to do this contest but unfortunately access point kept dying we're like oh yeah that was us so coordination also helps with us hotel does anybody want to speak about the hotel sure um are you looking for space usage or you know what that the logistics here yeah but you know the obviously you've been walking the half a kilometer back and forth through the convention space you know what the walk is like actually i think the space this year is incredible our wireless design has improved greatly just just due to the way that we've been able to use the space the hotel staff are fantastic we've had just a great partnership with them this year and while we had a good partnership with the riviera things are just really smooth here you know frankly it's it's a nice way to transition into a new hotel we have and I don't know if you want me to get into this right now but we have better bandwidth options here they have metro area thur net here so we have some flexibility to you know move up in bandwidth the infrastructure itself while different from the Riviera is has worked out well for our needs and really it seems like even the traffic flow has been better for for the humans so so the other thing is when we're at the Alexis Park it was a non-union hotel which meant we would grab a cherry picker and literally at like three o'clock in the morning be driving around the hotel space is driving this cherry picker going zip tiny access points running the cable down we all did that like everybody pitched in you know we'd get the network setup just before the attendees would come in because we really didn't have the space beforehand but when dealing with Union hotels we now can't touch certain things you know the lighting grid you know electrical things hanging things on ceilings hanging things on walls sometimes has to be done by Union staff so that that in itself was also a challenge for us because we didn't think like that we had to change our way that we were dealing with the hotel the infrastructure here Mac do you want to talk about the Farber yeah so moving moving from the rib to the Rios really it is a different stage of hardware infrastructure a different level of idf's different connectivity between them here we have a lot more option around well actually we have the same option but it's a standard option there's fiber between everything versus at the Rio we actually had some fiber patches we had some copper patches we had some long range Ethernet patches which were very interesting to some spaces sorry at the riff so it's here it's very different I mean we you know the hotel it's just significantly different from this time join in general one of the things that we have to consider with our infrastructure is it has to be spun up and torn down and matter of days and it's only used once a year so we tend to you know going back to the money item we tend to you know get what we can but we can't really justify something because we're not seeing a 360-day use out of it so a lot of our infrastructure tends to be fairly you know fairly older equipment we're still getting stuff off ebay there's certain you know purchases which we make of newer gear just for targeted applications the aruba the core for to a certain degree and a lot of stuff like that but you know we have sitting in our sitting as part of our infrastructure I keep looking at this I don't want to say here's what our infrastructure is versus here's the challenges we're going to go over the actual infrastructure in a bit so i'll leave that so bandwidth we actually have 100 megs bandwidth this here you know it's a metro area ethernet now not a white in my point to point so as i said that's the uplink we have is 100 Meg between all of the IDF's to the aruba gears that are we're sitting on gig backbones you know we could argue hey we should have more but realistically looking at some of the graphic trends I was looking at some of the traffic trends we haven't really been pushing more than a couple hundred Meg so it's I mean I think it's high for a con for a convention usage but it's not high for an overall usage especially compared to the gear we have I remember a couple years ago there's a dave Bullock whose photos are in a lot of these here but they Bullock did a article on the DEF CON Network and in the comments below people are like oh why don't you have gigabit because that's CCC we have 10 gig wait a minute we don't need 10 gig we can't justify it we have to be able to justify what we're going to pay for and you know that in Europe they may actually have companies that will donate bandwidth to them we don't have that luxury we're in a hotel we have to pay the proper channels and get the proper things encore is actually the one who provided that to us so having invention when we say justify it we're talking about you guys you're really what justify it so if you know you find something you want to use more bandwidth for outside of torrenting because a lot of people just see that is not easily justified but if you have some really application something you want to play with and test with here now this is a good network to do it on so think about that especially for next year and the other thing that that I think we kind of have to dispel is the the rumor that oh don't go on the DEF CON Network because you'll get hacked right because if you don't use the network we don't justify that we can add more bandwidth that we can add more things so it's kind of a chicken and egg all rights my job so keep pulling down stuff would you let's see device so wireless you want to talk about wireless yeah sure so for the wireless for any wireless implementation there are two major concerns one is of course coverage right you want access wherever you are in the in this case in the common areas but also here we have or any convention you have a problem that is user density so I can put one access point here and it's going to cover this whole huge space but once we have a lot of people it goes away really fast so technology evolved a lot on the wireless market and wireless devices the things that you're still limited to certain unit number of users that are going to share that one access point so then it comes to the to the user density is going a way to the only way to solve that is that you're going to need more access points and once we have more access points they're going to start seeing each other and they're going to start interfering with each other and that's not good either so wireless in the US usually we use channels 16 and 11 that's what we should use and so if you have more than three ApS in the same place they're gonna they're going to interfere with each other so it's good to have a solution that deals with that automatically the other problem that we had ended I was going to say in the past but we still have some funny things happening this device compatibility this usually happen a lot for the past few years it didn't happen but last year and I think we're going to talk a little bit about that we had an issue with iPads and you came back this year once he came to you put that in sleep mode wouldn't come back so you had to there was a workaround for that but again it's always like it takes like a couple people to say I can't get on the network we hear that really fast and we try to fix it all right the other problem we
have is time I mean we do not have an infinite amount of time we have basically three to four days to do set up we're here for a week on site we have lots and lots of gear that we bring with either gets thrown on it in a car or gets shipped from someplace all of us come from across the country and maybe even outside of the country we have two Canadians here up on stage who can represent right we're a team of ten people essentially that you know works throughout the year we'll have like a Precog meeting where we'll come out to the hotel and make sure everything is the same make sure things are working and then just do things over email every once in a while or have a conference call the team is kind of broken up into the infrastructure which is Mac and I the Wi-Fi which is Louise Eric and Heather and then the video which is Derek and we've got the two managers so Heather and lock as well so forget about
our ground pounders they're pretty critical well what don't smother ground support they're pretty critical yeah yeah you got to have the bean counters all right so that this is the map essentially right this is all the areas that we have to cover and we weren't really expecting I think that we're going to be able to do the reg desk or the schwag area just from the perspective that we hadn't gotten requests for it and Penn & Teller theater I think was that also it was the last minute yeah so we kind of had to scrounge this year because we were expecting ex and we got X plus but no we do with what we can and this is a lot of ground to cover if you look at this map it's it's a fairly large space it's covering this is like five or six hi DF right Oh 3 4 5 6 6 idf's one catwalk space and one sorry one under theatre space and one cable TV head in space so we had to put equipment into switches and all those nine different cabinet yeah at IDF is entered interdomain feed right here they're what we said is intermediate distribution frame there's his master local it's really defense force basically it's a wire it's a telco cabinet ever sorry telco closet or where you know where you drop all of your network saw all of your wall drops come back to what is considered an IDF and then that will link up to a central IDF which usually called the master and this year we didn't have to work through any bathrooms to get to ours yeah I still
wish we had better pictures from that one yeah there is a IDF in the bathroom in the rib it's awesome yeah like oh this is where our equipment that's damn right anyway so this is kind of basically the structure of the network so I'm not going to spend too much time on this because it's a really busy slide you're not gonna be able to see anything but at least gives you a concept of this is this was pre putting the stuff in place post is a little different because we had to do some workarounds but it's almost where we're at right now this is a map of where all
the access points are essentially you want to talk about this one is a little bit Louise what is important here to see is that we cannot so we have tools that say this is the optimal location for the access points according to RF logic and whatever the smart people put into software's that calculate that but of course we depend on many things including where the drops are how how high we can put the access points and all that stuff so this might not look optimal but that's how we have it and then we deal with in configuration with
basic rates and transmit rates and all the good stuff to make number one coverage work all over the place as you can see on that one and also avoiding interference and roaming because Rome is quite important right when you start which you turn on your device here in this room and you're going to be walking around you don't want to drop whatever you're doing on you on your device so that's important
here's the graph that we just pull that about two o'clock today as you can see we did peek at some point on our hundred med connection there's 22 pretty pretty big spikes there and apparently last night y'all went out was hearty because you can see that there's a big traffic drop alright so we're going to talk a
little bit about what worked for us in this space we'll talk also a little bit about what we have to improve on but so secure Wi-Fi I think does everybody like the secure Wi-Fi yes alright so I mean you can thank Louise Eric lock those guys for setting all that stuff up lock actually did the radius database back-end and that the I think t?a's also did the front end for that so you could go in and enter in credentials off site we actually have that server living here and then when we pop up our internet connectivity that's what you're talking to is directly into our network on that machine I think it's very important that that we have that because it allows for some sense of privacy I mean from the perspective that we don't actually ship those vlans off to the wall of sheep we made that consolation conscious decision last year we basically said all right if we're going to give people a secure network there's got to be some someone's that there's an understanding that it's secure which means that yeah maybe you have pop3 still enabled maybe you have to tell MIT to a router maybe you have to do something like that but we don't want the wall of sheep to get to grab it it's it's just the wrong thing to do we don't want to give it explicitly to the wall of sheep I think it's the best way to put it because I mean the traffic once it leaves our uplink is going across the Internet anyone can man in the middle at that point but we're not going to explicitly go out of our way and do it you know at the beginning we want to make sure you have some semblance it's still up to you to secure probably end to end just from the standpoint of you don't know who out a huge else is out on the internet so you do need to protect yourself but we're going to do what we can in that space and the other thing that we've done is we've disallowed peer-to-peer traffic so if you get on the Wi-Fi you're only going to see the mac address of the firewall and the access points that you're talking to and I could be able to see anybody else's traffic and I can be able to map other people on the network because you only have internet outbound does that make sense it's a good thing I mean it it's how we're trying to protect people from from getting pwned I mean that's why you don't use the open Wi-Fi so we just you can actually networking Wi-Fi you're more than welcome to check all your email and do all your banking on the open Wi-Fi wall of sheep gets that traffic so does everybody else around you exactly yes fire shoot right anyway internet connection hundred mega covered that anybody realize that we had ipv6 yeah all right has anybody gone to an ipv6 site no yeah oh no it's good you know from from what we saw you know points two percent of you did something with ipv6 mat which actually mirrors the rest of the general internet traffic so that's not too surprising but once the pirate base which is over everybody will look traffic will go up and it's true we should talk to them about that one so what was that we were tunneling at outbound right yeah we're using a tunnel broker or sorry tunnel with tunnel broker / hurricane electric oh good company if you want to do your own IV ipv6 stuff go with them let's see the Wi-Fi was updated we had a software update on the controller that allowed us to do some good stuff we were able to extend the network out with just a couple of access points using some mesh protocols where was it to starbucks basically yeah we tried to go to all the way to the casino but it really didn't work mother there were some issues that the casino didn't really care for that too much even know wifis when you gamble I you know I don't know I'd be nice to sit and casino and drink a beer and well we lost we'll see what our expansion line is for next year so big thing DCTV to the hotel rooms if you want to talk about that Derek sure all right so um yeah it's been a few years since we've had DC TV in the hotel rooms just because in the Riviera we didn't have access to their cable TV head end so really missed that and obviously a lot of people that sore feet over it so we were really happy to be able to get back in there again this year but we're still at a challenge and that was we had to pipe video like you know over a mile of cable effectively so the systems we were using before which ranelagh ballons weren't going to work because you just get a big blob and noise at the end and no audio so this year we running I pb4 sorry mpeg-4 over IP over a fiber trunk to the head end now the equipment to do this stuff is crazy expensive at the professional level so we're actually using some again ebay finds really inexpensive mpeg-4 and asean cutters that are designed for video surveillance and we rented some challenges actually finding decoding equipment they would play it back on the other end and those challenges came up right near the end so again thanks to sok for supplying us with some scan converters allowed just a retrofit what we brought to actually get the other end at that cable TV headend working but with five channels it would have been I think it's three thousand dollars for the two ends of the encoders for a professional level equipment so you know fifteen thousand dollar budget for DC TV that wouldn't have gone over well so instead we did it for basically a shoestring budget and I think it worked out pretty good yeah the original plan was to use PlayStation 3's actually as decoders and we found out unfortunately once they have a network in between them they started crapping out after about three to five minutes of streaming so we were scrambling trying to find boxes that we could run this stuff on and and basically Derek re-engineered it at the last minute and got it working you know our plan was to have a little bit more content on the TV channels but unfortunately we had to forgo that just to get the feeds there for now so we were literally plugging them in during the first talk wandering around here so as as people were arriving in we were plugging it in so that's how close we cut it with DC TV this year heard the finding the servers to plug in as well since weeding out the Playstations we had to scramble to get computers so now my work laptop is up in the CATV head end I can't get it back without a guy who's licensed by the Gaming Commission I think to actually unplug it from their network and same thing I'm sorry Mac but his his torrent server which he was going to have using this wonderful bandwidth is now serving videos too so he doesn't get any new television to watch when he goes back home so you can thank these guys for giving up their laptop so you can have TV alright so we also had workshops I don't know if anybody heard about that yeah he's mine this fight is just hungover it's DEFCON what do you expect he actually works trust me which is why he's now sleeping yeah I a snow I can hear him snoring all right I'm dead good he's fine alright so we're chance we tend to party just as hard as you do so haha and go to work every morning yeah we have to get up at eight I'm for our call is at eight so sometimes that's pretty heat I'm a slave driver yeah yeah if both slave drivers I don't know how they do it oh that's right they go to bed at ten when I leave here I go on vacation that's not true just one night so workshops we had workshops so we had to deploy some switches there that was new to us kind of a new new thing which I think would be good I think we'll see what we can do maybe we'll have more stuff going on there alright issues p 0
e we bought a bunch of POS switches off ebay unfortunately they are Cisco proprietary POS which is we got some power injected switches not some POA ayrshire power in line switches not p 0 e yeah yeah d phones but no access points that won't help yeah so we could run cisco IP phones on them all day long but not not wireless access points that are to the 802 dot 3 a us-backed you said iOS devices I heard cisco iOS I don't know what you were talking about no one told me Apple apparently overnight the dhcp stopped working we're not sure about that we're going to look into that so that's an issue to figure out if someone was tossing it or not we don't know to be fair people are parties no one should be on overnight last night it's fine like I said everybody's out drinkin ps3s we talked about new infrastructure and new hotel just having to figure out exactly what hooks up a lot of the time sight unseen etc etc and there are challenges that come with casino hotel fortunately the good hotels have their infrastructure designs that the convention services area is actually a purely separate network from the hotel and casino side and this this hotel is no exception they have their network very well separated there's also well anyway Wi-Fi penetration so just being able to penetrate into the spaces I think there's some potential dead zones but no as we figure that out we're going to get better at okay we need more access points or maybe we need to put access points in key areas so that people get a consistent signal so yeah we plan for the signal to bleed over to the walk away there to the hallway and it didn't do as well as we thought so that's something to get better next year I think it's the big giant metal shield on the fear dinky something about a car up caged bird a kid all right so now is the
a and Q right we give answers you ask questions as you can see there's a microphone up front here people want to come up and ask a question I don't know if it's up or don't all rush up at once or just yell it out will repeat the questions if they're not on Mike don't worry about it yeah I'm curious what the cost is for the internet access for the duration of the event and who actually are you paying the hotel an ISP what are some numbers there Heather actually I don't have the contracted price in front of me and happened months ago but it is done through the hotel the hotel already has that bandwidth set aside and they build baseball park I'm not asking to the pay was I honestly I don't have it right now it was mixed in with a bunch of other contract stuff months ago so I do apologize but that is actually bandwidth that the hotel has for all conventions we can go up to 200 Meg on that bandwidth beyond that they would need to bring an additional but super quick question on the diagrams yet up there you had a really cool application to map out the wireless ranges what is that called I think that probably is it is embedded on the Aruba system so it's called RS plan and you just throw their like the size and Alden is it open for other AP types or just no no no it's specific for the AP so that the controller itself has a management interface and that's where it's actually pulling that from you go in and you go all right place all the APS where we told them to put them right and put them on that map nice it allows the skin port of a map essentially into that so it takes all the fun out of the job thank you quick favor can you guys move the line over right up here please no you we're not the TSA that we're cool that's cool the video camera yeah we're doing this the wrong way we're supposed to use the Q&A room but stuff starts at six-thirty so it's kind of we hush go I not really a question just a comment for consideration for next year yes I just mentioned that you have a lot of equipment that you transport out here possibly a solution to save on transportation costs if it's just equipment that you only use once a year just get a local climate-controlled facility yeah we have talked about that and for some things that will work but we do have the challenge of having to pre configure and test and update and do that sort of thing this year we actually there's three of us that came out saturday night and we had already pre-configured a lot of the equipment and updated a lot of the equipment to for instance work with ipv6 and give us the additional features that we needed so we do a lot of that kind of remotely so it'd be hard to do that in in a storage situation but for some of our equipment we are considering yeah I mean cables and stuff we don't need to necessarily how out but we need to update firmware isn't what not just want to see if you guys had any recommendations for a newbie setting up an 80 to 1 x network on the cheap i guess for 80 2 and X what on the server side or the whole thing well we actually use today any enterprise or not really enterprise type of solution supports 80 2 and X wpa wpa2 with 80 to annex authentication you can get we use freeradius you can use free radius and sometimes it's could be a painter but configured but it's well documented actually these days for the most for the commonly used applications all right so i have 2 1 i'll just curious for the wireless stuff it like in the program and like you guys mentioned that goes straight from the like wireless to the firewall is that just by way of vlans because you said they have like access point isolation so like to clients can't talk to each other or do with some fancy stuff like l2tp tunnels or anything like that there are 2 so 4 H ap we have to ESS ids ET SSID on each ap has one VLAN so if you hop in ap one you jump on the secure Network once you run to another ap you're going to be on another VLAN and then internally all the traffic is actually encapsulated through GRE from the AP to the controller the controller makes decisions as for what VLAN that user belongs and what policies we do and we actually bridge everything up to the firewall so all the routing is done on the firewall very cool the second one is since you guys mentioned it I actually fired up TCP dump on my phone which usually does ipv6 I didn't actually receive in the RAS so are you on the secure or on the data here yeah probably on the secure not going to see anything because that's the whole idea because on top of it on top of the VLAN ISIL I was going to say for the art the ra the ra you should have been seeing that comes from there yeah I got our wall side yeah so that I mean I was seeing it plenty on our stuff so we were having better I was saying better are a packing of traffic coming float through then from the dhcp just because the way v6 does its address allocation so I'm curious on if we could see it afterwards alright cool again just one thing regarding traffic management reppin if you guys played a way around it all with dynamic percentage-based / max traffic shaping and all or is it not fifo it's going to say not / mac we do have outbound traffic shaping on the firewall we tend to give priority to https sh sort of all days it's protocol based yeah at the firewall level not anything at a station level or you know format okay are you asking us if we want to prioritize our traffic I'm just oh it was it was just because I've been investigating some of that or you know when bottleneck alleviation okay yeah a couple couple things one it seems to me the equipment you're using is not exactly specialized for Def Con have you considered maybe working with UNLV or Community College in the area to have them maybe split costs for equipment with them they use it during the year Def Con isn't during the traditional school year maybe you guys can take the equipment then something like that so part of that comes down to whatever negotiations and contracts we can do with those individuals and I don't know if we have any particular relationship with you and lvu particular but it's an idea one of the things we have been avoiding in the last few years is getting any kind of specific vendor promised or any specific relation promised that is not ours because given the timing given the pre config and given you know this resource allocation like if we if all of a sudden they decided they didn't want to let us use it one year and we're borrowing equipment we're really in the hole for that or after we try to make sure those are good there's a political reason they didn't want to give us the equipment I mean it's a it's a really great idea but we are hacking conference yeah well I mean if you owned a share of the equipment that might kind of cut the legs out from under him as far as holding back goes and the second
question I had is that guy still breathing yeah here's okay he fought the good fight he's at a very long week but he's been up in the morning and helping us out every day so hi do you guys offer any type of service consultation or anything to the host hotel to help mitigate some of the computer malfunctions that might be happening during the conference no comment not even we we don't unless they come to us and say hey we're having a problem do the interview that we did what we did with the riv they did ask us to for whatever reason block outbound connections from us to the rib websites for instance so it's a delicate line when working with the hotel that we you know though there's some back and forth but when they ask for something we you know we see what we can do but typically it it's the we're segment at off you know if you think something's happening at in the network because of us then you know we can talk about that specifically but when we have segmented off it can only come out through our head end so we have an idea of what's going on there this is very different from what we experienced at the Alexis Park where we pretty much owned their entire network and would come in and they I mean it was great from the perspective at the time we were very small or well smaller and we were able to go in there and just sort of take over the network and do it when you need to do and you know now we're dealing with much larger Network much larger you know venue and and then the whole issue between the casino the casino gaming issues and the and the convention center so it's a totally different ball game now but we do we have meetings throughout Def Con we're constantly in contact with hotel management we have very open communication about what's going on and what they're experiencing and while we don't Oh note offer to help secure their network per se we do make ourselves available for questions and if they do need something you know they know we're willing to help okay thanks and thanks for all you do thank you a proclamation I'm just had a question you guys should some crafts and metrics do you guys keep talking about and make a publicly available yeah you can actually see it on DEFCON networking org okay yep great thank you yeah we usually will put together a presentation and then put all those graphs up everywhere the graphs are actually up already oh yeah someone most of us eventually made it back to our hotel rooms and when you flip through the TV we basically saw the desktop of your laptops or change whatever center I don't know if you record those but it would it be possible to record the talks and then like cue them all up and like for the talks for the day just play them in a loop at night you mean like an MVR yeah right actually so the situation is such that we can't do that partially because the the talks are recorded and sold and also because with this hotel we have agreed not to replay talks do too since we're putting on a cable system that we can't lock down that means anybody who's in the hotel can see those talks so by not replying them we're providing them with a sort of plausible deniability yep they said something to talk that the customer didn't like but it's not being replayed so yes they asked us if the speaker's wouldn't swear we said we cannot oh that's what this is our no promisingly what made me think about it I was had to do some work and I was sitting there flipping through the channels waiting for stuff to happen and yeah much part of jeopardy and actually I'm thinking like if other people in the hotel who have nothing to do with Def Con are sitting there watching jeopardy we might as well show him something more informational yeah yeah it was fun but you know now next year we're looking at you know putting something in between the talks as well so that there's not that you know the VLC desktop sitting on the screen in between the talks but again this year while we did get DC TV up and it was you know was a a win it was still challenged so we were working with what we could look out for next year because it'll be better ok hi guys I was wondering if you guys were planning to put up maybe like documentation of how you guys did the things you did or maybe some like best practices for people that want to set up know in other environments that are hostile as well I don't know of best practices is the right term to use on this now but yeah so we've we've been debating about it well not really debating about it we've been lazy about it for a few years this year I did like four particularly the firewall configs the switch config those are actually stored in a git repository once i scrub it I'm going to may make it available on github scrub it and get some sign offs on it I want to make sure everyone's cool on it but yeah we're going to be putting that together and putting it available so check out def con networking org and we should have some stuff up there hopefully in a couple weeks thanks for all you do thank you I said one thing that would have been any our office doable or not but with the TV be kind of cool if we could we had like the power points in the corner as they're speaking yeah and I mean I know obviously probably so challenge and we're getting it mixed yeah well actually what we we actually wanted to do that we wanted to combine some of that stuff it's an equipment challenge that we weren't ready to face this year we may have a time and planning to actually do that next year so yes we agree we actually wanted to switch it over to just the powerpoints but again we'd have to have scan converters and stuff that we didn't have hardware access to if we hadn't had the challenges we did have I think there was a pretty good chance we would have gotten to doing that this year but we just ran at a time and we would have been changing things happen to talks if we do to ku has two quick questions I hear you use PFR walls that correct see I think you're specifically in charge of it is it fair to say that you think the same of PF gooeys like pfsense in the same sense that a lot of iptables folks think of iptables gooeys is it kind of looked down apart over just PF itself from the airline is it one big for a lot of ways it depends on the tool you have or depends on your level and the tool you have we're pretty comfortable with a lot of the interfaces and we do a lot of things that the interfaces don't necessarily want to do so we need to be able to custom is it but so again tools like pfsense are good for a an end user type person but when they get into our realm we've got 200 VLANs which would all have really goofy rules defined on it I mean if you go in and actually do a listing of the PF rule set on a pfsense box there's things that they do things that rules that they put in there to make it easy so you don't stomp on yourself you know screw yourself up essentially we don't want that we want full control of it so we screw ourselves on the line yeah we do it just fine without having to have a GUI to troubleshoot so but we're done yeah so we're gonna but PF is much better than I be tabled Oh real fast we're going to sneak it in
we want to hear your feedback we want to know what you think you know what improvements you want to see it cetera et cetera so at least I'm going to be available and a couple other people are going to be available over there for at least a little bit if we have to we'll move to the real track 1 Q&A space so come talk to us and you can always email knock at Def Con networking door to with your questions and feedback thank you thanks thank you