Antipatterns
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Alternativer Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 34 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung - Weitergabe unter gleichen Bedingungen 4.0 International: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben. | |
| Identifikatoren | 10.5446/42800 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
ownCloud conference 201824 / 34
1
3
4
5
7
19
24
00:00
Web logSystemverwaltungGüte der AnpassungDruckverlaufp-BlockBitPunktwolkeCodeWeb logComputeranimationVorlesung/Konferenz
01:10
HackerVideokonferenzÄhnlichkeitsgeometrieGruppenoperationPunktwolkeRechenschieberSoftwareentwicklerHackerWort <Informatik>Rechter WinkelProgrammierungVirtuelle MaschineComputeranimation
02:39
VideokonferenzSystemverwaltungExogene VariableAntimaterieAppletSystemverwaltungSoftwareentwicklerInternetworkingGruppenoperationHidden-Markov-ModellBitGüte der AnpassungExogene VariableDatenverwaltungDatenstrukturSoundverarbeitungSoftwaretestComputeranimation
05:04
SpeicherabzugKlasse <Mathematik>InstantiierungThreadKonstruktor <Informatik>Faktor <Algebra>Faktor <Algebra>ProgrammierungAppletSystemverwaltungComputerspielSichtenkonzeptFigurierte ZahlComputeranimationVorlesung/Konferenz
05:47
HackerNeuronales NetzSoftwaretestDatenverwaltungHackerKeller <Informatik>Schnitt <Mathematik>Prozess <Informatik>Gebäude <Mathematik>MultiplikationsoperatorSystemverwaltungPasswortWurzel <Mathematik>Hidden-Markov-ModellBitMagnetbandlaufwerkGüte der AnpassungBesprechung/InterviewComputeranimation
07:34
Physikalisches SystemProgrammSoundverarbeitungMultiplikationsoperatorGüte der AnpassungStandardabweichungDatenverwaltungSystemverwaltungPhysikalisches SystemZweiKurvenanpassungMaschinenschreibenSpieltheorieAuflösung <Mathematik>CASE <Informatik>MusterspracheMessage-PassingWort <Informatik>Rechter WinkelOvalComputeranimation
11:18
SoftwarePatch <Software>Physikalisches SystemProzess <Informatik>SystemverwaltungSoftwareComputersicherheitPhysikalisches SystemWeb-SeiteRechter WinkelDatenbankBenchmarkPatch <Software>BildschirmfensterSoundverarbeitungDoS-AttackeSpezifisches VolumenMailing-ListeE-MailEinsVersionsverwaltungp-BlockTermDeskriptive StatistikDienst <Informatik>Computeranimation
15:16
ComputersicherheitPunktwolkePrognoseverfahrenSinguläres IntegralInformationsverarbeitungSystem-on-ChipE-MailKünstliche Intelligenzp-BlockSoftwareMalwareEindringerkennungVirenscannerSnake <Bildverarbeitung>ComputersicherheitPunktwolkeVorhersagbarkeitComputeranimation
16:13
SpeicherabzugPhysikalisches SystemWeb logMathematikDienst <Informatik>Patch <Software>VersionsverwaltungServerZustandsdichteDeterministischer ProzessSoundverarbeitungSnake <Bildverarbeitung>Physikalisches SystemMultiplikationsoperatorParametersystemBrowserBefehlsprozessorPatch <Software>SoftwaretestSpeicherabzugNummernsystemInstallation <Informatik>VersionsverwaltungProgrammfehlerVirtuelle MaschineMAPPunktFormale SemantikSoftwareZusammenhängender GraphEinsGüte der AnpassungSystemverwaltungStabilitätstheorie <Logik>Open SourceQuellcodeService PackZahlenbereichDatenbankInformationsspeicherungSubstitutionGatewayFirewallWeb-SeiteProgrammierumgebungLoginGerade ZahlSpezielle orthogonale GruppeDifferenteRechter WinkelMalwareFamilie <Mathematik>Computeranimation
23:43
Machsches PrinzipDatenverwaltungStandardabweichungE-MailWeitverkehrsnetzGraphiktablettSicherungskopiePunktwolkeTLSSchlüsselverwaltungWiederherstellung <Informatik>PASS <Programm>Desintegration <Mathematik>SystemverwaltungSkriptspracheZusammenhängender GraphMultiplikationsoperatorHidden-Markov-ModellBeobachtungsstudieRechter WinkelMeta-TagKontrollstrukturSuchmaschinePublic-Key-KryptosystemTablet PCRechenzentrumMereologieSoftwareSpieltheorieVersionsverwaltungSolitärspielElement <Gruppentheorie>IntegralPunktwolkeSoftwaretestSechseckVirtuelle MaschineMailing-ListeProdukt <Mathematik>EinsProgrammierumgebungDoS-AttackeKryptologieMini-DiscFormale SprachePunktVirtuelles LANNotebook-ComputerSystemaufrufCompilerWiederherstellung <Informatik>KonfigurationsraumProgrammfehlerSicherungskopieInternettelefonieInformationsspeicherungElektronische PublikationTLSFreewareKernel <Informatik>E-MailRegulärer Ausdruck <Textverarbeitung>DatenverwaltungSchlüsselverwaltungDifferenteComputeranimation
31:13
SystemverwaltungIntegralFormale SpracheKlassische PhysikPhysikalisches SystemInhalt <Mathematik>Vorlesung/KonferenzComputeranimation
32:01
AppletPunktwolkePunktwolkeVirtuelle MaschineRechenzentrumMultiplikationsoperatorEinflussgrößeSoftwarewartungComputeranimation
33:19
ComputersicherheitPunktComputersicherheitBildschirmmaskeNichtlinearer OperatorHilfesystemBitFormation <Mathematik>E-MailComputeranimation
34:05
SpezialrechnerStatistikNetzadressePhysikalisches SystemMailing-ListeParallele SchnittstelleOffene MengeMultiplikationsoperatorNabel <Mathematik>WikiPunktRechter WinkelComputersicherheitOpen SourceComputeranimation
37:05
Machsches PrinzipProzess <Informatik>Inverser LimesKonfigurationsraumDateisystemSoftware EngineeringAnwendungssoftwareOrakel <Informatik>Web logRechnernetzSoftwaretestSpezialrechnerPunktwolkeHeegaard-ZerlegungGebäude <Mathematik>SchlüsselverwaltungInstallation <Informatik>SystemverwaltungComputersicherheitRegulärer GraphBootenSpielkonsoleArithmetischer AusdruckRechter WinkelOpenViewProzess <Informatik>Güte der AnpassungMultiplikationsoperatorFirewallEigentliche AbbildungSchlussregelMomentenproblemMailing-ListeReelle ZahlMinkowski-MetrikVirtuelle MaschineAbfrageInstallation <Informatik>Dienst <Informatik>SoftwarePunktwolkePhysikalisches SystemApp <Programm>InformationsspeicherungGesetz <Physik>Front-End <Software>Produkt <Mathematik>SoftwareentwicklerResultanteQuellcodeDruckverlaufMonster-GruppeSystemverwaltungDatenmissbrauchLoginNabel <Mathematik>PunktHackerATMBitComputeranimation
43:05
PasswortServiceorientierte ArchitekturWurzel <Mathematik>BinärdatenPhysikalisches SystemMereologieCodeZweiWechselsprungRoutingPhysikalisches SystemServiceorientierte ArchitekturPasswortProzess <Informatik>SkriptspracheBildschirmfensterQuellcodeBinärcodeMultiplikationsoperatorFormale SpracheSystemverwaltungPunktRechter WinkelRPCWurzel <Mathematik>Computeranimation
44:57
AutorisierungAnwendungssoftwarePhysikalisches SystemBootenAnalytische FortsetzungPhysikalisches SystemFehlermeldungGüte der AnpassungRechter WinkelBesprechung/InterviewComputeranimation
45:55
Physikalisches SystemProgrammiergerätSkriptspracheE-MailZeitbereichMalwareWort <Informatik>RückkopplungArchitektur <Informatik>SinusfunktionAliasingInternetworkingBildschirmfensterPunktRechter WinkelProgrammfehlerRechenschieberSoftwareentwicklerSystemverwaltungProzessautomationGenerizitätMereologieGüte der AnpassungMultiplikationsoperatorHilfesystemInhalt <Mathematik>FreewareWeltformelCodeComputersicherheitPhysikalisches SystemRückkopplungDatenverwaltungComputeranimation
49:14
VideokonferenzHypermedia
Transkript: Englisch(automatisch erzeugt)
00:08
And welcome to the second day of the OnCloud Conference 2018. It is my great pleasure and honor to introduce you to Felix Leitner from CodeBlob.
00:20
He will give a talk on admin anti-patterns and maybe a little bit to the background of Felix von Leitner. He does the FIFA's blog. It's one of the top 50 blogs in Germany and has a lot of insights, especially on coding, best practices, and of course on admin stuff.
00:47
It's really a great pleasure to have him here. I wish you a good talk, a good day. And like yesterday, my name is John. If you need anything, don't hesitate. Just contact me. I'll be around here and help you
01:02
with anything you need. Thank you very much, and have a nice day. Good morning. So first problem, this talk I was asked to do here,
01:21
I originally did somewhere else at the Unix users group. So it's in German. At least the slides are, so I'll try to translate. However, you will lose the famous German humor. I hope you can cope. Anyway, so I was a little surprised when I first
01:42
thought about own cloud, and I thought I should have a slide about this. So the hacking community has their own little slang words. And one of them is about taking over somebody else's machine. If you hack somebody or something, it's called to own it.
02:00
So the connotation is not good. Connotation is, oh, you hacked him so bad that you have more ownership rights than him now. So I was impressed by your naming of own cloud. That's like Oracle calling something unbreakable, basically.
02:21
So let's see. Again, I did this talk already. Actually, I did another talk similar at the 34C3, which is the annual CCC conference. Was also called anti-patterns, but it was about programming, software development. And then I was asked to do another one, but with a little spin on admins for the German Unix
02:43
Users Group. And I found out in preparing that talk that there's actually more to say about admin anti-patterns than about software development anti-patterns. Both talks are available on the internet. So if you want to see them, maybe you already saw them. There will be some overlap if you know them already.
03:01
So one of my problems is that I have no idea who's in the audience here. So I was like, hmm. So I asked John, and he says, oh, it's all admins. They will know. So let's do a little test. If you know what this does, please raise your hand. I don't need the details. I just want to know.
03:21
OK. All right. What about this? OK. Who knows how to do this by hand if it's not available?
03:40
It's getting more complicated. OK. Thinning out a bit. OK, good. I think we'll manage. So anti-patterns. An anti-pattern, the dictionary definition anti-pattern is something you do in response to a recurring problem that you hope will solve the problem,
04:01
but usually it explodes in your hands and is usually ineffective or even counterproductive. So I gave the talk a little motto, and I stole it from Mark Twain. I think it's one of the most profound insights
04:20
in the world. If you can read and don't, you're no better than someone who can't read. So don't be happy with Wikipedia being there, and you can look something up. Learn it. There's a little structure to the talk. It's like a running gag, really.
04:42
So it starts with some kind of problem. You have some problem, and then someone does something. And then this is how they do it. And then the effect, which is usually bad because it's about anti-patterns.
05:01
So this joke is really old and stale, and I wouldn't have done it. But it turns out there actually is a problem factory in Java, so I couldn't walk away from that. So about hacking and admins in general.
05:20
So I've been playing video games for a while, so my world view was influenced by video games. This is an old one, and I'll show you some character pictures. As you level up, you notice the figure gets older. Right? It's even better with the wizards, so look at the wizards.
05:42
Ooh, the beard is growing. So I figured, OK, that's how you know that someone is a good hacker. Well, I mean, who knows who that is on the left?
06:02
That's Alan Cox. He used to write the IP stack in Linux 2.0. So that is actually a hacker. First time I saw him, I thought he was a homeless guy who went into the wrong building. Well, no, he's actually, he's a hacker, and he's a good hacker. So my first admin job, I expected something like this.
06:23
Glorious beards, and I walk in the room, and it turns out there's nobody with beard anywhere. All these young people and shaved old people. And the oldest guy was just changing tapes. So I was like, hmm. Nobody tested me, nobody asked anything.
06:43
They just gave me the root password. I was like, oh. I was a bit confused, and that's when I learned that being an admin is not about knowing all these stuff, all these skills that you need to be a good admin.
07:03
Admin is about being young and being happy with small payment. So you can deduct something about admins you see in the wild. There's like two kinds of them. First one wants to solve problems, and the other one wants to administrate.
07:24
There are some admins who don't actually want to see problems go away. They're perfectly happy with there being problems around because otherwise they wouldn't be needed anymore. That's not good. So basically two kinds of admin. The first one is the new guy, and he wants to see things work smoothly,
07:43
and he wants to learn new stuff. And the other one is basically staying in the way and saying don't touch anything. You can see both admins causing problems every now and then. So I've been trying to distribute
08:01
the anti-penants between them. So a usual problem in administration is something should be automated. Something you do all the time, or you don't want to do all the time. So there's two things that can go wrong. First one is you automate too late in the game.
08:21
And we're like oh, this is easy enough to do manually. We don't really need to automate this, and then you do it manually all these years with all the typos and wasted time. So that's a very common thing you see, but the other one is just as bad. If you go okay, let's do this now, and then it turns out nobody actually needs that.
08:42
So time is wasted. I think most of the wisdom of the world you can find on XKCD, so I brought three of them. This is the first one. Pass me the salt. And he's like oh, I'm working on a system to pass arbitrary condiments. Yeah, so this is a common problem.
09:01
Don't fall for this. So what about the common case of automation? You go okay, hold my beer, I'm automating this, and you go admin goes away for six weeks. It's not heard of again. And comes back not having solved the problem.
09:21
But on the way found 23 other problems and solved those. So I mean most of these are a little, I mean they look like oh, who's that stupid? But you see, I mean this is from experience. I've seen all of these in actual companies. System administration is usually done
09:41
by underpaid young people. So the management doesn't really enforce or expect quality as much as they should. And I think it's a co-dependency. Both sides are wrong. You should hold yourself to a high standard. So if the admin comes back
10:00
and solves unrelated problems, usually the solution is based on some prehistoric stuff that's only available on this old VAX that has not been patched. It does more harm than good. And the second next case, right? You expect it like the upper curve.
10:21
You go oh, I'm writing the solution that's a short time and then the solution takes all the problem away and have all this free time. But that's not how it works. And we know it doesn't work like that before we do it. Nobody should be surprised by this.
10:41
So if you do it like that, usually it pays off eventually but everyone will be long dead by then. So I'm trying to illustrate this here with a well-known thing from the future. So another effect is that while you're trying to solve the problem,
11:01
nobody is touching it because it's being worked on. So it stays there. And sometimes it lays eggs. So yeah, I mean, this isn't good. Even if you automate something, you still need to solve the problem,
11:21
not wait for the automation because it may never succeed. Usually you have this problem that because nobody works on the other problem because it's being solved by someone somewhere, that business processes get stuck, right?
11:41
I mean, this is all, you probably have seen this before in administration. So next problem after automation is that you have software and it sucks. Usually software you run has known problems. You have an old version or there is a patch available.
12:01
So you know the software is bad. So the obvious idea is you go to the vendor who sold you the software, get some patches, right? So typically you don't send your top personnel to the vendor, you send like the intern and the intern goes to the vendor and doesn't find anything.
12:22
This is on purpose, by the way, because the vendor doesn't want new customers to see how bad things are. So they hide the security stuff. Some vendors go so far as to hide it behind a paywall or you have to sign an NDA.
12:41
Database vendors have started to put in their terms of service that you cannot publish benchmarks about their databases, which I find really abhorrent. So I mean, this is how things are done, right? And you send not your best guy, but the intern, right? So the intern doesn't find anything and if the intern finds something,
13:01
usually the description of the patch is useless. That's on purpose too. Right, the official excuse is that you don't want to give away to the attacker how to attack the system. But that's, I don't think that's the truth. Vendors just, you know, they have all the spoiler plates so nobody actually sees how bad things are.
13:24
So what's the intern supposed to do, right? There are some security mailing lists you can go to. These are old school mailing lists, there are other ones. But these mailing lists are really high volume. If you subscribe to them, that's basically all you do, right? The intern goes away and can't do anything else
13:42
because there's this flood of security announcements. And it turns out, usually because it's the intern, he doesn't actually know what software you have. So he sees all these announcements and you know, gets a few of them that he recognizes
14:00
and the rest is just wasted. So all this stuff happens in the wild. I mean, it makes you wonder how things work at all. So but let's assume the intern is very bright and actually finds a patch that is applicable. Finds a patch, wants to put it on the system, obviously.
14:22
And then the other admin comes and blocks him. Right, the old guy. And the old guy says, oh, we need to wait a few weeks to see if the patch is good. Who here has seen that happen? Oh, that's surprisingly few. I think you're lying to me.
14:43
So this is very common. It's like, oh, no, we can't put out this patch. It might crash. And the effect is obviously that all systems in the company have all the known security problems of the last months.
15:01
No surprise here. And this is all self-inflicted. It's like this huge shotgun that you point to your foot. There is no need for any of this pain. So the next problem is we have all this malware
15:21
in the company. What do we do? Well, the obvious solution is to get some snake oil products like antiviruses, network intrusion detection, whatever, Xeam. I have a nice, whoops, I have a nice illustration here.
15:44
Usually you can easily detect snake oil by how bullshit the marketing is. So I collected a few that I found very nice. The predictive security cloud. How about some cognitive SOC, whatever that is?
16:05
Oh, and blockchain, of course. So in practice, I've actually seen that at a gig. They gave us a two-core system, and the first core was busy running semantic snake oil,
16:23
and the second core was busy running McAfee, and they killed each other all the time. So the second core was logging all the attacks it prevented, and if you wanted to open a tab in IE, which is the only browser they gave me,
16:41
it took like two minutes. System was just busy. Not just CPU was busy. Storage was busy too. Storage was busy because they put each other in quarantine all the time, and then reinstalled. So it was really bad, and you would think
17:02
that someone notices that before they give you the PC for work, but this happens. Another thing that happens is giant monitoring systems. Sometimes they actually really monitor you, like what keystrokes you do and the webpages you visit, depending on your jurisdiction, but in general,
17:26
the common thing you can see is that all this monitoring usually ends up somewhere. Nobody is reading that. So usually when I do a pen test for a company, I ask them if they saw me, and I'm not trying to hide.
17:41
That's not the point. I'm not being paid to hide. I'm being paid to try out problems, and I've been seen once. Everyone has all this huge monitoring stuff, but nobody ever reads the logs. So I call those heights lifter, which is like a heating appliance.
18:01
You know, they cause all this heat, and it's nice and warm, but that's about it. So we talked about patches. Sometimes there's a service pack. This is a little older problem. The older admins among you will know this. And there used to be this, of course you want to install it,
18:20
but there used to be this mystical idea, no, you can only take the even numbers or only the odd numbers, and the other ones are bad, which is the same thing as with the patches, obviously. So same thing as before, right? You get all this malware, and you can't get rid of it.
18:41
So this patch thing is really important to me because it's so easy to see how not to do it, right? So this is a common thing you hear. We only install patches that we really need. Who has heard that before? Most big companies have their own department
19:02
to find out which patches are needed and which ones aren't. And this sounds like a good idea, but it's not. So first of all, you don't know. You have no clue. How would you know which patches are needed and which aren't? The systems are so complex these days. Even if you don't use a component,
19:21
it might still be used by some other component. So you don't know, right? Secondly, I've seen this firsthand. If I go to a vendor and do a source code audit and find 100 bugs, they will usually fix a few of them, at least. But if you look at the next patch,
19:42
none of them will be mentioned. Why would they mention bugs that nobody outside knows except me and I signed an NDA? So if you get a patch from a vendor, you can be sure that there's at least a dozen really horrible things it fixes that they don't want to tell you about because it would make them look bad.
20:02
Install all patches all the time, right? The third one, this I think is the important argument for big companies. The patches are tested by the vendor but only under the assumption that you put in all the other patches.
20:20
So if you skip patches, then you're on your own. How would the vendor support you if you have this fractally different landscape because you skipped this patch and this patch? You don't know. And neither does the vendor. They don't even have a machine with your exact patch level.
20:41
So always install all the patches. If you forget everything else about this keynote, remember this, always install all patches. So sometimes a new software version is coming out. It's a related issue to the patches, right? So you want to install it and then the old guy goes, no, no, it's a point O.
21:04
We don't trust point O. We are waiting for point one. And GCC is an important open source package. They just changed the versioning scheme so there is no dot O.
21:21
They start with dot one because nobody installs dot O anyway. So I mean you can predict how this is going to continue, right? This is also a common problem in practice. You see all these really old installations and nobody wants to touch them.
21:42
So let's update it. How do you do it? Well you put in all the patches. Well some of these patches aren't deterministic. So the effect is that you have three old systems where you installed all the patches
22:00
but they're all slightly different. So this problem is common enough that these days it's one of the main arguments for containers like Docker. You don't install the patch, you install the container and then you know your installation is the same as it should be.
22:21
And this is also, I mean there's no reason why it should have been like that. And I'm assuming that Docker will have a similar problem at some point and then we'll have the next great idea. But all of these are self-inflicted. Okay, so let's update something. Different problem. And then you sometimes see that the admin
22:40
doesn't want to update. Right, the admin says no, no, we can't patch because we will lose support if we patch. And at first I thought they're making fun of me but there actually is environments where you have like an old Solaris and you have some kind of Oracle database on it
23:01
or whatever, you know, substitute. And if you patch the Solaris, the Oracle support will tell you you're not supported anymore. This actually happens. I've seen this happen with Telco equipment like the SMS gateway at your Telco and what they do is they install three firewalls before it as if that, you know, does anything.
23:23
But you know, this is what happens, right? So we can't patch it, we lose support. And then you often hear, well it's not old, it's stable. Yes, it's old. Stable is bullshit. Then it's not stable, it's known bad.
23:43
We know it's bad, there is an update. You know, it's not, this continues to amaze me. You see all these really ancient machines and you go, who's using that? And they go, oh, nobody's using that. You know, it's too old, all the commands don't work. Well, yeah, you get the strangest excuses like,
24:02
well, we can't turn off SSL version two because we're talking to this, you know, mythical machine, we don't even know if it's there anymore. A customer of ours told us we can't turn off SSL two. Yes, you can, you have to, it's broken. So it's not stable, it's known bad.
24:22
I would like to change the terminology here. It's not stable, it's known bad. You have to understand that new releases don't fall from the sky, you know? Somebody is releasing them. They have a reason. You don't release something because you have free time. You release something because, you know,
24:41
the old version is bad and you have a better version. It's easy. Yeah, this one is starting to gain ground. You go like, oh, I can't work here. The admin is too strict, you know? All this stuff is configured too close, I can't install software.
25:01
I need my solitaire game. And then they go, oh, I want to bring my own device. And, you know, this is less of a problem now than it used to be. It used to be this, you know, your tablet isn't part of our device management, you can't bring it, you know? So what happens is that the CEO is forwarding his mail
25:22
to Yahoo and reads it with the tablet and the guest Wi-Fi. You know, I've seen this dozens of times. This is what happens if the admin tries to lock things down, it doesn't work. You have to work with your people, not against them.
25:41
So this is like a meta problem. You know, oh, it doesn't work, it's horrible, and you have to diagnose it, and the first study is, ah, the product is shit. And then, ah, the vendor is shit. Notice how the exclamations marks grow.
26:00
And then, oh, it's a compiler bug. You know, the excuses get more and more absurd. No, no, it's a kernel bug. This is like the, your hunters have to usually tell these stories about these mythical beasts they slew in the wild. Same thing with admins. And usually when, you know, someone looks over his shoulder
26:22
and sees, well, you missed a semicolon here. Don't go in that rabbit hole and go, oh, it's the compiler. No, it's probably not. I mean, there are compiler bugs. But your problem is probably not a compiler bug.
26:43
Most of these are really trivial if you look at them, so it's easy to make fun of them. But they will happen to you at some point. So this is a typical problem for admins. We want to know which of our hosts are online. We have this data center with all these machines. We want to know which ones are working, right? So let's do some monitoring, obviously.
27:05
So, hmm, you need to find out which hosts are there, so you need to talk to them. Where do you get the list of hosts? And I've seen this a couple of times. They get the list from pinging everyone and seeing who answers.
27:22
So you ping all the machines you know are there and are answering, and surprisingly, everyone is there. Right? So always, always look closely where you get your data, because stuff like that happens all the time.
27:43
So what do we do when something breaks? Typical admin problem, well, we have a backup, right? Who has a backup? Oh, okay, okay. Yeah, I've seen this a couple of times. Oh, you know, storage is expensive.
28:01
Let's put it in the cloud. Or, you know, maybe you have this deal with whoever, and they give you free cloud storage, and you say, oh, we have a few terabytes of cloud storage Let's put the backup there. So you put the tarball in the cloud, and it contains all the config files with the credentials and the personal data from your users,
28:23
the TLS keys, all of this has happened, you know? The private keys, obviously. The disk crypto recovery keys. All of that shit is in the cloud. And it usually gets found because, you know, there are search engines for that. One of them is called Shodan.
28:42
People find stuff on Shodan daily. It's so much that nobody actually realizes, usually, if something gets found, because it just gets drowned out in the flood. There's so much stuff in the cloud. So don't put your stuff in the cloud. This is a classic in the banking environment.
29:01
Oh, telephone calls are so expensive. Let's use voice over IP. And then they go, well, we have the switch, and there's a few free ports. And then it turns out that the Cisco voice over IP phones want a trunk port. So if you pull out the phone and pull in your network,
29:22
your laptop, you can go into any VLAN. And it's a different department, see? The people who bought the phones are a different department than the people who set up the network. So nobody realizes. This used to work in all of the banks.
29:45
So what if we need some integration? We have all these components, and we want them to talk to each other. I know scripting languages. Let's do scripting. And usually some admin hacks something.
30:02
The idea usually is like this. Ah, another XKCD. Like, I know regular expressions. And usually it works like that too. You have like amazed people on the ground, and the script is like, ooh. But you just want to solve the problem.
30:20
There's no quality assurance, no quality testing. Usually there's not even a version control system. There's no ticketing. It's just a small integration script. So in practice, the quality of integration is much lower than the quality of the products, the components you're trying to integrate. I don't know why this keeps happening,
30:43
because it makes everyone look bad. Usually you hear all these complaints in companies about how bad the components are, and it's not actually the components, it's the integration that your own people did. But, you know, the admin is like, well, you know, I logged in and it worked, so that's it.
31:04
Usually admins have enough on their plate that they lose interest immediately and start doing other stuff. Right, so I've seen a few companies, actually, I have an illustration for that. I've seen a few companies where the admin did the integration in some language they wanted to learn.
31:25
Not something they knew, something they wanted to learn. Right, okay, this is another classic. Oh, it's very slow, we need to find out why it's slow, so let's deploy some agents. And this is my favorite picture in the world.
31:41
If you look closely, you see it's completely useless. Everything's like connecting, loading, please wait, searching, it's like zero content, and usually monitoring systems are like that. They give you like, woo, Starship Enterprise. And there's no content.
32:02
But all the agents are deployed to take the measurements, really slow things down more. So yeah, desperation swapping from monitoring is a thing. Obviously everything is even slower then. This one really makes me sad because I heard it
32:20
and I thought it's like a satire or something. So it was a company and they outsourced their data center to some other company. And they told them, you have to keep all the machines. But we want you to guarantee us an SLA. And it has to be cheaper than what we're doing now. And you have to keep the people.
32:43
So how is the company supposed to do that? They skimped on support, obviously. No maintenance, everything broke down all the time, and this happens all the time. If you outsource something with unrealistic goals, then they're not gonna get met.
33:01
So the next step is, well, we go in the cloud. And I was like, why would you go in the cloud? The previous deal already fell apart. Why would the cloud be better? And then the obvious counterargument is if you put your data in the cloud, it's not your data anymore, obviously. But I got this excuse.
33:21
They said, oh, it can't get any worse than it is now. And AWS at least reacts to tickets we open. And I thought, how, what? Yeah, they don't help me, but at least I get a form mail, you know, form letter. We take your problems very seriously.
33:44
Yeah, okay, this is another one. Okay, I was a little bit self-deprecating. Usually you see companies at some point go, okay, we have so many security issues. We need a security team. Right, and they go, let's get the best of the best.
34:01
24-7 operations, you know, we call it our own cert. And you get people like this. And then everyone else in the company goes, well, it's not my problem anymore. And they keep on, keep on doing, causing more problems because, you know, security,
34:21
someone else is dealing with that now. Right? Okay, we're back to updates. This is the perennial problem. So I'm trying to illustrate something that, you know, sometimes I'm at a customer and I get to see what someone is typing, but I can't say anything, right? And you see them do something and go.
34:43
Right, so again, we're monitoring, right? And the guy goes, okay, easy. We need to deploy something. And then, hmm, where's the host list coming from? Right? Oh, it's probably in the Wiki. At this point, they usually go, pfft.
35:02
And they go, oh, let's go to the Wiki. And the Wiki is up again.
35:22
And then the host list is from 1997. Historical artifact. So okay, let's ping everything. Let's ping them all. And they go, okay. Pinging all of them. And then, you know, of course, command non-found
35:42
because it's a BSD. Whatever, okay, let's do it with the shell. And this is taking forever because every ping runs into a timeout for all the systems that are not answering, right? So it's like, oh, snore. Get a coffee.
36:01
At some point, I was so annoyed that I wrote a little ping that can do this in parallel. That's how bad this problem is, and it happens all the time everywhere. It's open source, you can have it. So I showed this to a customer, and he was like, oh, that's awesome. Let's extract the IP numbers.
36:21
And I kid you not, you know,
36:50
you could have waited for the other ping that time.
37:05
It's done. Okay, so I show them regular expressions, right? And they go, oh, I can do this. Right, so, you know, have you heard of AWK? Oh, yeah.
37:20
That's awesome. I heard this once, and I couldn't contain my laughter. I was like, oh, of course you could use HPE OpenView. Let's boot the console. Sure. Anyway, so this really gets on my nerves, because I think you owe it to yourself to know your tools It's like if you get somebody to fix your kitchen,
37:45
and the guy goes, oh, let's open the manual, find out what to do. I mean, it's like, what? No. So I like to call this the monster truck mode, because usually you have a problem, but it's not here, it's there. And on the way there, you see all these other things
38:01
that don't work. And you go, okay, I don't really want to deal with this. I want to deal with that. And you've fixed none of them properly, if at all. Right, so it's like you have this monster truck, and you drive over everything on the way. And the result is obviously that all the broken stuff
38:22
stays broken, right? This is a real problem in big installations. Everything is broken, everybody knows everything is broken, but nobody's fixing anything. They only fix the stuff that is halting production now, and the rest gets worse.
38:44
So everything stays broken, right? The result is something like this. You're like, not my job, you know? Let someone else fix that. Yeah, this is also very common. It's like, never touch a running system. Yeah, you do touch a running system.
39:00
That's your job. That's what you're being paid for, to fix the running system. So it continues running. And you see all these temporary hacks and fixes. And nobody ever removes something that is not needed anymore, but you just,
39:21
oh, let's add some more. Also, there's always time pressure. There's never a moment where you go, oh, I have time now, I can fix this properly. No, that doesn't happen. If you wait for that, it's not gonna happen, right? Problems just stay there. And sometimes they get worse.
39:42
They usually get worse in a moment when you don't really need it. This is one of my favorite pictures. When I go, oh, I think we missed a button there. Yeah, I already mentioned this. I mean, I'm not here to teach you how to use the bone shell,
40:02
but you really should learn that. It's like the least you can do, like literally, it's literally the least you can do as an admin is to learn your tools. Okay, I'm not going through all of these. The last one has probably bitten everyone in this room at some point. Oh, there's a space somewhere.
40:21
And one of the five names, it doesn't work. Yeah, learn your tools. Really, I mean, there's no excuse. It's not like your physician. You're the admin. That's also really annoying to me. If you go somewhere and you see,
40:41
well, you could have fixed it in five minutes, or you can manage it in half an hour. And they go, let's manage it. And then it still crashes, but it restarts. And for some people, that's more attractive than fixing the problem. I really don't understand that.
41:00
I mean, you see that all around you, even on Google. Sometimes the query gets stuck. And that's because some backend crashed somewhere. And you know, it's being restarted, but still, it's not working. Okay, the next one is quite common, too, when they go, well, I could do this, but it's not my job.
41:20
We have a department for that. Usually, it's like the firewall department. It's not my job. So we call the firewall department. And it takes about six months. And then the rule is in, and you can't get it out again. So this is like high comedy. If you ever have spare time,
41:40
and you know someone working in a firewall department, let them show you the list of rules. Half of them will be useless, not needed anymore. That machine doesn't even exist anymore. Same thing for logging. You have three teams. They have three different log destinations.
42:01
And they don't talk to each other. And sometimes they even cite data protection laws to not talk to each other, which I find really, really awesome. It's like, that guy is the enemy. I'm not giving him anything. Yeah, this is more development thing, but I'm still listening in here because sometimes people who move on the cloud
42:21
have the same kind of issue. So if you develop software these days, you upload your sources to GitHub, and then the rest is done by some service somewhere else in some other cloud, and you don't even know who's really doing it. And then people deploy the result into some app store without having seen it.
42:40
We have no reason to trust any of this. And when you talk to people about this, they go, well, I trust this more than our own people because they are fucking stupid. Well, hire good people then. Okay, this is a bit esoteric. I'm gonna skip this, but I'm showing you the picture because it's nice.
43:09
Sometimes in companies, they close the USB ports, and you can't log in remotely. You have to go through three jump hosts. And to log in in the jump host, you need to go through some common password broker tool.
43:22
And then in the end system, everyone is always logging in as root. So all that stuff is just useless. I don't know why people do it. One of the highlights of my career was when I was being asked to build a monitoring system for Zeeam.
43:41
So the M in Zeeam stands for monitoring. The guy bought a monitoring system and was, I don't trust that thing. We need a monitoring system for the monitoring system. Yeah, I've actually seen this happen too. So the groovy thing is not theoretically, right?
44:01
So the admin needs to do something, and it takes a language he wants to learn, not a language he knows. Maybe it is a language he knows, but usually it's something they think would look good on the resume. And then they go, okay, I'll fix this in small talk. And then obviously, instantly, that fix becomes part of the critical infrastructure,
44:21
and the admin at some point leaves. And then you have a small talk script that nobody understands. Actually, I've seen in the Windows source tree, binaries that someone built at some time, and they don't have the sources for them. But it's part of the build process. So this happens to everyone. It's not, I'm not talking down on small, incompetent companies.
44:42
This happens all over the place. Usually, I like to think of this. You have all these old people who understand stuff that if they die, it's gone. These are the Navajo code talkers from the Second World War. Yeah, also that.
45:01
It's like, you break something, admit it, own it. Own it. Don't go Hanzolo. You need something called blameless post-mortem. It needs to be possible to discuss what caused the problem without people leaving the room because they think they're being singled out.
45:22
Right, you have to establish that. It's a cultural thing. This one really, really pisses me off, too, when you see, oh, no, no, we're not rewriting that. It's still good, right? Everything, the whole system is really bad. You can't even boot it without error messages. And they go, oh, let's continue.
45:41
It still works. See, I can still move. It's not that bad. So usually, when I have this slide, people think of Windows, and there's a point, right?
46:03
There's a point about looking down on Windows, all this crap that the internet is trying to give you, but it's not a Windows issue. It's a generic issue. People install stuff they don't really understand. Notice the part when he says,
46:21
well, every time I remove it, after 24 to 48 hours, it's back. Yeah, it's the invisible hand of the market. Bullshit. All right, I'm done with most of the content. In the end of my talks, I usually have a few slides with advice, free advice.
46:41
Take it or leave it. So if you're management, you need to be happy if someone finds a bug. Don't punish anyone. Even the guy who wrote the code, that's buggy. Don't punish them. Give them an opportunity to learn. Show them the bug. Don't have anyone else fix it. Have that guy fix it.
47:02
And make sure you have a feedback system in your company. I've seen places where the security team fixes bugs and the developer doesn't even realize he made a mistake. Obviously, that's the same mistake again. Nobody told him. You need feedback. You need a culture of being able
47:20
to own your bugs and problems. So if you're management and you're planning a system, don't tell them how to do it. You don't know. If it's obvious enough that you can tell, then you're no help. If it's not obvious, you're probably making it worse.
47:41
Trust your people. If you're an admin, this is the goal. Never have emergencies. Automate everything until you have free time all day and then learn new stuff. That's your goal as admin. If you're doing firefighting all day,
48:04
there will be never any end to that. Some people still do it because they think they're not needed anymore if they fix everything. That's not how it works. So this is for anyone.
48:21
Don't look to others. Look to yourself. If you orient your own self-worth on others, it's gonna fail. There is no good outcome. So if the other guy is about the same, then why would you put in more work? It's good, right? No motivation. So if the other guy is worse than you,
48:41
you're ahead. No need to put in more work. If the other guy is better than you, you go, oh, I have no chance. So you give up, right? Never look to others. Look to yourself in the past. Be better than yesterday. That's it. If you have any questions,
49:01
I think I've already exhausted my time. But you can email me if you want to. And I'll be here for a few more, like half an hour or something at least. Thank you.