We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Debugging apps running in Kubernetes

00:00

Formal Metadata

Title
Debugging apps running in Kubernetes
Subtitle
An overview of the tooling available
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
New tools are coming out to make it possible to add breakpoints and debug running code in a Kubernetes Pod. This talk will present an overview of some of these tools. We'll cover tools that make it easy to update the code that's running in a Pod (Squash/Ksync/Telepresence). And we'll also cover how to connect your IDE to the code and set breakpoints. A really brief, high level overview of k8s pods & how to run an application there List out some the tools available that make debugging possible Overview of how each tool differs A demo of 2 different debugging tools
Mobile appCore dumpDemosceneWeb pageProcess (computing)Cartesian coordinate systemMultiplication signLetterpress printingProduct (business)DataflowCodeLine (geometry)Tracing (software)Open setDebuggerComputer animation
Cartesian coordinate systemCore dumpMedical imagingConnectivity (graph theory)Volume (thermodynamics)DiagramProgram flowchart
Different (Kate Ryan album)QuicksortPoint cloudIntegrated development environmentCartesian coordinate systemCrash (computing)Information securityMereologyConfiguration spaceVirtual machineProduct (business)AreaProgram flowchart
Physical systemInformation securityIntegrated development environmentPower (physics)Software developerProcess (computing)Cartesian coordinate systemComputer animation
Virtual machineComputer fileMedical imagingDirectory servicePhysical systemCASE <Informatik>Virtual machinePoint (geometry)1 (number)Directed graphPhysical lawPoint cloudBookmark (World Wide Web)Control flowMathematicsSet (mathematics)SynchronizationIntegrated development environmentComputer animation
Virtual machineCAN busVideo gamePoint (geometry)Flow separationCartesian coordinate systemComplex (psychology)Connected spaceControl flowPlanningNamespaceSoftware bugIntegrated development environmentConfiguration spaceWrapper (data mining)Java appletComputer animation
Virtual machineResultantVirtual machineTelepräsenzMobile appMultiplication signSoftwareVirtuelles privates NetzwerkSet (mathematics)Medical imagingConfiguration spaceSoftware developerCartesian coordinate systemDirection (geometry)Local ringIntegrated development environmentLimit (category theory)Point (geometry)Software bugTouch typingJoystickProduct (business)Control flowComputer animation
Virtual machineFormal languageData structureDirected graphMedical imagingCASE <Informatik>UsabilityComputer fileCartesian coordinate systemMultiplication signDemo (music)CodeSynchronizationComputer animation
Virtual machineMultiplication signComputer fileProcess (computing)Software developerIntelligent NetworkInterface (computing)FamilyThomas BayesBitCodePoint cloudOnline helpSynchronizationVirtual machineLocal ringIntegrated development environmentSoftwareLimit (category theory)Program flowchart
SimulationDemo (music)Multiplication signRow (database)Random matrixShared memoryTouchscreenComputer animation
WindowArrow of timeRevision controlCASE <Informatik>GoogolMobile appCodePhysical systemDefault (computer science)NamespaceVotingCartesian coordinate systemVirtual machineOnline helpCubeComputer animationSource code
VotingWindowLimit (category theory)Menu (computing)Standard Generalized Markup LanguageConnectivity (graph theory)Software developerCodeCartesian coordinate systemDirectory serviceSoftware repositorySet (mathematics)Computer animationProgram flowchart
Configuration spaceSampling (statistics)CodeComputer fileComputer animation
Moving averageInterior (topology)VotingCoefficient of determinationConfiguration spaceComputer fileSoftware developerMobile appComputer-assisted translationCodeComputer animation
Interior (topology)Moving averageDebugger2 (number)Hand fanMultiplication signResultantConnectivity (graph theory)Installation artDatabaseIntegrated development environmentVotingQueue (abstract data type)Computer-assisted translationOffice suiteCoefficient of determinationCASE <Informatik>Source codeComputer animation
VotingWindowDefault (computer science)Simultaneous localization and mappingExecution unitTendonResultantComputer-assisted translationStructural loadCoefficient of determinationCASE <Informatik>Computer animation
ArmIntegrated development environmentMobile appInstance (computer science)VotingPhysical systemCASE <Informatik>Gene clusterComputer animation
SimulationCASE <Informatik>Computer animationLecture/Conference
Duality (mathematics)Water vaporMultiplication signVotingVirtual machineMedical imagingDirectory serviceCASE <Informatik>TheorySubsetCodeComputer animationSource code
Integrated development environmentMedical imagingSet (mathematics)Directory serviceCodeConfiguration spaceCASE <Informatik>Cartesian coordinate systemVirtual machineSynchronizationComputer fileFormal languageIdeal (ethics)Source codeComputer animation
Installation artRandom numberView (database)SynchronizationDirected graphMathematicsComputer fileDirectory serviceDemo (music)CASE <Informatik>Computer animationSource code
VotingSchmelze <Betrieb>Computer-assisted translationDemo (music)TheoryDebuggerFront and back endsComputer animation
Conditional-access moduleVotingView (database)VideoconferencingDenial-of-service attackTheoryVotingMobile appDebuggerCartesian coordinate systemJava appletDemo (music)Process (computing)Software developerLocal ringVirtual machinePoint cloudComputer animationSource code
Hash functionComputer fileDrop (liquid)Parity (mathematics)Information managementCodePoint cloudComputer fileSoftware bugSoftware developerSource codeSource codeComputer animation
Control flowMoment (mathematics)DatabasePoint (geometry)Menu (computing)Software bugConnected spaceConfiguration spaceSet (mathematics)Computer animation
Electronic data processingMoment of inertiaModulo (jargon)Menu (computing)Planar graphMenu (computing)TelepräsenzVirtual machineCartesian coordinate systemConnected spaceMobile appComputer animationSource code
SummierbarkeitVoltmeterInstallable File SystemCodeDean numberTelepräsenzRight anglePasswordInstance (computer science)Thread (computing)Computer animation
Rule of inferenceMathematicsNormed vector spaceComputer configurationSoftware developerDifferent (Kate Ryan album)Limit (category theory)Plug-in (computing)Connected spaceMultiplication signTelepräsenzInstance (computer science)Set (mathematics)Direct numerical simulationResolvent formalismComputer animationSource code
SummierbarkeitDean numberExecution unitFatou-MengeAnnulus (mathematics)Direct numerical simulationResolvent formalismInstance (computer science)Control flowConnected spacePoint (geometry)Local ringCoefficient of determinationVotingMathematicsQueue (abstract data type)Computer animationSource code
Menu (computing)Normed vector spaceMathematicsControl flowQueue (abstract data type)Point (geometry)Computer-assisted translationCartesian coordinate systemDemo (music)Software bugMultiplication signComputer animationSource code
Annulus (mathematics)Musical ensembleInclusion mapWindowDemosceneComputer fontNormed vector spaceBulletin board systemSource codeDemo (music)CodeComputer fileConfiguration spaceCartesian coordinate systemService (economics)Default (computer science)Computer animationSource code
DemosceneAnnulus (mathematics)ModemDemo (music)DemonView (database)Lie groupComputer configurationComputer animation
DemonAnnulus (mathematics)View (database)SynchronizationConfiguration spaceCalculusLocal ringFormal languageMathematicsCASE <Informatik>CodeComputer animation
DemosceneOpen setPoint cloudComputer fontCodeContinuous functionSolomon (pianist)Normed vector spaceLemma (mathematics)Default (computer science)DebuggerLocal ringCalculusDemonHydraulic jumpPhysical systemCodePlug-in (computing)NamespaceDebuggerSource code
DebuggerRollenbasierte ZugriffskontrolleComputer fontOpen setJava appletWindowStack (abstract data type)Variable (mathematics)Thread (computing)Video game consoleSoftware bugCalculationMobile appNamespaceLetterpress printingPoint (geometry)Source code
Boom (sailing)System callVariable (mathematics)Thread (computing)Video game consoleDemosceneTerm (mathematics)Rule of inferenceService-oriented architectureSoftware bugStack (abstract data type)CodeLemma (mathematics)Online helpComputer wormCalculusServer (computing)Computer fileDefault (computer science)Physical systemProgrammable read-only memoryDemo (music)DemonWindowRootStructural loadMobile appCartesian coordinate systemMultiplication signMereologyReal-time operating systemComputer fileMathematicsCodeMetropolitan area networkBinary codeTime zoneBinary fileComputer animationSource code
Demo (music)CodeDemonExecution unitFacebookOpen sourcePoint cloudOpen setMultiplication signSource code
Transcript: English(auto-generated)
Great, so I was in a talk earlier today in the Go dev room from Andre where he was talking about advanced techniques of debugging Go, and he asked the audience, how do you debug?
And everybody shouted out, print line. And it's quite clear that debugging, you know, kind of lost its way over time. But at the same time, it's not because we've got this great dev room and there have been amazing talks from people talking about the debugging tools that are out there.
And I'm going to go through, this is mainly a tool talk, I'll do some demos, but the tools are still in their infancy, I'd say. And debugging isn't going to be your first problem when you get into Kubernetes. Your first problem is deployment. You want to focus on your CICD process, ideally with some GitOps techniques.
And then once you have a good flow of getting code into production, you then need monitoring. And there is a whole dev room on monitoring and tracing. Open tracing is a fantastic technique for finding out what's happening in your application. Once you got all that going, then on your day three, if you're that fast, you can
start debugging. And just so we're all on the same page, I want to just do some basic Kubernetes talk. Hopefully it's the end of the conference and you've been able to pick up at least enough where you don't need this. But just so we're all on the same page, while I keep going through, the container is the core, well, the pod itself, I guess, I think of as the core component in Kubernetes.
And in a pod, you have a container running. You can have multiple containers running. Ideally, it's preferably just one container, but there's no restrictions. You can have containers, you can have volumes with data in them or not, as you like.
There's networking capabilities that Kubernetes provides you. And this is kind of your basic concept of what you want to do with Kubernetes, is run your Docker image, run your application in a Docker image. It doesn't have to be Docker either, but primarily it is. And so those pods then run inside a node, and the node would be typically an actual
machine, and you can have multiple nodes distributed across all sorts of areas, and each pod can then be in different nodes and distributed out in all sorts of different ways. Kubernetes is there to provide the orchestration.
You tell it what you want, you give it the configuration, this is how I want my environment to be like, and Kubernetes does its best effort to give you that. So if a pod crashes, if your application has a crash and it shuts down, Kubernetes will do its best effort to spin that up. If the node gets lost, if you're in the cloud and they just take that VM away from
you, Kubernetes will do its best effort to shift those pods that were there to a new node or other nodes you do have. So again, it's a best effort technique based on what you want. So hopefully that's Kubernetes in a minute, and now into debugging.
And everything that I'm going to show you today I do not recommend to do in production. All these tools have huge security risks, let alone just taking your whole system down. So I gave a talk yesterday about ephemeral environments, I'd love to plug that, there's a lot of tools on how you can get set up in a non-production environment.
And I mentioned Andrey's talk earlier today, and there was a great talk from Lucas about one of the tools I'll talk about later that you can check out. And yeah, I mentioned that there's this one technique of just logging out what you
want to find in your application, but the debugging has so much more power that when you can utilize it, you can really get the best development process available. So with the tools, I want to start with two that I particularly find more, like, they're
my favorites. They're the first ones that I got these to work, and when I was able to set break points in my IDE on something running in Kubernetes, I thought, oh, this is great stuff. And I did it with the use of one tool with ksync, and the concept is that you have your file that runs in your pod, and it's just an HTML file, and you build that Docker
image and you put that file in it, then you run your pod in that file sitting there in the cloud or in your cluster, and you have that file in that Docker image, and you want to change that without having to recompile a whole Docker image and everything. And so ksync will sync a directory that's on your local machine with the directory that's
in the pod. And so then on your file and your local machine, you can change the file, and it'll get synced into that container. Your Docker won't restart. It just updates the file, and you now have the change that you wanted available to you in your running system. Once you have this going, you can then connect a tool called Squash, which is basically
a wrapper around debugging for Kubernetes. It'll spin up another container in its own namespace, which opens up the port connections, and so then your IDE locally will have that port connection configured.
So then when you're running the application and hit the point where you would have a breakpoint, it would then trigger in your IDE and pause the application while you develop. Squash is limited to Delve for Go and Java and GDP.
They had plans to support Python and Node.js this year, this past year, and didn't reach those goals. So of course, they're open for contributors to help get there. As I said, this was when I got these two tools working, and they each do their own
thing, so they don't have that extra complexity when something goes wrong. If something goes wrong with one of them, you know, this is where I need to focus my attention. You don't have one tool trying to do 70 things, and you're like, wait, what, is it this? No, it's there. So that's, again, the benefit of having these two separations.
There's another tool called Telepresence, which, when this works, this is even better. When this works, this is probably the best. The concept is that you have traffic coming into your cluster, into your pods, you know, communicating with other pods in your network and coming back out, and you want to debug something in one of these pods.
So with Telepresence, you create this tunnel into your local machine, and your local machine then has access to everything else that's in the cluster. And so, as I mentioned about not doing this in production, if you do this in production, and traffic comes in here and then goes to your local machine, and you're not running
the app on your local machine, that's the end of the traffic, right? Or, if you have a breakpoint sitting on your IDE, and somebody hits that breakpoint, yeah, that's the end of the, like, nobody's going to get back the results that they're waiting for. Telepresence, because it creates this interesting networking, at least when we develop it, or
we use it with our Go applications, we have to create a new VPN for this network, and you can't use another VPN while you have Telepresence running. So if you have, as we have a VPN for all our outside developers to be able to access
the internal networking, they're not able to use the VPN and develop with Telepresence. It's not compatible for two VPNs at the same time. At least with that technique of creating the network that uses Mac OS and Go. So the documentation on Telepresence gets into its limitations and what you can do
and how you can do it with different IDEs as well and integrate it with your IDE directly. So check it out. If you can get it to work, you're going to love it. If you can't, then try one of the other tools. There's also an important aspect with Telepresence is when you start the application locally, you know, if you have the settings in your pod that say, you know, which ports
something connects to, like, all the configuration that's available for the Docker image that's running in the pod, you need to know when you execute the command locally. If you need to know that this here is called DB on port 306, when you execute locally,
you have to know that it's DB 306 and so forth. Okay. So the next tool to talk about is Scaffolds, which basically combines the two of Ksync and Squash into one tool, and it's available in IntelliJ and VS Code as a plug-in.
And it works fantastic when you have just this smaller, like, the one application with less dependency structure, and it'll manage the syncing of the file into the pod. It'll know that if you have a compiled language that it'll need to spin up the new
Docker image, it'll spin that up for you, put it in the place that you tell it to, and pull it down, and start the pod again, and have everything running locally in your IDE, and you get a lot of fantastic usability when it works. And it was interesting, because I have this demo that I'm going to go through with this
application, and I was trying to get Scaffold to work with it, because I thought, oh, this is a great tool, this is how you should use it, but I couldn't actually get it to work on something that had this pre-built Helm charts that I already had my application and it was already there. You then have to do everything manually, and once you start doing things manually, something
doesn't work, and then you've got to figure out how to do it. So I need to get there, maybe I need to spend enough time with it. All these tools, as I said, they're in their infancy, they're going to take some time to get rolling, so don't expect to just flip it on and, ah, this is wonderful. So the last tool I wanted to talk about was the new IDE from Eclipse that actually
runs in the Kubernetes cluster. So all that networking and mismatch and trying to figure out how to sync files, you don't have to do, because your IDE is actually running there, with the pods started up with
the code that you are developing right there, so the IDE is a browser-based interface, but you just interact directly with it and get all the magic there. Lucas had given a great talk on this a little while ago, and it helped answer why, when I tried to set it up two days ago, I still was like, hmm, how is this going to work?
What's going on here? It's brand new, I think it's just been out a few weeks, so it'll be good pretty soon, I think, based on what I'm hearing. And the one limitation now is that if you're doing it in, like, your local machine,
if you're spinning up a minikube, as an example, and then installing it, that install process takes a little bit of time as well. And so it's not a quick up and running experience, at least if you're doing it locally. If you have a cluster in the cloud where it's running, you don't need that extra time of getting up and running, but if you have that cluster in the cloud,
other developers can then interact with the same code base, I believe. So demo time. I learned a lesson and recorded my demos, but what I haven't done was figured out how to actually give a demo based on a recording, so we'll see how this goes.
And yeah, it's not going good yet, because I didn't share the screen. That should go... Okay, so here I'm going to be demoing K-Sync, and I start out by running
minikube on the machine. And I'm using a slightly older version of Kubernetes, and not very older. Google only released it a few weeks ago, so I made it available in GKE a few weeks ago.
But that's because the application that I'm going to be installing there doesn't have the new API versions configured, so the app only works with 1.15 and before. So now everything's up. We have our cluster, we have default name space and everything, the Kubernetes system
name space, and I got Helm installed locally, but not in minikube, so install that. I have Armador installed, which again was in the talk I gave yesterday on how that works. So now I'm going to need to pull some code down, and there's this example voting app
that I went into that has these different components. And the idea is that I'm going to clone this repo, I'm going to be a developer of the voting app, I'm going to be this front-end developer working on a Python application, and so I need this set of code here.
And so I clone that code, copy it into another directory, and then I also want to have the Helm configuration for that code, and of course Docker doesn't give that in their sample file, but Codefresh had originally wrote it, and then I added it into Armador
and the documentation there, so we can easily pull down the Helm chart with the Armador configuration and merge that into the voting code. So as a developer of the voting app, this would be the code I would have locally with
my app.python, my Docker file, my Helm configuration. In the Helm configuration, I have the Armador file with the dependencies that I need, and
the Docker file that adds this app, and here we can see that there's cats and dogs that are available for voting, and in the example of using Ksync, I'm going to want
to switch from being able to vote for cats, because I'm just not a big fan of cats, but I love bunnies, so I'm going to work on making that happen. So first we're going to install the whole environment of all the components we need.
So there's the five charts to be installed. One of them is the voting app, and the others are the Postgres, the Redis, the results app, and the worker. Those got installed. Then we can see they're installed but not yet running, so we'll wait for them
to get running, which are 39 seconds, 50 seconds. Whew, that went quite fast. Time's flying. So everything should be running. The database is still initializing. Postgres is still a bit slow, but we should be able to now load up the front end.
So we'll use Minikube to open the front end. There's the cats and dogs, and we'll load up the results so we can see.
Yep, I voted for a dog. And here we go in and change cats to bunnies, maybe. Nope, not yet. First we want to set up Ksync.
So it's already installed, but it's not initialized into my cluster, so that runs. And then I start watching. And so Ksync will now just watch what's happening in the cluster. And then I'll have to create an actual instance to, I don't know,
to be watched, I guess. And from that, I'll need to know the, oh, maybe I'm showing you something else first. So here you can see that there's in the Kube system, so Ksync has to have these extra privileges too, which you,
again, wouldn't want to include in your production environment. So here it's just sitting there waiting and listening, and it's going to need the app selector for the pods. So the voting app has the label that will be needed
during the create. So first I show you some documentation on Ksync. Show you how I Google to get documentation. And quickly read through it all. Yep, yep, yep, yep, yep.
There we go. Get that selector. And so we need to take a look what our labels are in our pod.
There's a lot to read through here, but there's the voting app, and there's the selector. And I'm going to speed this up because we might be low on time. Hopefully I can talk fast enough where I make a directory. So I have this theory where, with Ksync,
it syncs the folder from your local machine to the folder that's in the Docker image. And in the Docker image, we have, hopefully, just a small subset of what you need to run. So in the actual code base, you have your Docker file, you have your application, you have all your configuration,
you have your IDE settings, and hopefully your Docker image is compiled and built without all that extra stuff and just the application you need. And so when I do a Ksync, I don't want to sync my directory of my code. I want to sync just an empty directory and get all the stuff from the Docker image first onto my machine and then copy things
from my local environment into that. And this is maybe because we work primarily in Go language as a compiled language, and you need to compile it anyways. I'm not going to have it in my local, in the directory of my code base. So I'm going to have to copy it over anyways.
And so here, I'm going in, and I'm showing you what's in the folder that's synced and then what's in the actual Docker image there. Now, of course, they built the Docker image with the Docker file. My whole ideal of separating out what's there is lost on this technique with this example. But you've got to do things right if you want to be right.
Baby steps. There we go. Now I change some bunnies. And I save the file. Again, it doesn't sync anything yet because I only saved it into that local directory. But then I copy it into the Ksync directory.
And we see that there it goes. It copied the file, reloaded. The pod didn't actually stop, but there's our bunnies. We can vote for bunnies. And, well, we voted for cats. That didn't work.
So I think that's the end of this demo. And then, whoops, how do I? So let's try to fix the bunnies on the back end. And my theory is that it's a problem in the worker.
And so we have the voting app in the front end and the worker that converts it into then showing the result app. And the worker is a Java application. So in theory, in this, I could have used Squash. However, I'm not a Java developer, so I ran into a
whole bunch of problems getting that to work. So in this demo, I'm going to use telepresence and be able to spin up the worker on my local machine while it's connecting to the Redis and the Postgres in the cloud. So I start up IntelliJ, go into the worker code.
And again, I took it from the example. I took their code. So this is me as a developer of the worker. I want to debug the worker. And I go into their source code. There's just one file.
And it does the connection to the Redis and to the database. And then it just waits for things to happen. And in a moment, I'm going to set up the configuration to be able to debug. Don't know what I'm doing there.
There we go. Set a break point. And then go into the menu and yep, yep, yep.
OK, so debugging. It's waiting for Redis, because I started the application, but I don't have the connection to Redis. I'm just running an app on my machine. So now I need to turn telepresence on.
And what I need to do that, so I have this Redis. It's actually interesting. So I already have the telepresence command set up. I'm going to swap out the deployment of the worker to be
a telepresence instance. And so I'm going to do that. I have to get my password right. And then telepresence starts up. Then I was going to show you about how there's ways of doing this with Minikube. There's plug-ins for development with IntelliJ.
There's a whole lot of different options here. And here you can see that I'm using the TCP connection with the VPN. So that has the limitation that only one telepresence. And so I'm still waiting for Redis. What actually happens with this instance is that the Redis DNS name doesn't resolve.
So I have to change the Redis to be the port. I think this is just setting on my local Minikube thing. But the DB resolves. So there we go. We're connected. We got Redis and the DB running. And then we're going to set a break point to see what
happens when we vote. And we're going to go and we're going to vote. We're going to vote for a dog. And it doesn't change because it hit the break point here. Yep, there it goes. It votes in the queue. And to speed things up, yep, so we vote for B. We can
see that it's actually coming in just as the letter A and not as bunnies or cats or anything. So it's not even a bug in the worker. It's a bug in the front-end application that I didn't change what I needed to change. So that was a nice demo. I have five minutes left to try to do a demo of Squash
that's not here. I uploaded a zip file with this code that I've got. So in VS code, I've got just this simple application with a Kubernetes configuration of just a service
and a pod running on port 880. The Docker file is really straightforward. I uploaded the default Docker file into Quay. And the application is really, I was looking at this. I must have taken this from somewhere because it doesn't
make any sense. But it gets the point across. And so if I first run it, kubectl apply. And then I've got to port forward.
So we can see that isAdd is true. So I'm trying to add option one to option two. So I'm trying to add two plus two, and it equals zero. So clearly there's a bug there. And so now I need to first set up the Ksync, because again,
Go is a compiled language. So I need to compile the changes and have it with the debug available. And so let's do a ksync-watch and a ksync-create. Nope, that's not it.
So it's in that name. Call it calc. Yeah, all this should be good. So now I've got this thing. Yep, there we go, running ksync-get. So now we're in the code.
So if I do make ksync, this is just going to build the application and save it into the ksync folder, which I think should have worked. And the port forwarding is still there.
And so in the code in here, I can do, I already installed the Squash plugin into VS Code. Yeah, that's basically all you need to do. And Delve needs to be installed. And then you start up Squash. And Squash is running in another namespace.
So kubectl get namespace. So there's the Squash debugger. It runs there. And as I start it up, it says, hey, you know, I've got all these namespaces. Where do you want to debug something? I've got it running in SQ. And here's the example app.
And which debugger do you want to use? Delve. And so now it should just start up. There we go. And I can set a breakpoint somewhere in the code and around here after this print calculating.
And load the app. It didn't work. What if I change this and then do another make?
That's probably why it's not actually updating the binary. kubectl get pods. I think I'm a little low on time here to find out why
this isn't working. But again, I uploaded this code base. And so based on part of what I said, you might be able to piece this together on your own. One last thing to show you is kubectl exec minus it p minus nsq bin h ls minus all.
So here we can see that the application was updated an hour ago, but based on time zone. The file got synced.
It just didn't change. Time's up. Thank you.