Deadcode - a tool to find and fix unused (dead) Python code
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 131 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/69420 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
CodeState of matterElectric currentStrategy gameType theoryCodePresentation of a groupUsabilityStrategy gameMereologyState of matterComputer animationLecture/Conference
00:47
State of matterCodeSheaf (mathematics)Electric currentShift operatorBit rateChannel capacityCognitionRevision controlInformation securityFraction (mathematics)Variable (mathematics)Rule of inferenceCodeUsabilityRevision controlChannel capacityParameter (computer programming)Local ringMoment (mathematics)CASE <Informatik>Numbering schemeAttribute grammarMeta elementAdditionOverhead (computing)Social classVariable (mathematics)Fraction (mathematics)Information securityPosition operatorEndliche ModelltheorieComputer configurationRule of inferenceVulnerability (computing)Multiplication signAuthorizationMatrix (mathematics)Data miningSoftware maintenanceComputer animation
05:23
Sheaf (mathematics)PrototypeCodeShift operatorType theoryMaß <Mathematik>Computer fileTendonLattice (order)Distributed computingCategory of beingTerm (mathematics)Bit rateComputer configurationEntire functionSocial classVariable (mathematics)Function (mathematics)Sorting algorithmInheritance (object-oriented programming)Execution unitData modelEndliche ModelltheorieCodeRule of inferenceFlagType theoryComputer fileDifferenz <Mathematik>Computer configurationUsabilityInheritance (object-oriented programming)Position operatorSocial classDifferent (Kate Ryan album)Endliche ModelltheorieMedical imagingTerm (mathematics)ExpressionMereologyProjective planeCASE <Informatik>Line (geometry)Positional notationComplete metric spaceTunisComputer animation
10:17
Sheaf (mathematics)Function (mathematics)Letterpress printingSocial classInstance (computer science)System on a chipOpen setShift operatorInstallation artCodeVariable (mathematics)Positional notationElectronic mailing listCommon Language InfrastructureFunction (mathematics)Computer fileLine (geometry)UsabilityTunisCodePosition operatorDifferenz <Mathematik>Computer configurationInstallation artComputer animation
12:28
CodeSheaf (mathematics)Finitary relationAbstract syntax treeComplex (psychology)Strategy gameMereologyCodePresentation of a groupNamespaceLine (geometry)BitCASE <Informatik>Task (computing)Resultant2 (number)Sound effectTheory of relativityAbstract syntaxSocial classAssociative propertyFiber bundleWeb browserMetadataHuffman codingFlow separationAbstract syntax treeComplex (psychology)Computer animation
17:26
CodeShift operatorPseudodifferentialoperatorFreewareSineSystem on a chip2 (number)Ordinary differential equationSatelliteOptical character recognitionDesign of experimentsObject-oriented analysis and designCodeStatement (computer science)Radical (chemistry)Plug-in (computing)Dynamical systemLibrary (computing)Fluid staticsFunctional (mathematics)Electronic mailing listChainDisk read-and-write headMereologyRight angleCASE <Informatik>Product (business)UsabilityLine (geometry)Speech synthesisVariable (mathematics)Real numberSystem callMultiplication signComputer configurationCode refactoringPrototypeBitAbstractionProjective planeMathematicsDoubling the cubeLogicProcedural programmingWeb pageNetiquetteImplementationSoftware developerAlgorithmInterface (computing)QuicksortRule of inferenceSoftware crackingMultilaterationComputer animationLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:04
Thank you. Today I'm going to present a tool which I'm still developing. It's called .code, and it is used to find and fix unused Python code. My presentation consists out of four parts.
00:22
Firstly, I will present current state of .code detection using existing tools. Secondly, I will provide a set of features of .code package. Thirdly, I will show how to use it. And finally, I will compare two different strategies
00:42
of detecting unused code. Let's begin with definition of .code, because there might be different definitions. For me, .code is no longer used, no longer needed code,
01:05
which should have been removed, but it was not. So this code is still in the code base. And this type of unused code creates technical debt,
01:22
because it consumes cognitive capacity, because you have to sometimes understand it. It consumes time, because unused code has to be maintained. For example, when a Python version gets upgraded, you still have to update all your code base,
01:44
including unused code. And in addition, unused code might increase security risks, for example, if unused code brings in outdated dependencies, which
02:04
have security vulnerabilities. So debt code could be compared with evil agents from the movie metrics, which are waiting to cause you trouble. And we definitely want to get rid of them.
02:27
I want to share a short story of mine. Once I have bumped into a pull request, which contained a new feature, but it also contained an unused class.
02:41
Probably the author wanted to use this class at first, but he decided not to. And forgot to remove it. And I was surprised that this type of unused class has slipped into a pull request undetected,
03:04
because we were using bleeding edge linters like rough with all the rules enabled. This made me curious and led to an investigation. I have reviewed all the rules implemented in rough,
03:22
and there are more than 700 of linting rules, and found out that there are only several rules which check for unused code. And those rules are for unused imports, local variables,
03:40
and unused arguments. So it's only a tiny fraction of all possible unused code cases in a code base. So rough is not capable of detecting globally unused code at the moment.
04:05
I have searched for other tools, and found the most popular and capable linter called vulture, which is for finding that code. I tried using it, but I have faced several issues.
04:26
The main issue was that vulture provided a lot of false positives, and there were no ways to tune out this behavior. Now, if an attribute is added to a model type
04:48
the CMR or even in class called meta, this attribute has to be also added to vulture configuration.
05:01
And this adds a huge overhead in using this vulture package. In addition, this package is only capable to find unused code items, and you have to remove them manually yourself.
05:25
I tried looking into ways to improve vulture, but I have concluded that this would require too significant modification. And I have decided to create a new tool called dot code.
05:44
And this dot code tool is implemented in Python, but I think that it might be re-implemented in rough and re-implemented in Rust and integrated into rough
06:05
eventually if it gets recognized. So my tool has three main advantages over vulture.
06:20
And these advantages are, firstly, it provides a lot of different options to tune the behavior. The second advantage is that it provides slightly more unused code detection rules.
06:44
And third advantage is that it provides an option to automatically remove unused code. And let's look into more details
07:01
about these three features. So regarding tuning out the false positives, I have identified three terms which
07:20
are highlighted in this image. These are name, body, and definition. So now it's possible to ignore part of expression or whole expression based on its name, not only the name itself.
07:44
Another way to ignore code items is by inheritance or decorators applied. You can list a decorator name or base class,
08:03
and all subclasses might get ignored. So here we have a simple, short configuration example which tunes most of false positives for Django projects. First, it ignores all meta definitions.
08:24
It ignores bodies of data classes, type decks, models, and, well, in my case, it removed most of the false positives.
08:46
That code supports 14 types of checks, and it is possible to disable these checks by writing no quality assurance comment in line
09:02
or by specifying these checks per file. There is one check for empty Python files, and this check is working only for files which are not
09:21
using dundernaming. The final feature is a fix option, which allows to remove those findings from the code base.
09:41
And also, it is possible to use dry flag, which allows to see the diff which would be applied to files, but those files are not getting changed. And sometimes it happens that after a cleanup,
10:04
file becomes empty, and that code removes that file completely. Also, if it's not named using dunder notation. So how to use it?
10:22
It is pretty straightforward. You have to install that code package by running pip install.code command. And you can use CLI command by providing a list of Python files which have to be checked.
10:42
And here we have a code example which contains a lot of unused code entries which are highlighted in purple. And all of these unused code items are reported in the command line.
11:04
And if we add minus minus fix option, the output is almost the same, but we get one additional line at the end, which reports that seven unused code items were removed.
11:22
And the file itself gets updated. I want to once more highlight that it is important to fine tune false positives before applying fix option, because useful code might
11:47
be impacted as well. So you can use dry option to double check whether all findings are being correctly removed. And it is also possible to specify a single file of which
12:05
diff has to be shown. And it is possible by providing file name to the dry option. So even if a lot of files would have to be updated,
12:23
you can see a diff of a specific file. So let's move to the last part of my presentation, which is about comparing two different strategies of detecting unused code.
12:42
And in general, detecting unused code is a complex task, because abstract syntax tree doesn't provide a relation between used code definitions
13:03
and defined code. So both vulture and dot code uses abstract syntax tree, but these two packages have to create this association,
13:21
this metadata between defined and used code themselves. And let's see how the strategy of vulture package works. So this strategy is implemented in three steps.
13:43
First step is for finding all defined names. Second step is for finding all usages. And first step is for reporting those findings. So in the first step, we find two names, foo and spam.
14:03
And in the simple code example, these findings are correct. And on the fifth line, we can see that foo class and spam method is being used. So used items are also found correctly.
14:25
So in this simple case, there won't be any findings. But let's complicate this simple example a little bit by adding additional class. This class is bar, and it has a single method, spam.
14:44
And this method has the same name as method of foo class. And let's execute those three steps once again. In the first step, we'll find foo, spam, and bar definitions.
15:01
And in the second step, we'll find foo, spam, and bar usages. And as a result, nothing will be reported, even though bar spam method was not used.
15:20
This is because Vulture doesn't take namespace into account. Let's see another strategy which takes namespace into account. So the same code example analyzed
15:44
using a different strategy would find foo, spam, and bar spam definitions as separate entries. And in the second step, only foo, spam usage would be detected.
16:01
And as a result, bar spam method would be reported correctly. Well, it's important to point out that there are a lot of cases when constructing a namespace is
16:25
hard or even impossible for a usage. So in those cases, a dot code falls back to Vulture strategy and compares names without taking namespaces into account.
16:47
And having this more precise way of detecting unused code has a nice side effect, which allows to prune Python code
17:03
tree more precisely and hence allowing to reduce the size of a Python code bundle. And this might be useful when code size is important, for example, when it has to be served in a browser.
17:22
And let's draw a conclusion here that dot code package prevents technical debt from appearing out of unused code. And I hope that this package will be useful to you. So thank you very much for your attention.
17:46
Thanks for the talk, Alberto. And thanks, everyone, for listening. We can have any questions. And please line up behind the microphone. So the problem you're solving here seems to be extremely similar to or maybe identical
18:02
to essentially a rename refactoring. The algorithm would have to do, if I've got a code base and I've got a name and I want to rename that name, find the places that have to be modified. Have you considered just leveraging an existing rename refactoring implementation and using that as your detector? Or are they different?
18:20
Maybe I'm wrong and they're actually different problems, but they seem very similar to me, which is why I ask. No, I have not considered, but I will. And OK, totally other question is, can you detect unused code that's unused because of logic?
18:40
You're detecting unused names, but if I have a chunk of code that can't be reached because the code will never, ever get there under any circumstance, do you detect that as well, or is this purely a sort of name-based dead code detector? Yes, there is a rule for unreachable code, for example, which goes after return statement
19:01
or any terminal statement, so yes. Oh, so if it can be statically identified as unused, but you don't do any dynamic? Yeah, yeah, it is static code checker. OK, it's very cool. Thank you for. Hi. If you have a library, most of the API functions
19:22
will never be called, so they might be detected as dead code, but you can list them under the dunder all variable. Is dead code able to detect these functions and don't list them as dead code? If you list your functions in the dunder all variable?
19:41
I think so. I can't answer for sure now, but probably it might be a new feature which will be implemented. OK, thanks.
20:01
Hi, thanks very much for your talk. So this is also half a feature request, but if you're seeking to close the gaps that Ruff leaves, maybe it's something worth thinking about. So this is just a toy example of the toy in my head, but it's more about the general idea. So say I've got code A equals B equals 1,
20:20
like a chain assignment. I then write code that does something with A. I write some code that does something with B. But then I decide to get rid of the code that does stuff with B. Oh, sorry, with, yeah, with B. So this will still technically be used because it is part of the chain assignment, right, which is really just two assignments after another.
20:40
But clearly, I'm not doing anything productive with it. So I'm fairly confident this would go through the cracks with Ruff. But it might be worth thinking about these kinds of cases as well, because to me, that seems like dead code, even though it is technically used. Yeah, for sure. It's a good insight.
21:02
And probably it will be hard to implement those checks. But yeah, definitely it could be added to requested features. Thanks for the speech. I have a small question.
21:22
Sometimes we have situations about unused variables that we don't need. We apply the variable, but it's the value, the return value of the function. And we just don't check it. So if we delete the whole line, it
21:44
means we will delete an important function. But we need to delete only variable. And is there any way how to solve that? Yeah, I have thought about this case. Currently, the whole line would be removed.
22:02
But I am thinking about an option which would allow to slightly adjust the execution. And it would allow to remove only the assignment, the variable, but leave the function call as it was.
22:24
Thanks. Hello, these next three questions are from our remote attendees. So the first one is, what are your thoughts on classifying code that is only temporarily dead? Real world example. Customer asks us to remove a third party API.
22:42
But they ask to enable it again after one year. During this time, we have done refactors around the surviving third parties, potentially simplifying due to less moving parts. If you remove the feature, we have to redo a little bit more work to reenact it.
23:01
Do plus one refactor abstraction EDC. If we keep it alive, it costs more to do those refactors, but less time to re-enable the future. Where is the borderline? Well, probably the answer is it depends.
23:22
It would be possible to kind of not apply that code fully for those places where it is expected that code will eventually be used once again. But as a good practice, as we kind of
23:43
try to keep only working code in our code base, it would be possible to use Git to restore that removed code later. OK, here's the second question.
24:02
Why did you decide to write an own package project to solve the problem instead of adding missing, lacking functionality to rough, pile int, et cetera? May that code be someday integrated into existing linters? The main reason was that I am now
24:21
able to prototype and experiment way more flexibly and faster. And I kind of tried to imagine how modification of vulture package would look like. And it seemed that I had to add a double amount of code
24:47
than there was in vulture package. And some of those changes were breaking. So I kind of decided that it is way easier to create a new tool instead of trying to fight
25:05
the owners of vulture package. And I'm currently prototyping and trying to standardize this procedure. And when it's done, when the dead code package is stable, I'll probably try to migrate the logic into existing
25:23
linter, most likely to rough. And I saw that there is a ticket in rough GitHub page which discusses an interface for external developers
25:45
to write their own plugins. So I hope that this interface will be created and we will be able to create plugins for rough. OK, thank you.
26:01
Thank you very much.