IntelliJ Elixir - Elixir Plugin for JetBrains IDEs
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47256 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020349 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Plug-in (computing)InternetworkingBlock (periodic table)Computer animationXML
00:28
Integrated development environmentFormal languageWeightWeightAdditionClosed setSoftware developerIntegrated development environmentOpen sourcePlug-in (computing)DebuggerSource codeComputer animation
01:09
Computer fileCodeProtein foldingComplete metric spaceDisintegrationDebuggerTemplate (C++)File formatDivisorConfiguration spaceParameter (computer programming)Software development kitData structureCodeStack (abstract data type)CryptographyDifferent (Kate Ryan album)Latent heatData structureRevision controlWeb 2.0View (database)Symbol tableMessage passingMixed realityMathematical analysisErlang distributionElectronic program guideComputer animation
02:08
Erlang distributionFormal grammarCompilerParsingTranslation (relic)Normal (geometry)Compact spaceJava appletRevision controlPlug-in (computing)
02:45
AutomatonRegular expressionErlang distributionInfinityComputational physicsModal logicFormal languageSocial classInterpolationVirtual machineRecursive languageTuring testMathematicsTrailFormal grammarRegular expressionInfinityRecursionFormal language
03:08
InfinityStack (abstract data type)AutomatonRegulärer Ausdruck <Textverarbeitung>Computer fileCode2 (number)TrailForm (programming)Maxima and minimaLevel (video gaming)Complex analysisFormal grammarHierarchyDiagramPushdown automatonJava appletText editorMultiplication signWritingPartial derivativeRecursionInterpolationEscape characterComputer animation
04:08
Stack (abstract data type)AutomatonErlang distributionInfinityComputational physicsModal logicFormal languageSocial classInterpolationVirtual machineRecursive languageInterpolationControl flowLink (knot theory)
04:25
Formal grammarRegular expressionRecursionParsingDerivation (linguistics)Formal languageContext-free grammarSymbol tableTranslation (relic)ParsingRecursionDifferent (Kate Ryan album)Compilation albumText editorQuicksortNP-hard
05:25
Computer configurationInterpreter (computing)CodeBlock (periodic table)System callParameter (computer programming)Associative propertyRule of inferenceMacro (computer science)Context-free grammarComputer animation
06:16
CodePrice indexReverse engineeringInformationMenu (computing)Tablet computerInclined planeHill differential equationFunction (mathematics)Term (mathematics)Network socketInheritance (object-oriented programming)File viewerDebuggerComputer fileFile viewerStack (abstract data type)Different (Kate Ryan album)Macro (computer science)Error messageView (database)Formal languagePlug-in (computing)Source codeParameter (computer programming)System callInformationTracing (software)Line (geometry)FunktionalanalysisTable (information)Attribute grammarModule (mathematics)NumberDebuggerFlow separationMultitier architectureLocal ring1 (number)State of matterAlgebraic closureBytecodeCompilerMathematical optimizationRun time (program lifecycle phase)Subject indexingString (computer science)LengthTemplate (C++)MappingCodeErlang distributionAtomic numberSoftware testingDirectory serviceDot productOrder (biology)Arithmetic meanImplementationForm (programming)BitInheritance (object-oriented programming)Selectivity (electronic)Goodness of fitPower (physics)Matching (graph theory)File formatPattern matchingDisassemblerText editorMathematicsComputer animationSource codeXML
12:10
Module (mathematics)Meta elementConfiguration spaceDebuggerLevel (video gaming)Mobile appRevision controlIntegrated development environmentComputer animation
12:32
Game controllerSystem callFrame problemLine (geometry)CodeView (database)Frame problemTemplate (C++)Module (mathematics)Computer fileComputer animationSource code
12:53
Stack (abstract data type)Frame problemVolumenvisualisierungVariable (mathematics)Thermal expansionMeta elementCASE <Informatik>View (database)Variable (mathematics)DebuggerGraphical user interfaceField (computer science)Computer animation
13:24
Interpreter (computing)CodeLevel (video gaming)CodeLibrary (computing)Frame problemLimit (category theory)Sound effect
13:44
CodeInterpreter (computing)Modul <Datentyp>Library (computing)Function (mathematics)Macro (computer science)Module (mathematics)Pattern languageErlang distributionFunktionalanalysisSoftware bugLine (geometry)Library (computing)Telephone number mappingCodeDebuggerComputer animation
15:00
Erlang distributionMacro (computer science)CodeKerr-LösungKernel (computing)Function (mathematics)Form (programming)Erlang distributionForm (programming)Declarative programmingComputer animationSource codeXML
15:28
CompilerMathematical analysisFluid staticsParsingNetwork topologyInclusion mapCodeDeclarative programmingMereologyServer (computing)Asynchronous Transfer ModeQuicksortComputer fileCompilerCodeFormal languageComputer animation
16:10
CompilerMetaprogrammierungFunktionalanalysisTrailLine (geometry)Cartesian coordinate systemComputer animation
17:14
File systemMixed realityComputer fileBuildingParsingMetric systemSource codePoint (geometry)ImplementationOpen setSoftware bugPlug-in (computing)Software development kitServer (computing)Erlang distributionFunktionalanalysisFormal grammarDebuggerFormal languageStatisticsNumberExtension (kinesiology)Context awarenessSystem callData miningSoftware testingCASE <Informatik>Parameter (computer programming)Module (mathematics)Computer animation
21:31
Open sourcePoint cloudComputer animation
Transcript: English(auto-generated)
00:05
Hi, I'm Luke Himoff. You may know me as ChronicDeath on the internet. I'm full of autoimmune diseases. If anyone else is, there's a really good gluten-free bakery like three blocks away called Chambaland that everyone should try. Really good. I'm a member of the LumenCore team, an architectural engineer at Dockyard, and
00:24
most importantly for this talk, the creator of the IntelliJ Elixir plugin. IntelliJ Elixir is a plugin for JetBrain IDEs. IntelliJ, Community, or Ultimate Edition, of course, which supports any language, but also works for those coming from Objective-C background using AppCode, C and C++ using C-Line, SQL
00:41
using Datagrip, Go using Golang, PHP using PHPStorm, Python using PyCharm, .NET using Rider, Ruby using RubyMine, or JS using WebStorm. In addition to IntelliJ Community Edition, there is also a PyCharm Community Edition, so both those are completely free and open source. But the debugger and decompiler built into IntelliJ will actually decompile all the closed
01:05
source IntelliJ IDEs just fine, which is really helpful for plugin development. IntelliJ Elixir features support for viewing internals of Beam files, building whole projects, code folding, combing and uncombing code, code completion, Credo integration, which is a stack analysis tool for style checking, a graphical debugger, EX templates, which
01:26
are used for our web frameworks, fine usage, formatting, Go to declaration, related or symbol inspections with quick fixes, live templates, refactoring, run debug configurations, show parameters, SDKs, with full customization of both the Elixir
01:44
pass and the Erlang pass, so you can mix different versions of Erlang by selecting the code pass exactly. So if you ever see the release guide for Erlang where it's like, you can mix this version of the crypto package, but you need all these dependencies, you could try
02:00
those out and not have to have this specific version of Erlang, you can mix and match. There's also a structure view and a syntax highlight. One of the trickier features template was syntax highlight and parsing when the project started. It is a translation of Elixir's native lexer, which is written in just normal Erlang code, so it's just recursive Erlang to flex for IntelliJ Elixir, and flex
02:22
is a lexer generator that generates really compact Java code, and Elixir's native parser, which is written using yak, which is an Erlang version of yak, which is yet another compiler that was originally written in C, to grammar kit, which is a parser generator that some of the JetBrains people made for
02:43
making these sort of plugins. Since the Elixir lexer is written as Erlang functions, it can be a full Turing machine, but flex can't. Flex is able to generate a very efficient grammar by translating regular expressions, but not extended regular expressions that you're thinking of, like just the mathematical definition of regular expressions, to a finite automaton, which is what makes
03:01
it so fast. A finite automaton, though, lacks one important feature needed for Elixir, it cannot track infinite recursion of the language. Elixir requires this because you can nest any expression inside of interpolation, including more interpolation, and I have a feeling that an editor's grammar should always be forgiving, because most of the time you're writing bad code,
03:23
because you're writing partial code as you type, so I wanted to make sure that it wasn't like, well, I'm just going to find this as a regular expression, and six levels deep, and if you're doing more than that, the editor will just give up. Like, no, I wanted the grammar to work the way people could potentially write bad code, so I want infinite recursion. To do that, flex has an escape hatch, because you can write arbitrary Java code after you match on the normal regular expression.
03:45
So I was able to add a stack in Java, and that way keep track of when you're entering a new interpolation by just treating it as entering a new Elixir file over and over again, and keep track of when you exit the... And that actually elevates it to a more complex form called a pushdown automaton. This is actually the minimum level of complexity you need
04:02
in the hierarchy of Venn diagrams for this, to actually track nested braces or parentheses. Now that the lexes were equivalent, interpolation worked fine. When I got to the parser, I thought I had a lucky break. Both Yek and GrammarKit uses BNF, and from college I'm like, BNF, great, that's universal.
04:21
And so I copied it over, changed the early link to Java, I'm like, this is gonna be easy. Yeah, nah, no, no, BNF is not a universal. It is actually totally tied to the parser generator you're gonna target. The big difference here is that Yek is rightmost derivation, which means even though it reads like a human,
04:40
it actually starts matching symbols reading backwards, for left-to-right languages, of course, while GrammarKit is leftmost derivation. The reason for that is that rightmost derivation are really fast parsers, which are great for compilers that don't have to be forgiving, but leftmost derivation can be like, this is what I need to keep going, which is really good for an editor.
05:01
But the problem is, one, for rightmost derivation, you want one sort of recursion on one side, and for leftmost derivation, you want the other kind. If you just do it directly, you'll just have recursion that never picks up a new symbol. So they're completely incompatible, and they're kind of translatable as a human, but it is like an NP-hard problem
05:21
to have a general solution for translating a grammar, so it's all by hand. Good luck. The problem with translating from rightmost derivation to leftmost derivation is Elixir's do-end syntax and optional parentheses. This code has two possible interpretations. The do-end can be a block argument to either the left call or the right call. Elixir associates the do-end
05:41
with the outermost left call. It turns out this happens for free in LALR, but for LL parsers, a special rule needs to be defined in the BNF that disallows do-end blocks inside them. This unfortunately meant duplicating large chunks of the grammar, because the do-end block can appear in almost any operation, because Elixir is mostly defined with macros.
06:02
There is no module keyword. It is def module, and that is a macro, and it is all built up, so everything potentially can support do-end, which is only used for macros. When experimenting with optimizations, I like to use the beam chunk viewer. A beam file is composed of chunks.
06:22
There's an ATU8 that stores the UTF-8 atoms. It has replaced the older atom chunk, and so you can view the actual characters in the atom and see what is being treated as a constant. The string eight chunk shows something unexpected. It's not a table of strings, but a pool of one continuous string, and the reason why this works is the bytecode itself in the beam
06:40
has an offset into this table, and then the length of the string to copy out, but what you can do is that under certain conditions, the Erlang and Elixir compiler will be like, oh, this string is the suffix of another string, so I can just smush them together, and your pool table will actually be more compact than you would expect if you have common suffixes or prefixes, and you can actually look at this to be like, did that optimization happen?
07:03
While the string eight chunk only holds strings, the lit t table or literal table chunk holds all literals, so this way you can double check that your literals really are literals and they're not being computed at runtime, which this is slightly faster, which shows up as a true literal. The import chunk is the import table. It maps an index to a module and function
07:21
that was imported. Before I wrote the chunk viewer, I didn't actually realize remote calls are just declared in a separate chunk like this. I thought it was just a call to the runtime with the module functionality that I want you to call, so it's kind of interesting to see that modules know this just statically. Those imports are imported by the export table chunk.
07:40
The labels map to labels in the code chunk. Anonymous functions defined by the module are in the fun tier function table chunk, but somewhat confusingly, there is also the local function table chunk, or loc t chunk. However, this has more entries because if any named normal defined function is captured,
08:03
it also shows up here, so you can see all the weird names for the dashes are actual anonymous functions that have derived names, but all the ones from 12 onwards with build state were capturing that to use as an anonymous function closure that we can invoke with apply. The line number information for stack traces
08:20
stored in the line chunk is composed of two tables that are used to normalize the file name. You may have noticed that the file table can hold multiple names. This may seem unexpected, but Erlang actually supports an attribute to say that this chunk of code actually comes from a different file, and that's used all over the place in Elixir because of the macro, so that when a macro pulls in code from another place, we can actually blame it on the file it's coming from
08:41
for better stack traces than just assuming the line where the macro occurs being the place where the error happened. It also is used in EX for Phoenix views so that you end up blaming the EX file and not the view module that it goes in. The information from these chunks are used in the code chunk. By default, IntelliJ Elixir will inline
09:01
as many references and other chunks to view the code. So this language does not exist. I made it up, but it is as close to what Beam disassemble a tool in Erlang spits out, but more flexible and more Elixirish looking syntax. You can also turn off all the inlining to kind of see what the code looks like, but the code in the code chunk
09:22
is not Erlang's external term format. It is a more compact form that is especially adapted for code. So it only occurs in code, and nothing can really read it except Beam disassemble in this editor. I think people did a port to VS Code after I put this out, so there's a VS Code plugin that does it too now.
09:42
By looking at the code, we can find if changes in our source impact Beam files. So one of the big things to always look for is if your pattern matching is super efficient, you'll get a select eval instruction. So any arguments you have with your friends about like is this code more efficient, you can just look and see if it generates good byte code. You can also do cool things
10:00
like if you argue like do you match your, if you're trying to do a power match with like nested matches, does the order matter? You can look at the code and see is the code longer to change that order. Sometimes it is because matching first means it's the first variable which means it gets the next call very easily, and if you flip it the other way, it's not. The debug I or debug info chunk is optional,
10:22
but if it is there, the chunk viewer can show the internal implementation. If you click on the whole module, all the debug info's code form will be converted back to code. This looks a little bit messy with extra parentheses because that's just how macro to string works, and I'm not actually using macro to string, but I re-implemented in Kotlin to do this.
10:42
If you click on a function, all the clauses of function are converted back to code, and you can do the same thing with individual clauses. This also works for Erlang, but since this is the Elixir plugin, not the Erlang plugin, I translate the Erlang to the equivalent Elixir for you. So if you're like I don't really want to learn Erlang,
11:01
you don't have to. If you see like, so this is what I show you in the debug info, but this is what the actual code looks like. So it's close, but see, I'm automatically changing the variable names to be Elixir-ish. You know, we can't use capitals because then they turn into aliases,
11:21
so like they're slightly incompatible syntax, and this way you can understand if you're not up to speed with reading Erlang code yet. One of the original reasons I created the IntelliJ Elixir plugin was because I wanted a graphical debugger for Elixir that rivaled the one in RubyMine for Ruby. That debugger in RubyMine allowed me to level up my understanding of Rails quickly, and I felt that the same would be true
11:40
if I had a graphical debugger for Elixir. Let's say I want to understand how EX templates work in Phoenix. I can go into an EX template, place a breakpoint in the gutter marked here by the red dot. I can then right-click test directory to run the test in the debugger, and it crashes. Oh no. What happened is that the debugger can't debug NIF modules
12:02
because the way the staffing debugger in Erlang works is it is an interpreting debugger, and you can't interpret NIF since they're just C code. But we could ignore it, so we can tell it either at the individual run configuration level or in the entire IDE if the run configuration isn't set yet to be like, ignore this file, and it won't do it anymore.
12:21
This is also a bunch of stuff you see there as pre-ignored because those are usually like middleware stuff that if it was interpreted, your app would run so slow that it would be painful to debug. It ships with a bunch of defaults, which makes my version of the debugger slightly faster than VS Code's because it doesn't ship with that. Then when we rerun the debug configuration, we stop at the breakpoint line as indicated by the blue highlight.
12:43
Once stopped, we can look at the stack frames to figure out how Phoenix actually calls our templates. So templates are actually compiled into the view modules in Phoenix. So you see here it says user view. It's not in the template file. We can also click on the lower stack frames and see their variables, which in this case lets us understand how Phoenix layouts render their wrap views.
13:02
In the selected frame, we can see the variables and their values, and one of the benefits of a GUI debugger is that you can choose how you see information, so I'm able to expand the signs and find the values of the username field. I always found this super useful, going all the way back to when I used Eclipse. I loved Eclipse for C++ because of this GUI support for deeply nested structs, and I loved it so much
13:21
that I was able to do it here for Mighty Bugger. If you're in the top stack frame, you can also run arbitrary code. That's the limitation of the underlying DBG library in Erlang is that it stops you from running arbitrary code in lower frames because it might have a side effect and might mess with the stack so you can only do the top frame, so I unfortunately have to do it here. This looks really simple. You're just doing dots,
13:40
but that Elixir code is actually a bunch of map lookups, but it's not obvious. Here's more complex code. I'm using a with macro, with pattern matching, and do end, and it all still is evaluated because it's translated into Erlang code the same way that the IX Pry debugger does it. I just copy their code. The API is very unstable, though, and Jose Vellin, the creator of Elixir,
14:02
allows me to use it, but he says I'm allowed to use it because he knows I won't complain when he keeps breaking my stuff. I actually did fix a bug with 192 plus last night because of that. You can also step, so we're able to step here. The reason why it's at the top twice is that a lot of Elixir macros
14:21
blame the first line of a file, so we step and we go up twice because that's it printing that HTML. Even though we see it on line eight and 11, it actually assigns that HTML as being happening on line one of the module. To understand why a function exists, is it safe to change the lead? It can be helpful to do find usages. We can find modules.
14:41
We can find individual functions. This also works with library functions, so I can right-click find usages on enum into, and it'll find all the uses of that function. If you're just trying to understand something in the standard library, how do people actually use this function, or why is this function used here? You can go and find everywhere else someone uses it to understand code better.
15:00
There's also, the reason why this works is there's a decompiler that doesn't depend on debug I. It goes into just the exports and loc, and it can do the Elixir. It can also do macros. It can do special forms. It can also do Erlang in the same way,
15:22
and that's how all the go-to and declarations work. So that references, and go-to declaration doesn't use the compiler LSP. Part of this is, like I said when I had the question earlier today, is that the Elixir language server depends on the Elixir compiler, and there's no forgiving mode.
15:40
So if your code doesn't compile, a lot of the Elixir language server stuff just breaks. So instead, this is all built on using GrammarKit, which, again, is a JetBrains tool for writing these sort of parsers, and that, because it's an LL parser, is forgiving, so I'm able to, if you have broken code, I can still find usages in your file even though it's broken.
16:00
And it works if you don't compile, so the Elixir language server requires that mixcompile completed for it to work. And mine doesn't. But what this also means is that the compiler won't keep it around, it won't track all the imports. It'll track that import happened, but it won't allow you to really blame it
16:22
where it came from. It'll be like, this was imported from this module, but it won't tell you, this is the line the import happened on. So by me statically now analyzing the code, I'm able to show you the actual function called the top, but the second entry, I can be like, here's the line where you imported that function. So if you just have a bare function, you're like, where is this coming from?
16:42
You can click on that second entry and be like, oh, here's the import line that's bringing it in. That's where this is coming from. So I find that very useful. This also can walk through quoted blocks, so it can go pretty deep. It's okay with Phoenix, but Phoenix does this weird thing where if you do the use,
17:02
the application stuff with colon controller, it immediately doesn't apply, and that sometimes works. It's touchy because it's very weird metaprogramming from anywhere else. The source port is my GitHub at IntelliJ Elixir. It's in the marketplace in JetBrains,
17:21
so you can install it whenever. I have donations for everything I can find, Open Collective, PayPal, GitHub's own sponsor thingy. Any questions?
17:45
Am I aware of any implementations of the parser besides mine? Not that I'm aware of. Mine is a complete re-implantation, so I actually found bugs in the original native implementation because I have way more test cases because I didn't trust that I understood how to use grammar kit. It took me a year to get it right.
18:01
I kept finding bugs in the actual Elixir Erlang and YEC that they had to fix. I haven't seen anyone else be nuts enough to do it. Everyone else I know is just using the Elixir language server now, and that's just using whatever mix compile can do.
18:23
What kind of refactoring should I support? Because the JetBrains API for refactoring is related to references and find usage, once I implement find usage, I got rename support, so function renames work across files, not just in the current file. You can rename parameters and variables,
18:40
and that works too. I don't think module rename works, and that's mostly because I would have to also support asking you if you want to rename the file and never got around to it. I mean, Elixir doesn't require the files to match, but it annoys me when it doesn't because I'm a build engineer, and I'm like, the file system should match the files,
19:01
even though the language doesn't require it. Any other questions? Yeah. No, there's a, did I have to only do
19:21
the parsing grammar kit? No, the find usage is what's called an extension point in JetBrains. Open API is the thing they call it, plugins target, and I'd implement that, and there were some weird gotchas about whether a function should, if you go to definition on a, when you're already on the function call, should it go to itself?
19:41
And Elixir made it even harder because the def keyword is a macro, so if you go to definition on the def keyword, you technically should go to the def macro, but if you go on the name of the function, it should just go back to the function, and I have to go back to the function now, but I had to do some weird things to make find usage not think you're already at the definition, so it works. I'm not 100% sure I used the API correctly, though. It was mostly seen in the debugger
20:01
and figured out why it didn't work and making it miss those branches, but I've never had someone from JetBrains tell me I'm doing it wrong, so I don't know.
20:37
I don't see why I couldn't show this
20:41
as a language server to other stuff. I don't know how I would host it. I never really thought about it. I know people have made their plug-in. Right, I know people have packaged their plug-ins as jars to kind of do this for other reasons before, but I've never tried it, and for a number of users, it varies a lot.
21:02
If I don't do releases for a long time, I get between like five and 15,000 installs, but I don't know how many actual users, because I don't have any metrics. I just have the download statistics from the thing, so I don't know if that's that many actual users. All I have is unique downloads to go by,
21:20
so maybe 5,000-ish. I'm not sure, though.