Are we *actually* IDE yet?
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Untertitel |
| |
| Serientitel | ||
| Anzahl der Teile | 15 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/52190 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
2
15
00:00
ProgrammierumgebungStrömungsrichtungEntscheidungstheorieKontextbezogenes SystemWort <Informatik>Vorlesung/Konferenz
00:21
ServerFormale Sprachet-TestFormale SpracheCodeServerComputerspielTwitter <Softwareplattform>Wort <Informatik>Inhalt <Mathematik>FlächeninhaltFokalpunktSelbst organisierendes Systemt-TestGoogolSoftwarewartungReelle ZahlAvatar <Informatik>MultiplikationsoperatorComputeranimation
01:02
WechselsprungAuflösung <Mathematik>ParserVollständigkeitCLIFormale GrammatikAutomatische IndexierungDesintegration <Mathematik>TypsystemMenütechnikServerFormale SpracheGenerizitätOrakel <Informatik>Übersetzer <Informatik>RückkopplungEindringerkennungCompilerMereologieInhalt <Mathematik>BitCompilerFormale SpracheParserServerAuflösung <Mathematik>VollständigkeitSchlussregelSchnittmengeUmwandlungsenthalpieTypentheorieMathematische LogikIntegralCodeSoftware Development KitFront-End <Software>DifferenteMultiplikationsoperatorWechselsprungCASE <Informatik>KontrollstrukturPhysikalisches SystemPunktGüte der AnpassungÜberlagerung <Mathematik>PerspektiveFunktion <Mathematik>SelbstrepräsentationBildschirmmaskeAutomatische IndexierungToken-RingStreaming <Kommunikationstechnik>Formale GrammatikBimodulSystemaufrufCLIProgrammierungLeistung <Physik>ProgrammbibliothekQuaderMailing-ListeTexteditorQuellcodeProjektive EbenePlug inResultanteCachingGanze FunktionDatenstrukturRechter WinkelAnalysisZahlenbereichFunktionalDämon <Informatik>LimesmengePlastikkarteSyntaktische AnalyseAbstrakter SyntaxbaumEntscheidungstheorieEindringerkennungSoftwareentwicklerDebuggingKette <Mathematik>AbstraktionsebeneLipschitz-StetigkeitPartikelsystemComputeranimation
09:29
HypercubeArchitektur <Informatik>Ein-AusgabeGruppoidATMDateiformatWechselsprungHierarchische StrukturVererbungshierarchieAuflösung <Mathematik>Abstrakter SyntaxbaumVollständigkeitAbgeschlossene MengeTermSoftwareentwicklerPrototypingp-BlockServerProgrammierumgebungFormale SpracheElektronische PublikationResultanteMessage-PassingFahne <Mathematik>AnalysisMultiplikationsoperatorProjektive EbeneRegulärer AusdruckDifferenteEndliche ModelltheorieOrdnung <Mathematik>TexteditorPunktCASE <Informatik>CompilerMathematikSichtenkonzeptMultiplikationValiditätSkalarproduktAuflösung <Mathematik>Ein-AusgabeProdukt <Mathematik>DateiformatOrdnungsreduktionATMCodeVersionsverwaltungNichtlinearer OperatorÜbersetzer <Informatik>ENUMTypentheorieInstantiierungSchnittmengeVollständigkeitHierarchische StrukturLeistung <Physik>HypercubeWechselsprungDreiecksfreier GraphBitDatenbankProtokoll <Datenverarbeitungssystem>TopologieRelativitätstheorieInverseCursorURLFunktionalAbfrageDatensatzComputeranimation
16:48
Syntaktische AnalyseProgrammierumgebungSyntaxbaumCompilerAbstrakter SyntaxbaumPuls <Technik>WärmeausdehnungAbgeschlossene MengeObjektverfolgungGreen-FunktionHackerAlgorithmusÜbersetzer <Informatik>CachingWiederherstellung <Informatik>PhasenumwandlungPhysikalisches SystemWendepunktSampler <Musikinstrument>Ein-AusgabeFunktion <Mathematik>AbfrageTypentheorieBillard <Mathematik>GruppoidATMMakrobefehlProgrammbibliothekProjektive EbeneModallogikPaarvergleichProgrammierungSchnittmengeGebäude <Mathematik>Notebook-ComputerTeilbarkeitTypentheorieQuick-SortÜbersetzer <Informatik>Abstrakter SyntaxbaumParserDateiformatSyntaxbaumEin-AusgabeFunktionalAbfrageEndliche ModelltheorieTeilmengeStabilitätstheorie <Logik>DokumentenserverDifferenteArithmetische FolgeToken-RingFunktion <Mathematik>Billard <Mathematik>Physikalisches SystemSyntaktische AnalyseWärmeausdehnungResultanteCodeQuellcodeLokales MinimumKartesische KoordinatenMakrobefehlNichtlinearer OperatorProgrammierumgebungATMCompilerAutomatische IndexierungDatenbankRohdatenEllipseFront-End <Software>Lipschitz-StetigkeitStandardabweichungArithmetisches MittelAbstrakte SyntaxComputeranimation
21:51
CompilerProgrammbibliothekBridge <Kommunikationstechnik>CodecUnicodeClientDialektMakrobefehlGruppenoperationNatürliche ZahlInterface <Schaltung>ProgrammbibliothekCompilerProxy ServerCodierungCodeComputeranimation
23:29
ProgrammierumgebungServerFormale SpracheDateiverwaltungImplementierungVirtualisierungInstantiierungAuflösung <Mathematik>CodeComputeranimation
24:14
BitOffene MengeComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:05
So today I'd like to talk about the current Rust IDE story and more specifically Underline some of the historic context behind it and also talk about the tools themselves Which includes the design decisions behind them and their current capabilities
00:20
But first a couple words about me Most most of us spend most of the time on github and on twitter rather than in real life So chances are you probably recognize me more by my handle and by my avatar rather by my real name I'm the main thinner of the rust language server. I
00:41
Started my journey with rust in 2017 as a Google summer of code students so there I worked on the RLS itself under Mozilla organization and I'm the member of the official dev tools team and The dev tools themselves are my primary area of focus for me in Rust itself
01:03
So the content will be split into four parts beginning with early days, which I consider to be from 2014 to 2016 and Because I will cover a bit of dates I thought it'd be good to visualize them on a timeline such as this to put things into a little bit of perspective here
01:23
and so it'd be good to establish a good point of reference for all the dates that we'll cover and I believe a good point of reference would be rust 1.0. Which is like the first stable release which guarantees us that Once you write our code when we upgrade our rust toolchain
01:41
We're guaranteed that the code won't break which was often the case in the rust 0.x releases however the More than a year prior to that and Dating initial comment dating back to March 2014. The tool is called racer, which some of you may probably know and
02:05
so I did some good archaeology and pulled this snippet from one of the initial comments and As you can see the syntax remembers different different days So for example, we have you went which is now you sighs if I recall correctly and we have syntax for tilde T
02:21
Which is now a box of T respectively So how racer did is that this was it used the internal rusty parser so called lib syntax a parser roughly is a program that transforms your source code into some form of a representation that again that then can be
02:42
further analyzed and worked worked on and on top of that it actually bolted a very simple name resolution system and And Once you think about this We can use the name resolution to power one of the staple ID features such as jump the definition So in this example, we have a very simple code if we were to resolve a path a colon colon B
03:06
We need to scan the code see what a refers to so in this example We this would be a module a and then once we resolve that part of the path We need to we move on to the next one. So that's B but then we find the B and the nested scope that's introduced by module a and
03:26
Once we do that once we resolve fully the path and that's obviously oversimplifying but for the sake of this example Once you have the B, we know that it points to struct B. And so we can jump to that definition in our editor
03:40
It also included a completion engine So we have two types of completion very simplifying again one is scope completion So that's very similar to the name resolution But instead of actually stopping at one segment and resolving that we get a list of all the valid possible scopes that are further Introduced in that scope right but also there is a dot completion, which is a lot more interesting
04:06
but to actually correctly do that we need to have a fully functional type system because a structure can implement a number of traits each pulling their own methods and so to get an accurate list of methods that are
04:21
Callable for a given structure. We need to know which exactly traits are implemented for that structure So you can see how this can get a little bit a little bit more messy So racer did have a very heuristic all approach that but nothing very complete in the rust C sense And it was also designed as a CLI tool so it wasn't like a demon or language server as we do have now
04:46
most most of the time Couple like a year later and September 1 2015 That's the birthday of the IntelliJ plugin for rust and then the developers were faced with a decision
05:00
whether to reuse what they have so that's racer or to reuse or to use the Set of tools that are introduced by and developed by jet brains for their IDs to do their own support for the language and in the end the decision was to reimplement their own kind of compiler front-end using that set of tools and
05:23
A fun fact it was actually written in kotlin. So that's jet brains JVM based language, which wasn't even 1.0 at this time so how it actually Did how they actually supported the Generated the compiler front-end is they use the parser generator called grammar kit from jet brains and a quick refresher or
05:45
Introduction what parser generator is is that it's a piece of logic that accepts a grammar specification So there'll be a set of rules that define your language and on the output you typically get two programs Call lecture and parser. Well lecture does Simply
06:01
simplify Simplifying. Yeah simplifying is that it accepts transforms the stream of characters and Outputs a stream of kind of abstract tokens. So in this example, we have let token which is a keyword Then we have identifier token, which is a and so on and so forth What parser does is it accepts that stream of tokens and on the outputs it outputs?
06:25
Sorry it outputs Syntax tree some form of a representation that then tools and the compiler itself can use to perform further analysis And these can be syntax trees can be concrete or abstract, but I won't go into that right now
06:43
so having that They initially had a very Limited set of features visit but as time went on they added more and more So initially they started with a basic name resolution So that's very simple similar to what racer did this time went on the it obviously got improved and expanded upon
07:04
They also supported indexing. So for example when you take a final references feature for ID What you need to do to actually answer that request you need to walk the entire crate and see What's you know from the entire source code? What reference is that? You know a given definition
07:23
So it did this it does that It caches the result and that's basically roughly what indexing is They also introduced a cargo integration support so now IntelliJ plugin knows what a cargo d'automel is how your project is structured that you can have a workspace of many crates and
07:42
What's interesting is that initially they have a very? Horistical based type system But in 2017 they re-implemented it in kotlin to be like a real deal and maybe Not on par with what rust C has but something very very similar
08:00
at least aspiring to be complete But leaving that for now, let's move on to the official Rust ID story in a way. So this all started when raw RFC one three one seven dubbed ID support was The PR containing it was opened. So that was in October 2015
08:23
and It covered mainly two things. So one is What how an ID tool looks like what it does? How does it communicate with editors and so on and the other one? the second point is how we can actually extract the data from rusty itself that powers all the
08:44
Smartness behind the tool and after long debates a couple of couple of months later the RFC was merged and So the bike shed it has been resolved the name has been picked. So the name rust language server was picked
09:01
But actually the discussion about the data was around was revolving around to access whether we Lazily compute the data whether can we adapt rusty to have this incremental infrastructure and Then there was no resolution. So was what was agreed upon was that rusty
09:23
Should dump all the data all the smartest data and then we can improve their status quo incrementally from there So coming next to the RLS which is From between 2016 to 2018 and that's obviously not to say that development has been stopped It's just that it was most active and defining development going around there then
09:46
The prototype was initially created by Nick Cameron and Jonathan Turner and at the end of August 2016 and what's interesting is that Actually for a second it actually pulled Tokyo and hyper so the protocol was based around HTTP
10:02
Rather than what was eventually used the Microsoft's LSP Which is language server protocol which didn't exist at a time when the ID was opened and debated So a couple of days later. It was officially renamed and officially announced at Roskon 2016 and
10:22
It was designed to be the IDE tool to rule them all basically. So whichever request you wanted to Well to request from that tool it had to be able to answer that so in order to do that It actually pulled a couple. Sorry a couple of different tools such as racer for the auto completion rust format for the code formatting
10:44
clippy to integrate with the external linter and cargo to orchestrate builds and to detect project layouts To come back to the data and how we can extract that from the rusty itself an
11:01
Additional pass was implemented that was enabled via the dash Z save analysis flag And what it meant to do is it meant to save the analysis? pass results to a separate file in this case JSON but unfortunately the name kind of stuck and we call the data itself save analysis and
11:22
That's to show that RLS is actually decoupled from rusty itself so it does not rely on the compiler internals as much as clippy for example, so whenever we Break stuff into rusty or change the internals. The RLS thankfully does not break as often But since the compilation is always local from the point of view of a given crate and rust seed
11:46
Cannot does not know how to compile multiple crates at a single time At the same time in a single session. We need to lower these Multiple JSON blobs into a single coherent view that only then can be used to query by the RLS itself
12:04
So to sum it up the mode of operation is as follows whenever user changes input So for example, they modify a file They instruct RLS instructs through a seed to regenerate the JSON blob Which only then is lowered down to the database which only then can be used to query by
12:23
RLS and you can see it takes a bit of time to fully complete the cycle So to quickly explain the data format that powers the save analysis and the RLS It was with time it was Expanded upon which means the feature set of RLS also has expanded but initially it had only definitions and references
12:46
So for instance if rust see encounters like a struck definition or an enum definition or function it records all of that into the JSON and also when it walks to create and it sees references and Uses its name resolution pass to find references to an actual definitions. It also records that
13:05
So we can use that to power jump the definition We only need to query reference in a given location at a cursor same thing We have with final references But we need to traverse the inverse relationship Relation of the references too quickly for a given point of file find all the references or they all refer to single definition
13:26
But as he said as time went on We expanded the data and we included stuff like tree hierarchy so we lowered we recorded every ample block for a given definition and with this we could quickly answer queries such as Which trades does a given definition implement or roughly? What are the methods that were introduced in ample blocks?
13:49
introduced for a given definition Also, it saw some niche cases like D globbing So when this feature was designed the glob imports were kind of frowned upon So what we did is we recorded
14:03
which exact definitions have been pulled by rusty itself and Then if a user wants to replace that with actual imports they can just do that on the spot Interestingly enough at one point. It also had the electrical scope borrow data that was when borrow checker was based on the AST and that was to visualize all that all that scopes in the editor because
14:29
Borrow checker was deemed to be a For the newcomers, so we wanted to be able to visualize the data So they can more effectively reason about it
14:42
But in the end we knew that we will want to move off to the non-electrical lifetime models So in the end, we just abandoned the data altogether However, this brings me to completion and that's a good question how we can model completion and that That save analysis model and the answer is we can't really because we'd have to record
15:05
every possible completion for every expression that is in the crate which bloats up the save analysis format considerably and if we Well, we can see that if we have more data Then we need more time to emit that to a file and when we have more data in the file itself
15:23
We need more time to lower that into a single coherent view as I was mentioned previously so this unfortunately increases the latency that's perceived by the user and Also, it's very hard to model because whenever user imagine the user types like let a equals expression dots how RLS works is it instructs the compiler to
15:45
Redo the compilation on that crate, but that does not parse right? It's not a valid rust code So rusty just goes yeah I don't know what to do with this and it just omits it and skipped it skips it all together
16:00
Which brings me to RLS 1.0 release candidate which was announced in the August 2018 So if you remember that times you can remember that it was not well received people did agree that It was not ready for 1.0 with main pain points being the high latency and
16:22
Lacking completions at a time We kind of sidestepped that issue by just saying okay RLS version is the same as the rossed version So, you know, we're kind of officially 1.0 and pass it at but it's still their work There is work to be done on this end And so we work continuously to reduce latency and improve product support and in general improve RLS as we go
16:48
brings me to rust analyzer Which is a very interesting project that had seen its birth in 2018 it all started with RFC called lib syntax 2.0, which was open at the end of
17:00
2017 and What this RFC proposed is to introduce a parser that would output not AST but a concrete syntax tree, so that's Literally how the rust source code was parsed including comments and any other tokens And you can see how this can be useful for different tools. For example for a rust format or
17:26
Even ID that needs to answer like requests for example extract the given function outside of the scope including the comments and whatnot in The meantime the incremental infrastructure has matured and the rusty which was not originally
17:42
accounted for or considered in the original RFC in 2015 and It's The first stable release of incremental compilation has landed with stable rust 1.24 release which was in February 2018 and it turned out to the design turned out to be very good. And so it inspired another crate called salsa
18:06
which Yeah Whoo so it's the initial comments dated back to September 2018 and It basically models a functions a queries which are functions from type K to type V of two different types
18:23
One type is input so we can think of this as a function that pulls value outside from the environment and the another type is a pure function which can be thus Nymwise or cached because a pure function the result of a pure function only depends on the input solely on the input
18:42
we can see in practice how this can Work in the ID setting or the compiler. For example, if we type check a given definition or a body We don't need to do it again if something else changed and the source code that does not affect the definition for which the work has already been done and
19:03
So with this combined with the research work that that went into lips index 2 and salsa We actually had a pretty functional tool We have a tool that was able to parse rust source code and output Syntax tree that then was analyzed using the salsa database
19:22
One of the main goals for the rust analyzer was to explore how an ID ready compiler front-end might look like especially it Emphasized laziness heavily as I said, you don't need to with this lazy approach as salsa demonstrate We don't need to redo most of our work comparison in concert
19:42
Contrary to the RLS where when you modify something very slightly You still need to recompile everything and lower everything into the single coherent database One of the goals is to reuse what we can so we do not aim to reimplement the type checker instead we want to reuse chalk, which is
20:02
reimagined sort of type system for for rusty that's implemented outside of the actual compiler and Another goal is to be separate completely separate from rusty, which is both good and bad so rust analyzer fully compiles on stable and This greatly improves the contribute ability factor. So to give you an example on my fairly recent laptop
20:27
Building rusty from ground up with debug settings takes an hour and 20 minutes So you can see how that can scare off potential contributors But on the other hand, we don't have parity with rusty So we don't accept the same set of programs that rusty does and we don't get the same diagnostics
20:47
So to quickly recap the mode of operation comparison to the RLS is that whenever user changed it changes inputs rust analyzer only informs the salsa of that and then whenever user asks Something of rust analyzer only then it asks the salsa to do it
21:05
Thus doing the mean the minimal amount of work necessary So the park the progress is that rust analyzer mostly works, which is great. It actually handles the Rust compiler repository itself, which I consider to be a great milestone and recently macro expansion has improved by quite a lot
21:25
So we support a considerably considered considerably big subset of real-life rust code including rust standard library But on the other hand, we don't still we have diagnostics and we sidestep this by running cargo check on the side
21:44
and also Stuff that needs indexing does not work it. So for example final references feature now Around the beginning of 2019 a working group dubbed RLS 2.0 has been formed somewhat organically
22:02
the rust all hands Which was in February and the goal was to Be an experiment to see how rust C itself can be adapted to Be more of a ID ready in that sense and they based the work on the rust analyzer what we have already
22:22
One of the other goals was to library fire us see has to find in how we can split and modularize rust C into more Fine-grained and pure crates are in an interface and contrary to what we have right now Which is just a bunch of crates that just happened to work
22:43
And also somewhat of a secondary nature is how we can bridge those two tools As to not to be so confusing for the end user when it comes to the ID support So the first fruit of the work for the working group was the rusty underscore lecture crate
23:01
which was merged upstream in July 2019 this year and what it actually did is the PR was opened that pruned all of the lecture code that relied on the Internals of rust C and replaced it with fully stable Compiling codes which actually means that we can now share the same lecture code
23:24
Across the compiler itself and the rust analyzer, which is great Which Brings me to today. What do we do today? we obviously continue to improve the existing tools separately and Together so separately rust language server and the rust analyzer
23:43
But we do try to unify the ID efforts as in not to be so confusing for the end user For instance RLS and rust analyzer had their own virtual file system implementations or their own LSP server implementations, so we plan to cut down on that and sure hopefully more code rather than less and
24:04
Rust C itself sees a lot of cleanups internally So who knows maybe something like name resolution will be extracted into a separate crate in the near future This brings me back to my first opening question are we ID yet?
24:21
I'd say Not yet, but we're very very close so you'll have to just bear with us for a little bit longer. Thank you