Build for your microarchitecture: experiences with Spack and archspec
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47267 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
MikroarchitekturMikroarchitekturWeb-DesignerProjective planeComputer animation
00:35
SupercomputerSoftwareVirtual machineUsabilityInstallation artCombinatoricsCloningLimit (category theory)Formal languageInstallation artConfiguration spacePhysical systemResolvent formalismPolygon meshDistribution (mathematics)Different (Kate Ryan album)Perspective (visual)Software frameworkCombinational logicSoftwareVirtual machineFormal languageBuildingComputer animation
01:33
SoftwareCollaborationismSupercomputerPhysicsFood energyConvolutional codeLaw of large numbersVirtual machineProjective planeArmComputer animation
02:08
Revision controlInstallation artConfiguration spaceRecursionConstraint (mathematics)SpacetimeCombinatoricsFlagDefault (computer science)Revision controlCuboidDifferent (Kate Ryan album)CompilerFlagComputer animation
02:44
Menu (computing)Parameter (computer programming)Scripting languageRevision controlSource codeComputer animation
03:27
CombinatoricsSoftwareKolmogorov complexityConfiguration spaceUniqueness quantificationGraph (mathematics)Directory serviceBinary fileModul <Datentyp>ArmBuildingStack (abstract data type)Directory serviceDifferent (Kate Ryan album)Hash functionInformationConfiguration spaceCompilerCartesian coordinate systemRevision controlComputer animation
03:57
outputConstraint (mathematics)AbstractionConfiguration spaceRootInstallation artConstraint (mathematics)QuicksortData structureGraph (mathematics)Type theoryPartial derivativeComputer animation
04:32
Digital rights managementIntegrated development environmentComputer fileInclusion mapAbstractionRegular graphSet (mathematics)Endliche ModelltheorieSimilarity (geometry)Software developerComputer wormComputer fileQuicksortVirtual realityVirtual machineInstallation artVirtualizationEndliche ModelltheorieOpen sourceBinary fileIntegrated development environmentComputer animation
05:05
Open sourceBinary fileCodeBuildingEndliche ModelltheorieBridging (networking)Virtual machineMetadataComputer architectureRevision controlImage resolutionGeneric programmingVenn diagramBinary fileStandard deviationImage resolutionDigital rights managementMetadataVirtual machineConsistencyLimit (category theory)Goodness of fitInstallation artSoftware maintenanceLatent heatComputer animation
06:27
DiagramPortable communications deviceBinary fileMikroarchitekturIntelVenn diagramComputer architectureCodeSubsetPhysical systemExecution unitMikroarchitekturSet (mathematics)Electric generatorQuicksortVector spaceCircleComputer architectureBenchmarkResultantCoprocessorDifferent (Kate Ryan album)BitVenn diagramCanonical ensembleComputer animation
07:54
SupercomputerLatent heatComputer architectureBinary fileStandard deviationExploit (computer security)FlagCompilerStandard deviationVector spaceRight angleFLOPSPortable communications deviceDifferent (Kate Ryan album)Focus (optics)NumberVirtual machineComputing platformMikroarchitekturComputer architectureComputer animation
08:59
CoprocessorInformationComputer architectureInformationBefehlsprozessorCoprocessorQuicksortEndliche ModelltheorieSystem callComputer-assisted translationExtension (kinesiology)Computer architectureMereologyLibrary (computing)FamilyComputer animationSource code
09:50
FingerprintLibrary (computing)CompilerFlagKeyboard shortcutLatent heatMereologyCompilerInformationKeyboard shortcutLibrary (computing)FlagAdditionType theoryComputer fileComputer animation
10:37
Data modelMikroarchitekturBefehlsprozessorFlagBinary fileCompilerTerm (mathematics)File formatMathematical optimizationRootLibrary (computing)Endliche ModelltheorieQuery languageProof theoryFamily10 (number)Power (physics)Semantics (computer science)Pairwise comparisonCodeCompilerInformationRevision controlString (computer science)Exception handlingComputer architectureIntelMikroarchitekturConsistencyVenn diagramSet (mathematics)MultilaterationFlagQuicksortComputer architectureArmBinary fileSheaf (mathematics)Electronic mailing listComputer fileCASE <Informatik>AliasingObject (grammar)Mathematical optimizationGraph (mathematics)BefehlsprozessorInformationData dictionaryCompilerQuery languageRevision controlExtension (kinesiology)Vector spaceElectric generatorComputer animation
13:51
MikroarchitekturFingerprintMikroarchitekturComputer architectureBinary fileHash functionInternet service providerInformationComputer animation
14:55
Common Language InfrastructureInstallation artBenz planeArmVirtual machineCompilerCompilerError messageTable (information)FingerprintCodeQuicksortComputer clusterPhysical systemVector spaceError messageType theoryVirtual machineCompilerCompilerComputer animation
16:15
LogicQuery languageRegulärer Ausdruck <Textverarbeitung>Range (statistics)Latent heatComputer architectureQuery languageVirtual machineQuicksortOpen setLine (geometry)Parameter (computer programming)Computer fileLogicBuildingConfiguration spaceCompilerComputer animation
17:27
Distribution (mathematics)Binary fileRepository (publishing)Execution unitDifferent (Kate Ryan album)Binary fileInternetworkingWebsiteComputer architectureStack (abstract data type)Virtual machineBuildingLatent heat
18:27
Distribution (mathematics)Binary fileSoftwareRepository (publishing)Run time (program lifecycle phase)Computing platformComputer-generated imageryInterface (computing)Lemma (mathematics)Execution unit1 (number)Physical systemRevision controlDifferent (Kate Ryan album)QuicksortInternet service providerStandard deviationRun time (program lifecycle phase)FlagComputer-generated imageryInterface (computing)Portable communications deviceStack (abstract data type)Virtual machineRight angleComputer architectureComputer animation
19:41
FingerprintBinary fileFormal languageKeyboard shortcutInformationFormal languagePrototypeProjective planeArmArc (geometry)Message passingKeyboard shortcutImplementationParameter (computer programming)Numbering schemePhysical systemComputer architectureBuildingFraction (mathematics)InformationCoprocessorGene clusterInstallation artComputer-generated imageryFiber bundleDatabaseBenchmarkBinary filePairwise comparisonStandard deviationSubsetGoodness of fitCASE <Informatik>Different (Kate Ryan album)Revision controlPoint (geometry)Latent heatEndliche ModelltheorieOcean currentBefehlsprozessorOpen sourceMachine codeComputer fileDescriptive statisticsGroup actionWeb pageQuery languageRun time (program lifecycle phase)SupercomputerComputer animation
26:37
Point cloudFacebookOpen source
Transcript: English(auto-generated)
00:05
So, next up, we have Todd talking about microarchitecture with SPAC and archSPAC. So give it up for Todd. Thank you. All right. Thanks.
00:22
Okay, so I'm Todd Gamblin from Lawrence Livermore National Lab. I'm the lead developer on the SPAC project. I'm going to talk about some of our recent work to enable microarchitecture specific builds in SPAC. So what's SPAC? It's a tool that enables software distribution for HPC. It automates the build and installation of your software.
00:40
You can clone it from GitHub. You can say SPAC install package, say HDF5, and it'll go and build that for you along with all of its dependencies and get it working on your system. The packages are templated, so they're designed so that you can very easily get a build working on a new machine you've never seen before. The builds are designed to be very generic, and you can build lots of different configurations
01:01
of things. And it pulls a lot of low-level aspects of the build system up into the dependency resolver so that you can build all these different combinations. So the real goal here is we want to make it as easy to distribute HPC software as it is to use some of the other interpreted language ecosystems that people are used to. HPC is a pretty painful ecosystem from a build system perspective.
01:24
And you saw how important that was in the last talk. They weighted the ability to install the mesh framework very, very highly in their criteria. So I thought that was interesting. SPAC's pretty popular. It's used worldwide. We have a lot of contributors. We're up to like 500 different contributors now. And we're used on a lot of the world's fastest machines.
01:43
So the Summit machine at Oak Ridge uses SPAC to deploy 1,300 packages for their users. We use it at Livermore. And then we have a partnership with the Fugaku National Supercomputer Project in Japan to build for ARM 64 FX when that machine hits the floor. Then finally, we're the deployment tool for the Exascale project in the US.
02:04
So we're using it to basically tie the ecosystem together there. From a user perspective, SPAC is pretty simple. You can say install a package. You can say install it for a particular version. And then it gets fancier from there. You can say, I want to use a particular compiler in a particular version.
02:20
I want to build with a particular option. I want to inject flags into my build so I can build 60 different versions of the same package with different flags. I can set particular targets, which is what I'm going to talk about today. And then you can customize all of that for your dependencies. The syntax is recursive. And so you can get pretty specific if you need to. But otherwise, we have sensible defaults.
02:42
So you can install something easily out of the box. Like I said, the packages are templated. If you're familiar with Homebrew or other package managers, the SPAC package is looking off a lot like that. And so they're simple Python scripts. But you can say what versions are available. You can express options. So those options that you specify on the command line, they come from here.
03:01
This says I can build with or without MPI, and I can build with or without OpenMP. And then you can have optional dependencies and other fancy things in your package. And then these script pieces down here, so the CMake arguments and install method, they can query basically how was this package actually configured. You can say am I building for this compiler, am I building for this target.
03:23
And so it makes it very easy to write generic build scripts for your package. We let you install all of that pretty easily. You take the configuration for the whole DAG, so imagine this thing with all of the information filled in, all the build options, compiler information, and other stuff, targets.
03:41
Hash that, and we generate a unique directory for every package that we install. So basically everything can coexist. You can have lots of different versions of things. You can have two completely different stacks for different applications installed separately, and it's our path together so things know where to find their dependencies. And then the way that this works internally is you take a spec like this that the user
04:04
types on the command line. We produce sort of an abstract graph of what that is with partial constraints on it, and we do a solve to figure out what a consistent DAG would look like with everything filled in. And this is what gets passed to those package templates, and then the packages basically,
04:21
they query this structure to say how should I build this thing. And when we install, we install with full provenance, so you know everything about how the thing was installed. We also support virtual environments with sort of a manifest lock file model. You can use all of those same specs here in a YAML file. You can say these are the three things I want to install in an environment with a
04:42
little extra configuration, and we'll concretize the whole thing, spit out a lock file that you can use to reproduce it later, and you can use this sort of like a Python virtual environment, but for anything, not just for Python stuff. So, SPAC builds all this stuff from source, and you know that's pretty convenient.
05:01
It's useful for HPC machines where people are used to installing from source, but source installs are slow, and most people, if given a choice, would prefer to use a binary package manager. So, Conda's popular for scientific stuff, and you know, Homebrew on the Mac is pretty much a binary package manager these days. People like the speed of binary installations.
05:21
But, binary installs are typically much slower than what you get from building from source, which is why HPC doesn't typically use them. You want to optimize for your particular machine, otherwise you will probably not get even close to the best performance. And most of the distros don't distribute optimized binaries. There's limited support for that. Is there a question?
05:40
Yeah. Also, often the binaries are out of date, and it depends on the maintainer. Yeah, that's true. So, yeah, you can get bad binaries. So, the Python ecosystem is rife with horrible, horrible binaries that people have built on their own and put on PyPI. And so, you know, we'd like to have the best of both worlds. We'd like to make it so that you can optimize your build for specific machine models, but
06:03
we'd also like to have binary packages. And SPAC can already make binary packages, but there's some trickiness to distributing these and just a lack of standards. And so, that's what we're going to talk about today, what's needed. So, we need metadata for this. That's pretty much the gist of this talk. And then, there's also a good dependency resolution that you need to actually make the
06:22
thing consistent. I talked about that yesterday in another one of the dev rooms. So, all right, who has seen this Venn diagram before? Yeah, a few people in this room. So, what's a microarchitecture? Does everyone know what a microarchitecture is, if I say that word, do people know what I'm talking about?
06:41
Okay, so, your Intel processor, it's x86-64, that's the base instruction set. But with every generation, Intel does add new instructions to your processor. And so, with Skylake, they started offering AVX-512, which is, it's a vector instruction set, it's wide. And it looks like this on generations after Skylake.
07:00
So, essentially, what this is showing is compatibility between different Intel chips. And if your architecture is, if you're in one of these inner circles in the Venn diagram, you're compatible with all the others that contain you. But if you're not, then you have some instructions that are not included. And it's gotten really nasty because Intel doesn't really design instruction sets, they
07:20
design new instructions for benchmarks to make them look fast with successive processor generations. And this is the result. So, you can see here, this is Skylake, it sort of diverged into Cannon Lake and Cascade Lake, and then Ice Lake contained that, and Tiger Lake is upcoming, and then KNL is over here with its own set of AVX-512 instructions that are different from all the others.
07:40
And there's really not a set of names for these things that people can use. There are names like Skylake and Cannon Lake, but there's no tool that you can use to get that name easily. And so, I'll talk a little bit about that. Like I said, you care because optimizing for your microarchitecture is critical in HPC. I mean, it can be a 10x performance difference between not using the SIMD instructions on
08:04
your chip and using them, because that's really, if you look at the numbers for how many flops an Intel chip will give you, it's coming mostly from the vector instructions, and so we care about that. And then, you know, targeting generic microarchitectures, again, that won't use the vector instructions because they're not portable, and most of the distros focus on portability.
08:25
Most people just build from source and specify march equals native, but the problem with this is that if you say that when you build, you're going to get a binary that's built just for this platform that you're on right now, and you don't know where else you can use that. So, there's nothing about march equals native that says that it even builds something that
08:41
is, you know, is portable anywhere else besides the machine that you built. So, there's no standards for detecting what you're on, there's no standard for labeling binaries, there's no standard for programmatically finding what compiler flags to even use to optimize for your specific architecture or to understand which things are compatible. All right. So, has anyone ever tried to figure out what kind of processor they have?
09:03
Has anyone ever done this? Yes. So, you can use proc CPU info on Linux and sys control on Macs, and this is what you get. So, if you say cat proc CPU info, you get all of this, and this is the part that sort of tells you what CPU you're on. It says genuine Intel CPU family 6 model 79, and it's a Xeon 2695.
09:24
Who can tell me what that is? All right. So, what, right? And the other information that you get out of here is this stuff. These are the features that your chip supports. These are all the instruction set extensions that are supported by this chip. And so, you know, that doesn't make much sense either.
09:41
Humans call this architecture Rodwell. And so, you know, that's great, but it's unclear how you would get from there to this name. So, that's why we released this new library. This is a standalone library. It was originally part of SPAC. It's still part of SPAC, but we want to make it more available to other tools so that
10:00
hopefully these names can become more widespread. It uses fine-grained but human-readable labels like Broadwell and Haswell instead of, I mean, these are also in there, but you have the specific names in addition to the generic ones like x86-64. You can query things. You can say things like, does this chip support AVX-512, to ask it whether it supports a particular
10:20
feature. And you can query things like what compiler flag should I use to optimize for this chip. It is a Python package, but we would really like people to make other types of bindings because we'd like this to become widespread. And that should be easy because most of the relevant information is in a JSON file. So what's it look like? We model a microarchitecture like this where basically it's a class.
10:45
It has a name. It has parents. That's that DAG that's defined by the Venn diagram that I showed earlier. So we know that, say, Ice Lake comes from both Cannon Lake and Cascade Lake, that it's compatible with both of those. You know the vendor features, the compilers, and then what CPU generation it was.
11:03
So all that information is encoded. You can get at that through a Python API. The JSON file just stores all of this. And so this is thunderx2. It's from ARM architecture 64, so it's from the generic architecture. This is the vendor. We keep that around just in case you want to distinguish.
11:21
And then the feature list is here. There's a set of compiler sections that tell you that, for example, for GCC versions 4.8 to 4.8.9, you use these flags. But for version 4.9 to 5.9, you use these flags if you want to optimize for thunderx2. And there's some other things in here.
11:40
So for example, even the information about features in PROC CPU info is horrible in that certain features disappear even though they're still supported by the architecture. So for example, Intel has an SSE3 vector extension. And that was subsumed by SSSE3, which was a later feature set that includes SSE3, but
12:00
you would have to know that to query things consistently. You can't just say, does this chip support SSE3? You have to say, does it support SSE3 or SSSE3? And you would have to remember that. But not with this. We have aliases for these features, so you can actually do a simple query. It's pretty easy on the Python side to query things. You can just import archspec.CPU as CPU, and then you can say, here's some chips.
12:22
I'm going to get nahalem broadwell thunderx2 out of this sort of dictionary that the API exposes. And if you say nahalem.features, you get a list of the features. If you want to ask if a feature is in the architecture, you can say avx in thunderx2 and it says false, and neon in thunderx2 and it says true. So this is very easy to query.
12:41
And then you can actually compare the architecture. So there's a partial ordering defined by that sort of ancestry graph. And you can say nahalem is less than broadwell, and it will say yes. And you can say nahalem greater than broadwell, and it will say false. And so what this tells you is that you can run nahalem binaries on broadwell but
13:01
not the other way around because nahalem is older and broadwell includes all the instructions that nahalem supported. And if you want to detect your host, you've got that down here. You can just say CPU.host and we give you a microarchitecture object with all this information. It's easy to query compiler options. So if you want to say, you know, what are the optimization flags for
13:22
broadwell, you just say broadwell.optimizationflags. You give it a compiler name and a version, and it goes and figures out which one it should, which flags it should give you based on the version ranges that I showed earlier. And it'll say, you know, that's not supported. If you want to try to build for Ice Lake with GCC 4.8, well, too bad. Ice Lake didn't exist when GCC 4.8 came out, so it doesn't support it,
13:43
and we just tell you that. So that's the API. And so you can see, you know, what this enables in build tools by looking at SPAC because this is where it came from. So SPAC knows your microarchitecture. If you say SPAC arch, it spits out something like this. It says you're on Linux with Ubuntu 18.04 and you're on a broadwell chip or
14:02
you're on a Mojave Mac with Skylake. And all of the builds in SPAC are now built with full provenance on the target. And so what that means is that if I say install zlib here, we no longer run with march equals native. So for every binary that we produce, we have a specific target that we give it,
14:23
so we know exactly where that binary can be used. And so our goal is to eventually be able to distribute the binaries easily. And to do that, you really have to know where it can be used. So this is what happens internally. It uses the compiler flag options to say, oh, okay, when I compile for this machine,
14:40
I'm going to say march equals broadwell, mtune equals broadwell, and then all the other options that the build system provided. And that's all stored in the spec.yaml, and it's included. All this microarchitecture information is included in the hash that we give the package. You can list all the known architectures. So you can say spac arch dash dash known targets, and you get this kind of nice table of all the different supported architectures.
15:02
I think there's 40-something, almost 50. And you can say on the command line that you want a specific target. So you can say spac install this package for ice lake or for sky lake or for whatever. As you'd expect, it knows which compilers can optimize for your machine. So if you are on, say, a broadwell machine and you try to build with GCC 4.8 like this,
15:25
we'll give you a warning that says, hey, you know, build compiler, and so you may not get as good a performance as you expect from this. We think this is going to create a sort of new type of OCD for building, where people are, you know, they're like, oh, no, my compiler is too old,
15:43
even for very simple packages. But, you know, at least you know now. This one, so you'll get an error if you try to compile for something the compiler doesn't support. So I think, you know, a lot of people will compile with the system compiler on a relatively new machine and think that they're getting an optimized binary,
16:01
and they're not because it can't even generate code for all the vector instructions on the machine. And so we will error out if you actually say, you know, I want broadwell or I want sky lake with an older compiler like this. So the other thing that this has done that was quite useful is it really simplified the logic
16:21
in some of our build recipes. So like I said, spac package files are designed to be very generic, and we don't want, you know, we want to be able to take a package file that we use one place and use it another. And previously, you know, there was a lot of gymnastics done in the files to figure out, you know, what kind of machine am I on, which of these options
16:40
on the configure line should I enable for this machine. And with these new query capabilities and the new target stuff, you just ask the question. And so for packages, you know, like open BLAS or FFTW, you can just say, hey, if I'm greater than broadwell, then use the broadwell option. If I have avx-512, enable avx-512. Most of the packages expose these things as configure arguments and sort of put the burden
17:03
on the user to turn all the things on. Some of them try to do some fancy detection, but this also enables us to build deterministic builds. And so because the package recipes themselves are querying things and enabling features based on what they're building for, we know that, you know, the whatever the compiler outputs is going to be what we labeled the binary
17:23
as and where we can use it. So our ultimate goal, as you might imagine, is that we would really like to be able to distribute optimized binary. So we'd like to make the HPC experience as easy as other things. And so, you know, in the world we're working towards, we would have build farms going in all the different laboratories for specific machines
17:42
because we have a lot of different architectures at Livermore and other sites. We'd have build pipelines going there and, you know, once we're done building things locally, we could actually upload these binaries to the internet and you will know where you can use them. So Stack will say, oh, hey, there's a binary mirror. It has Haswell binaries. I can use those on my Skylake.
18:00
And so you could potentially say build a DAG that includes as many binaries as you can reuse or, and then build the things that you don't have. The other thing you could do is say, no, I really want the whole stack built for Skylake, and then it would say, okay, I'll use the Skylake binaries, but I'll rebuild all the other stuff for which there's no Skylake binary available. So this enables us to do more powerful things.
18:22
And the idea is that, you know, users could come, they could access the mirror, and they could get binary packages when they need them with Stack. And we would have a system that's as easy as some of the other ones out there. So with the library, with ArcSpec, any tool can leverage these features. And so this isn't just for Stack. We're putting this out there for everyone.
18:40
And so even, you know, some examples of tools that might use these features are container runtimes. So in HPC, we have this sort of conflict between performance and portability. If you want to run a container in HPC, you likely don't know what architecture the container was built for. There's no standard for labeling these things there either.
19:02
And so we're hoping you could use these names to label containers, and then eventually, this kind of support would get into container runtimes. So if you fetch a container for your machine, you get something optimized for your machine. Orchestrators could do the same thing if they know that they have a lot of different nodes in sort of a heterogeneous cluster, like a Kubernetes cluster or something like that.
19:20
They could pick the right container images for the right nodes, and you could get better performance that way. And then build systems could provide intuitive interfaces for name targets. So right now, we have that in Stack. We inject the right flags. But CMake or some other system could take this and use it to make it easier to specify what particular architecture you want to run for.
19:41
So we'd really like people to contribute. We would love to see these names become standard. Kenneth from EasyBuild has already agreed to contribute, although he hasn't done anything yet, so we're waiting on that piece. That's true, he did do the logo, that's right. So yeah, I recreated this from a Kenneth prototype.
20:00
So okay, I can't say he didn't do anything. But Kenneth is in the ArtSpec group on GitHub. He has joined the group. So yeah, I guess I really can't say that you didn't do anything. We like vendor contributions. One of the things we would ultimately like to do is have Intel and ARM and other folks contributing descriptions of new architectures to the large JSON file that we have
20:21
so that this stuff stays current. And that ultimately benefits them because they want to make their architecture look good. If it's easy to optimize for a chip, then people should start getting better benchmarks for those chips. And then finally, we'd really like more language bindings. We're planning to make Go bindings because we would like to get this into the container ecosystem. Things like Docker are written in Go.
20:42
And others are welcome. So please go to GitHub. It's github.com slash, it should say ArtSpec there. It's github.com slash ArtSpec. And then you can already leverage this stuff in spec if you want to use it there. So thanks.
21:03
Any questions for Todd? Yeah. Thank you, that's fantastic. I have a query related to standards work because I've heard things about container native application bundles.
21:23
Shouldn't this fit into that? Reach out into the container ecosystem. Yeah, I think it would just be a labeling scheme. Like you can already put labels on Docker containers but there's currently not a standard for this kind of stuff. And so what you want is for the, that's why we want a Go reference implementation. So the runtimes could start saying,
21:41
oh, I see that that version of the image has these labels and I can prefer certain ones over others. So do you hear any interest from them? This is so new that it hasn't picked up for. This is brand new. So the read the docs page went up, I think, an hour ago. So we have not advertised this widely yet
22:01
but please go out and tell people about it. This is just so super useful. I work for a major oil company on ARM processors and back to your point about binary installations. If you go into Debian or Ubuntu and you do an LSCPU,
22:20
it simply won't identify the ARM processor. You have to backport the bleeding edge LSCPU installation. It just doesn't recognize, because there's so many different ARM variants, as you know and this would have been extraordinarily useful. Well, we have to stay current too. So we will have the same problem that LSCPU has if you don't keep our spec current, right?
22:42
But that's one of the problems with, if you stick with your distro packages, as you say HPC people don't, if you stick with your distro packages, you end up with some really horribly outdated MPICH, horribly outdated QG systems and language versions and we really should do better and that's what this will look like.
23:00
Thanks. There's a question back there. Hi, Doug. So in the case of a new architecture appearing,
23:21
will Architech try to match the closer architecture that Architech knows about? I mean, that's what the Python bindings will do right now. So it's matching based on the features from proc CPU info. So it basically traverses that DAG and says, what's the maximal subset
23:41
that I can find of the features? And so if Architech doesn't know about your platform, it will probably identify it as the one before. What we also hope is that vendors will actually contribute to this, right? Yes. Intel has a new CPU. We hope that they send us a small pull request to tell us about the new CPU. Yes. Same for AMD, same for ARM, same for...
24:10
Thanks. On that previous comment, Intel have a database online called Arc that has so many parameters for all their chips
24:21
and it has an API as well. So this is something you could automate. Yeah. I believe other manufacturers have too. I'm just less familiar with them. Thanks. Pass. That's it. That's just an Intel mark picking name. They won't need to forget that.
24:41
You can look up things like Haswell and Broadwell, yeah. We could use that as a data source for this too. I'm pretty sure we looked at something online from Intel and I don't know that it was as clean as this.
25:03
And also, Intel has a lot of information about things like model specific features and so you can get information about the MSRs on different chips. But that is way too fine-grained for... This is designed for compatibility. We're really targeting the binary ecosystem and so we want a name that says this binary will run on this architecture
25:22
given the instructions that are in it. And I think that's lacking. But we'll check it out. If we can use that to back this, that would be great. Thank you for the talk. It's a really cool project. I have a question though. If you have a cluster of a grossly heterogeneous architecture, would you be able to come up with
25:42
a essentially minimum subset that spans all the available architectures that would be optimized for... Yeah, you can just search the DAG and find the one that's less than both of the architecture or all of the architectures that you support, right? So that's all you have to do because you have these comparators and you can reason about it. In fact, that's what we tell people to do
26:03
who have heterogeneous clusters with SPAC because it is specific, right? It'll say, oh, we're building for Skylake and the front-end nodes are Skylake but the back-end nodes are Sandy Bridge and so bad things happen. And so you can configure it and say, always build for Sandy Bridge if that's your lowest common denominator. What we don't have right now is a good way to label fat binaries
26:22
and so I'd like to look into that and figure out how we can label things. This is two architectures, right? Okay, we are out of time. Let's thank Todd again. Thanks.