LOFAR: FOSS HPC across 2000 kilometers
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62017 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Open sourceComputer animation
00:36
SupercomputerCodeFrequencyLogic synthesisSoftwareSign (mathematics)System callLevel (video gaming)Connectivity (graph theory)Open sourceMultiplication sign1 (number)Computer animation
02:12
Diameter
02:38
Type theoryWaveDifferent (Kate Ryan album)Square number
03:08
Parabolische DifferentialgleichungReflection (mathematics)AnalogyPhysicalismDistanceArtificial neural networkLimit (category theory)Line (geometry)Data structureMedical imagingWaveMetreComputer animation
04:17
Data modelMathematical analysisFood energySynchronizationThermal radiationAstrophysicsWaveParticle systemType theorySpacetimeFrequencyObject (grammar)Computer animationDiagram
04:57
Universe (mathematics)Inheritance (object-oriented programming)World Wide Web ConsortiumWavePulse (signal processing)Pole (complex analysis)
05:17
Computer clusterSupercomputerCross-correlation2 (number)Computer animationEngineering drawing
06:07
Vertex (graph theory)Network topologyTorusBit ratePower (physics)Graphics processing unitServer (computing)Network socketFloating pointMultiplication signExtension (kinesiology)BefehlsprozessorSoftware developerSoftwareGraphics processing unitComputer animation
07:16
Sample (statistics)AverageData storage deviceCorrelation and dependencePhase transitionCross-correlationWorkstation <Musikinstrument>Process (computing)Musical ensembleBit rateoutputBuffer solutionStokes' theoremField (computer science)Shift operatorBefehlsprozessorGraphics processing unitTerm (mathematics)Kernel (computing)Read-only memoryOpen sourceCASE <Informatik>Uniqueness quantificationKeyboard shortcutComputer-generated imageryMathematicsLinear mapNonlinear systemVector potentialBlock (periodic table)Field programmable gate arrayFirmwareOffice suiteZoom lensData modelConvolutionSource codePredictionResidual (numerical analysis)Medical imagingDeconvolutionInversion (music)Function (mathematics)Flow separationDomain nameMedical imagingDifferent (Kate Ryan album)Translation (relic)Uniqueness quantificationKeyboard shortcutBitTransformation (genetics)Source codeNoise (electronics)MereologyWorkstation <Musikinstrument>outputSemiconductor memoryDeconvolutionProduct (business)FrequencySlide ruleProcess (computing)Link (knot theory)Dependent and independent variablesMiniDiscBuffer solutionLine (geometry)Cross-correlationSampling (statistics)IterationFinitismusMultiplication signComponent-based software engineeringPoint at infinityDataflowServer (computing)Error messageAnalogyFast Fourier transformWavePhysical systemOpen sourceIdeal (ethics)Point (geometry)BefehlsprozessorExterior algebraDigital signal processingData managementAreaRevision controlVariety (linguistics)Universe (mathematics)Closed setFirmwareOffice suiteLevel (video gaming)Interface (computing)PlanningBlock (periodic table)Game controllerComputer hardwareComputer animation
17:07
SpacetimeMedical imagingOverlay-NetzDomain name
17:50
Asynchronous Transfer ModePhysical systemComputer hardwareDistribution (mathematics)Linear multistep methodMedical imagingHeat transferLink (knot theory)Instance (computer science)Presentation of a groupSingle-precision floating-point formatUniform resource locatorMathematicsPoint (geometry)Web pageProduct (business)State observerLevel (video gaming)SoftwareWorkstation <Musikinstrument>Communications protocolMereologyFlow separationMedical imagingSoftware developerAstrophysicsMultiplication signMetreSquare numberCollaborationismMusical ensembleComputer animation
21:27
Open setOpen sourceStokes' theoremBefehlsprozessorAverageKernel (computing)Data modelThermal radiationCodeFrequencyLogic synthesisWeb pageMultiplication signFrequencyScaling (geometry)Field (computer science)Focus (optics)Point (geometry)MultiplicationWorkstation <Musikinstrument>Position operatorSampling (statistics)Band matrixRaw image formatMultilaterationBuffer solutionProcess (computing)Product (business)Physical systemResultantCollaborationismHookingSoftwareMereologyBitComputer animationProgram flowchartEngineering drawing
25:37
Program flowchart
Transcript: English(auto-generated)
00:09
Okay, we're ready for the next talk by Courtney who's going to talk about open source radio astronomy. So hi everyone.
00:21
I work for OSTRON, which is the Dutch Institute for Radio Astronomy. We're a government institute, so we're publicly funded. Apart from being publicly funded, we have a lot of research grants, and that's basically what pays my salary. And I'm going to talk about our biggest instrument that we have, which is called LOFAR.
00:42
And we actually utilize a lot of open source software there, almost exclusively, there's some caveats. My name is Courtney Looker by the way, I'm also into amateur radio, my call sign is Papa Delta 3 Sierra Uniform. This is going to be quite a relaxed talk, we're going to give a high level overview of all the different components that are there.
01:01
There's quite a lot of them actually, so it's not possible to go into detail with the time we have. This is my first FOSTAM talk ever, and also my first talk ever in a lecture hall, so that's quite interesting. Now some background, I mentioned that we're a government institute, we firmly believe public money means public code, and we stand by that in almost everything we do already.
01:24
We have an open source committee that also ensures that we do that. And we have basically two very big telescopes, one's called LOFAR, which stands for Low Synthesis Telescope, also called WSRT. There's some sister institutes that we work closely with, or are related to us.
01:45
One is called CAMRAS, which maintains a telescope that we've stopped using, and the others are JIVE and EVN. I'm not going to talk too much about those today. What I want to tell you is that there is this principle that our radio telescopes work on that is called very long baseline interferometry, and this enables us to do radio
02:05
astronomy in a way that wasn't possible with traditional radio telescopes. This is the whole map of LOFAR. There's 54 stations in total, roughly 25 of those are located in the Netherlands. I say it's around 2,000 kilometres in diameter, but that's no longer true because the
02:24
one in Rosen is new, and we're now about 2,500 kilometres in diameter. This diameter is also called the baseline, which is where the very long baseline interferometry comes from. If you then zoom into a station, you see all these tiles, and you see these little
02:42
squares. Those are the different types of antenna, and that is what makes this type of radio astronomy so interesting, is that we don't have a single antenna to catch radio waves. We have lots of them, about 20,000 in total, actually, which is quite substantial. This centre is called the SuperTERP, and it's located in Exlo, the Netherlands.
03:06
So how can we actually combine this data? I told you that traditional radio astronomy relies on a parabolic dish or a single antenna, and we try to scale those up, make them bigger and bigger, but of course, physics are at the limits at some point.
03:20
You can't make a structure from steel that's like 500 metres in diameter, so what we do instead is we combine smaller antennas to act as if they are a parabolic antenna. And the trick about a parabolic antenna is that all radio waves, no matter where they are incoming from, they all have an equal distance to the receiver, so we need
03:43
to emulate that with our antennas, and we do that in two ways. That is an artificial delay, an analogue artificial delay, by just making the line that it needs to travel across on the PCB or the coax cable longer, but we can also do it digitally after the data is being sampled.
04:02
And then we can aim into the sky and create a very narrow beam that observes a very small portion of the sky, and that allows us to zoom really deep into space and make very detailed images. But what is this radio waves actually? What are those? What are we observing?
04:20
Well there's two types of radio waves that are being emitted by objects in space and the galaxy. We're only going to describe one phenomena today that's called synchrotron radiation. Basically if you have an ion, a charged particle, you accelerate that, then it starts creating radio wave emissions. The frequency and the intensity at the frequency that is actually very dependent, that's all
04:44
details that are not very interesting for this talk, but one of these entities that emit these types of charged particles are sometimes black holes, and we'll see an example of that at the end. So I mentioned black holes, there's other types of radio astronomy that are very interesting,
05:01
we can also model our own ionosphere, or enlightening. Pulse waves are pretty interesting, these are stars that are rotating at a periodic interval, and they have very strong radio waves coming from the poles of those stars. So what does LOVER actually look like? Very small antennas I told you, we can see on the left that there's like wires attached
05:24
to those poles, and those are actually dipole antennas, and if you configure them like this where they are like a V-shape, they are called inverted Vs. These are the low-band antennas, and on the right side you see the high-band antennas, they're like a clover shape, like a tie shape.
05:42
Then we combine all these antennas, 69 low-band antennas and 69 high-band antennas in a station, and we send the data at around 3 gigabits per second to our HPC clusters. There's a two-phase cluster here, the first is GPU processing, where we do correlation and beamforming, and the second is central processing, which is more like CPU-based.
06:06
In the early days, our computing cluster looked something like this, we had IBM Blue Genes machines, which were based on PowerPC, and they had a 3D Taurus interconnect, which is actually a quite interesting interconnect.
06:22
This was problematic because utilizing the floating-point vector extensions required manually rewriting assembly, which wasn't that nice and was pretty hard to find developers who were willing or capable to do that. So we moved to commodity hardware, GPUs, two CPUs per socket, two GPUs, two sockets per
06:45
server, two GPUs per server, lots of networking. That is really what you see here, we had 32 gigabytes of 10-gigabit Ethernet, and then in 2018, when we upgraded, we had 24 times or 23 times of 100 gigabits in FiniBand,
07:06
but you also see that there's a lot of 10-gigabit Ethernet per device, and I'm going to go into that, why that is, in a minute. If you look at the data flow, or more like a software site, then you see that the antennas
07:22
have ADCs, so these convert the analog waves that are incoming to digital signals, and then we do beamforming on the stations, and we send the data to the correlator, and this correlator also does the correlation afterwards, and you can see that once the
07:44
correlator is done with it, we store this to disk, and once it's stored on disk, then it's made available to the central processing. So the correlator and our GPU cluster, Cobalt, are doing streaming, and the central processing is more like your traditional HPC.
08:03
When we look at the data flow in Cobalt, there is all this incoming 10-gigabit Ethernet, and this is why we have four or three 10-gigabit Ethernet links per Cobalt server. They are streaming the data, and we configure per station where it needs to send its data to.
08:21
Then once it's there, it's being transposed at roughly 240 gigabits, and once it's transposed, we do have two pipelines that essentially run in parallel, one is correlation, and one is additional beamforming, so we actually beamform twice in a sense. It's a little bit more complicated than I'm sketching here, but I'm keeping things simple,
08:43
because stations also have the capability to not beamform and send un-beamform data, we have a special buffer that's called the transient buffer, where we dump raw samples and can send those to clusters, but the general pipeline is what I'm sketching here. If I zoom into these two pipelines, the correlator pipeline and the beamformer pipeline,
09:06
I don't want you to look at the details too much here, because that's not interesting, and I really don't have time to explain it, but the trick is, almost everything you see here is based on signal processing, digital signal processing. That's what we're doing, we're using the fast Fourier transform, finite input response filters, and transforming the data
09:26
in the frequency domain, if you will. Then it's put back into CPU memory at Cobalt, some final transformations are being placed, and then it's put into the disk so that SEP can work on it.
09:41
At Ostrun, we do a lot with software, and I've showed you now how the data flows, but I haven't told you what software components are making that data flow happen. And for Cobalt, it's actually one solid product that lives in the loafer repository, please don't all now visit or GitLab instance at once, because it will die if you do that.
10:05
So try to spread that out a little bit over the day, I'm sure I will upload the slides soon after, so you don't have to remember all this. And then all these tools that are listed here, basically except for Cobalt, which lives in the
10:21
loafer repo, those are more like what you would find on the SEP side of things. I'm going to explain, I want to address that this is just the tip of the iceberg. On our GitHub repo, you can find a lot of more stuff that you can play with, and I would encourage you to do so, because it's quite interesting.
10:40
Then there's also Casacore, which is heavily being developed at Ostrun as well, but it's not actually just a product just by us. A competitor or like a friend of Casacore would be Astropie, so very widely known packages in the industry. If you look at the radio astronomy toolkit, so the necessary things to go from antenna to
11:04
image, if you will, then these are your friends. There's DP cubed, ASW clean, and IDG, and Rattler, AO flagger, and I'm not going to talk too much about every beam. So what do these things do? What are we looking at here?
11:20
DP cubed is where you define a complete pipeline, so you have the incoming data, you need to do some transformations on the data, maybe you want to identify some noise sources that might be in your data, and eventually you want to create an image, and for this imaging you need deconvolution as well, and you also need gridding and degridding.
11:40
So AO flagger is where you identify noise sources, this can be anything, like radio instruments are very sensitive, so one noise source in particular would be electric fences, windmills, solar farms, bed quality, LED lighting. Then we move to the imaging part with ASW clean,
12:02
because when you have a radio telescope consisting of many small antennas, in between your antennas there are holes, and that means that you're not receiving the data as if you would have a very large parabolic dish, there's some differences. This creates some kind of fringing in the image that you need to filter out,
12:22
and that's what ASW clean together with Rattler and IDG are doing. And IDG is your translation from the data domain to a domain that is useful for radio astronomical imaging. So I talked a little bit about CasaCore,
12:41
now it was industry widely used, it's based on all the packages that have been around for a very long time, but we've actually switched it around now, so now CasaCore is built on top of these older packages rather than CasaCore depending on these older packages. There's several unique features here,
13:02
the UV domain is pretty interesting, so that's the domain about the plane that is filled, so those holes in your surface area, if you will. And there's some Python bindings here, so these are all very nice tools that you can just play with.
13:23
We also use a lot of open source tools, and we're doing quite well, there's still some closed source software, I'll get into that in a minute. So we use OpenMPI, OpenMP, Slurm, GitLab, Grafana, and actually the part that I work on is PyTangle, which is a SCADA system. So with supervisory control and data acquisition,
13:45
that's basically your interface that we have on the individual stations, and those stations then configure underlying hardware with the antennas and the ADCs, and they report how they are configured to a higher level system.
14:01
We also use Prometheus, Docker, and a variety of interesting open source tools. This is just the tip of the iceberg as well, there's much more. Next to our SEP cluster is also pretty interesting, which is actually where we use Slurm, we also have a DOS-SYS cluster, which is a cluster shared with many universities within the country.
14:23
Things where we can improve. Well, we use CUDA, so that's not really open source, compared to OpenCL or Falcon. We're using WinCC for monitoring, maybe you've heard of that package, it's Windows-based, hence why it's called WinCC. We're trying to phase that out for
14:43
Grafana and Prometheus, that's going quite well, I'd say. We have a lot of closed source SPDA Fender blocks, so if you have your Xilinx or what have you, or your Altera, then they, for instance, offer IP blocks to implement 100 gigabit Ethernet interfaces,
15:01
and they're not too keen on you sharing those with the whole wide world. Then InfiniBand firmware is pretty closed source, I believe there's open source versions of that, but I don't even know if they work quite well. And then the main area that we're actually struggling is with office management tools. So this is definitely the area that we can
15:23
improve the most at Astron, we use Office 365, Slack and Zoom, and as you can see, Copano, Metamos, Jitsi, there's definite open source alternatives to this, so there's no real reason why we should be using this. Of course you need to host the infrastructure, and that also costs money, so there's some little way there. I'm not
15:42
saying that it's definitely cheaper, but there's open source alternatives to this. Now I want to show you, I told you about IDG, that does the gridding and de-gridding, I told you about VSW clean, and the Dravata part that does the deconvolution now, and I want to show you how those tools work in practice. So we have an ideal point source,
16:04
this is our most ideal radio source that can possibly exist, it creates a very sharp point in the sky, we put it toward the gridding, and we get a point function. What this basically is, is this is the error of our telescope. So we now know, okay, this is the error it's going to generate in our images, because we don't have complete filling of the UV plane,
16:25
there's holes in between our antennae, and then we can use the VSW clean image together with Rattler for deconvolution to create an iterative process in which we iteratively remove the noise from the image. So actually I'm going to see, oh yeah that's nice.
16:44
So here you see these lines, these lines are the fringes that I've told, and if you then perform these iterative cleaning process on the what are called calibrated visibilities, then we iteratively see that this image is drastically improved.
17:01
So now some example of this, what is the science that we do with LOFAR, how does this look like, well this is a paper by Eric Timmerman, so you can look at it when you spare time, and what we're basically seeing here is we're seeing huge jets of this synchrotron radiation emissions that I talked about, you can see that they actually over
17:22
millions of years they vary in intensity, and at the center of this image is actually a black hole, but you can't see that because it's a black hole, and then on the background of this image there is an overlay of what's the optical domain, so not the radio domain from the Hubble Space Telescope. This is what we've been able to capture with LOFAR.
17:45
So we're doing groundbreaking science, and we're gonna do a lot more, we're in the middle of a big upgrade that's scheduled for the end of 2014, win2c is going to be replaced with CAFANA, we're thinking about ALERTA, but I've heard that CAFANA has persistent
18:00
alarms during actually FOSTAM today, so we might not need ALERTA, and we're using Prometheus. We had these low band antennas and the high band antennas, I briefly skimmed over that because the details, yeah, you have to cut some corners somewhere, but basically with LOFAR2 we'll be able to use both of them in a single observation. We'll also be able to
18:23
use multiple beams, so it talks about the beam forming, currently LOFAR is only able to have a single beam per observation, and we will also be able to point at different points in the sky, change that during an observation, and we call this mega mode, don't ask me why.
18:43
Yeah, we're completely revamping the SCADA system, we're now using PyTango, we have sufficiently upgraded hardware, UNIBOR2s, we actually sell those to external institutes as well, so they're available, and we're drastically improving the timing distribution, so we're currently GPS based, everything is synchronized using GPS, all the stations across Europe,
19:04
and we're going to use the white rabbit protocol that's made by CERN, that's based on the precision time protocol. Now very briefly this mega mode, what would this schematically look like, so this is basically what's running on cobalt or GPU cluster, and we do imaging and beam forming,
19:21
and now we have one beam and several pointings, and they stay the same during observations, now we can have multiple beams, and we can re-point during the observation, that's going to create a lot of flexibility for the astronomers, and I'm going to be very excited with the science that is going to come from this. I want to leave you with some links,
19:42
as mentioned our astron repo, the astron website, there's a very interesting page about 10 years of LOFAR, because we've actually existed, LOFAR has been in production since 2008, so that's been since quite some time, there's this very nice map on which you can actually see all the physical locations of all the stations, how many antennas are active or working or broken,
20:06
so this is all open data, you can just look at this, and there's some history about all presentations that I've done in the past. So, any questions?
20:37
Maybe first a short comment, the Chinese built a 500 meter dish, but that's just,
20:46
but what I really wanted to ask is whether you have collaboration with other astrophysical observations, like square kilometer array or something like that? Well, actually we collaborate on the square kilometer array, so there's definitely,
21:01
can you repeat part of your question, because people were just leaving? Well, whether there is a shared development in software and stuff? Yeah, yeah, for sure, for instance on CasaCore, as I mentioned, but also WS-Clean, we see contributions from external collaborators, and especially the Jive,
21:23
the part of Jive that I showed at the very beginning, let me see, shouldn't be too long, so here I mentioned Jive and EVN, this is a huge collaboration of parabolic dishes that are all connected, and all the data is sent centrally to Dringleau at the headquarters
21:48
of Astolung, and that's actually where the EVN network processes all this data, but all these dishes that we use, those are not ours, right? Those are from other parties.
22:07
Yeah, someone's asking, since everything's processed digitally, can these telescopes focus on multiple targets at once by processing the data multiple times? That's an interesting question, and that depends, as I said, you have the transient buffers which
22:23
dump raw samples, but typically what we do is we already do beamforming on the station, and if you do the beamforming on the station, you're already looking at some point in the sky, you're only sending the result data from that beamforming to this cobalt cluster, you can't beamform again then, the data is lost, it's reductive in nature,
22:43
but if you would send the raw sample data to cobalt, and it could somehow process all the data, which I don't think it has the bandwidth to do so, then you could, at a later point in time, point at any point in the sky again, which is, that's the job of the transient buffers.
23:03
Thanks, maybe a naive question here, would there be any ways or interests for amateur astronomers, or radio astronomers of course, to help or work with astronomy?
23:27
Well, there's definitely job positions on our page all the time, I think, I don't know if most are in the field of radio astronomy, but what typically happens, and I can briefly explain, is we have a system called North Star, in which astronomers submit their proposals and
23:42
describe what they want to do with their instrument, and then we have a community that looks at that, and that actually accepts these proposals, and then they are scheduled. This is actually a very good question, because I completely skipped this in the talk, but I wanted to talk about this, and then things are scheduled in a system called TMS, and that basically looks at, okay, what part of these stations are required,
24:04
and to do these observations and collect these data, then these data are collected and processed on cobalt and sap, and the data products are made available to these individual astronomers who've done that, and they get exclusive access for a period of time to do their research. Okay, thanks, I was more thinking about, just if someone is like in Africa, with a homemade
24:30
dish, like is there any way to like maybe capture something with an SDR, and add a little bit with data, or the scale of things is so different that...
24:42
What's actually very important, or it's rather, we need to model a lot of things and calibrate a lot of things, so that's why all the stations are roughly similar in shape, similar in shape, have similar hardware, so it would be definitely possible to buy your own station, build your own station, and have the same hardware, and then hook it up, that happens
25:01
all the time, different countries do that, buy stations, and then we add them, but having vastly different hardware, and then hooking this up to this system would be very difficult, it's not designed to do that. Okay, so if you would make like a very cheap station, that could be built by amateur astronomers, you could deploy that a bit everywhere in the world,
25:25
and then make your public radio astronomy like that, interesting, thanks.