AV-Portal 3.23.3 (4dfb8a34932102951b25870966c61d06d6b97156)

Technology of the Characterisation Virtual Laboratory (C-DEVL project) #2 - July 18

Video in TIB AV-Portal: Technology of the Characterisation Virtual Laboratory (C-DEVL project) #2 - July 18

Formal Metadata

Technology of the Characterisation Virtual Laboratory (C-DEVL project) #2 - July 18
Alternative Title
Characterisation DEVL - July 18
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
06.07.18 - Lance Wilson Characterisation DEVL (Data enhanced Virtual Lab). In June & July's TechTalk events, representatives from each of the DEVLs will introduce their projects from a developers' perspective: the problem the project is trying to solve, the tech stacks deployed and to be developed, the approaches of their software development and community engagement while developing tools and applications. The demonstration/discussion of 8 projects are scheduled as follows: DEVL #1 June (1st): Astronomy (Robert Shen), Marine Science (Roger Proctor), Terrestrial Ecology (Gerhard Weis), and Climate (Clare Richards). DEVL #2 July (6th), we welcome speakers from: GeoScience (Carsten Friedrich), Characterisation (Lance Wilson), BioScience (Jeff Christiansen), and Culture and Communities (Sarah Nisbet, Alexis Tindall).
Presentation of a group Virtual reality Voting Set (mathematics) Sampling (statistics) Distance Identity management Identity management
Point (geometry) Onlinecommunity View (database) View (database) Virtualization Mathematical analysis Computer programming Connected space Number Wave packet Data management Data mining Data management Virtual reality Software Integrated development environment Software Videoconferencing Integrated development environment Vertex (graph theory) Marginal distribution Point cloud
Point (geometry) User interface Consistency Decision theory Mehrplatzsystem Disintegration Motion capture Set (mathematics) Online help Image registration Mereology Neuroinformatik Number Time domain Prototype Mechanism design Different (Kate Ryan album) Single-precision floating-point format Software Integrated development environment YouTube Posterior probability Point cloud User interface Source code Consistency Projective plane Computer network Stack (abstract data type) Instance (computer science) Open set Entire function Replication (computing) Single-precision floating-point format Component-based software engineering Process (computing) Software Computer hardware Modul <Datentyp> Prototype Boiling point Window
Consistency Monad (category theory) Multiplication sign Set (mathematics) Instance (computer science) Stack (abstract data type) Disk read-and-write head Supercomputer Roundness (object) Type theory Different (Kate Ryan album) Software Software repository File system Configuration space Endliche Modelltheorie Physical system Point cloud Operations research Source code Stapeldatei Building Moment (mathematics) SYSTRAN <Programm> Ext functor Bit Staff (military) Stack (abstract data type) Translation (relic) Open set Supercomputer Process (computing) Scheduling (computing) Physical system Identity management Asynchronous Transfer Mode
Execution unit State of matter Structural load Configuration space Bit Configuration space Configuration management Rule of inference Social class
Probability distribution Point (geometry) Meta element Polygon mesh Beat (acoustics) Module (mathematics) Euclidean vector Hecke operator Consistency Code Connectivity (graph theory) Multiplication sign Gene cluster Password Set (mathematics) Mereology Rule of inference Revision control Different (Kate Ryan album) Operator (mathematics) Set (mathematics) Configuration space Endliche Modelltheorie Traffic reporting Social class Curve Standard deviation Structural load Software developer Code Basis <Mathematik> Bit Unit testing Component-based software engineering Latent heat Software Repository (publishing) Password Mixed reality Cube Moment <Mathematik> Quicksort Resultant
Point (geometry) Cluster sampling Vapor barrier Service (economics) Java applet Mathematical singularity Gene cluster Client (computing) Web browser Web 2.0 Web service Positional notation Root Set (mathematics) Integrated development environment Identity management Social class Task (computing) Identical particles Authentication Key (cryptography) Moment (mathematics) Physical law Login Cartesian coordinate system Software Integrated development environment Password Internet service provider Order (biology) Normal (geometry)
Cluster sampling Group action Game controller Service (economics) Electric generator System administrator Gene cluster Group action Mereology Dimensional analysis Web 2.0 Web application Computer configuration Password Pauli exclusion principle Videoconferencing Right angle Authorization Routing Sanitary sewer Social class
so oh there goes our samples so there's
a few things that we like to presented a more than anything else this is going to be a technical presentation on the technology that were using to support Federation and also the technology uses for distance and identity but before we do that we just wanted to define a characterization vote for barros cooter
is during the background fundamentally he's a program of work with the characterization virtual laboratory which is a program work to connect Australia and characterization instruments with data management environments tools and mouth pipelines of Australian research the achievements of the program today that we've connected over a hundred asperity instruments many of the fun by Ingres of a significant value to Australian researchers both financial and in department reduced and we've had over 2,700 users of video because a lot of the software that we develop is reused for cost HPC facilities from the researchers point of view it's really just a place where a researcher can access their characterization of data and many of the software's and tools that they might need to process now the
sea devil itself is doing a number of things it's connecting instruments it's continuing to create rich online environments and it's undertaking a training and spare data program the part that we're going to talk about specifically today is the rich online environment so it's really just one segment of what we - but we can't necessarily present you tight acknowledged acceptable program and one of the things that will focus on use the Federation of the characterisation video across more the characterization beyond desktop is the desktop environment that we provide for our research user community largely it's been run across two nodes Mining's consolidate to cloud and it's not massive and we're now federating that out so that it's run across a significant margin of nodes and so the
notice of it we're developing and deploying under this prototype fashion at uwa on the Palsy research firm at youtube as well and a prototype is beginning of the of a Windows CBL instance and also the points in software a University of Wollongong which is part of the studio as well it's important to actually understand that many of the instances are actually quite different and any different partners in the project required decisions so we have gone through a little process to try to work out what Federation and what CD all nodes across Australia multiply as an example YouTube wants to integrate with an existing computer whereas UWA any instance there will be an instance very similar to what we've run at March and it will run basically from schools they're very different instances but what we have done is we've
developed a set of Federation principles that are consistent across the entire network posterior little that will be across container and basically this is what it means to be asleep you'll note it's a single user boil a single spine up and registration mechanism through a single help desk a single shared software step in as much as as possible considering the technical and licensing challenges and a single desktop user interface and you are branding menus background whistles in a sense what we're saying is the consistent user experience across all of the deployments that we're undertaking there's a number of things that are conserved just yet such as be the ability to capture data at number of these instruments that's inconsistent and I probably should because olan's instruments different and so there's
different technology stacks that we're deploying and it's largely these two architectures in wise to deploy from schools on opens bank using out and really out our spirits but our ansible spirits that we use to deploy a cluster and chris will talk about that and the other one is to integrate with an existing high performance computing system and basically provide a new layer over the top which makes that stupid I won't go into this in too much detail because we don't have a lot of time but those are the two models from there over
to Chris all right so please stop me if I get too excited I'm actually quite pleased with the technologies that we've developed between all this stuff because for me it's actually really simple basic bits of technology but we've come together to make something really useful so as I'm presenting this is my phone I'm a little bit dismissive of there's not much to see here but on the other hand it's actually really what we've got as a technology stack to be pulling what amounts to a cluster and it might not be a really high performance muster but it's the cluster moment kind of head mode we have a set of batch we have a shared file system and we most all this round on staff because it's pretty elected in one job bring it in front of us and a woman kid now I'm going to talk about that first a little bit laugh it up about the desktop delivery technologies stuff that actually faces the user now you don't
know it's a configuration management or much like a chef but it's been - you define the configuration and you ask them get up there so you define me instead of one being we end up using
this for almost everything you can stay here and have this lovely example T of probing on the knowledge in a class to find out the uptime so it actually replaces tool but PDSA quantity and it's dumb as bricks it's it's basically wrapping SS that you can do a recitation do bad school but it allows a little bit of Python rule it means you can write how rough my show strip flips you protected state you by the yellow flower defining what you want it's got loads of support around the world state can hang together ideas from other people now salt would have been just as good honestly there's nothing special about
ansible move on if you want to run out of all I think that's great and I think it's one of those things given by coming to us the other really important part is good now we've got slightly weird gear setup with our our definition of a plaster because we've actually found there's loads of bits that are common to every cluster are there the time because I think they're a good idea and I think most people most people know that you better run a cluster you probably going to put L my if you're going to run a class D you're probably going to an abyss to the usual local be getting software off so if I just separate all of our ansible components stop which can get reused and start which is the discrete variables which vary between clusters and most importantly passwords as a result we've got the victim include git repositories one of which we're completely other than anyone can go on neon which passwords in and then we will be horrified if they ever got out to anybody else and we actually have a different password basis were pleasantly different reporting each of the classes we brought now we do is get some models and this it's confusing aspect for most people because get defines mrs. anete isn't bringing in penalties so it's actually really useful to say we've got this set of problems but a common things we do want all of our clusters and the inter cluster depends on that and it allows you to do is sort of QA where the the upstream roles and most recent versions might change but you might have a few a matters ain't ready for your cluster but anybody who do to the kid they know it's confusing in that amount of time and now that you've got a second to get it in there and you have to make sure which one you think an operator to keep everything it's there so it makes a little bit of a learning curve even for high tech people use these a solution really want this scenario leadership piece of code and non-shared piece of code a really important component is
also how we make practice we follow a very standard software developer principle in the one person's going to actually work and submit every question another person's going to review it at that point we may swear a little bit real easy in there anyone from your hand before it we don't have a beat COI button for our postures because it turns out really hard to figure out what on earth a unit test is pretty useless but about when we can go ahead and break and we have one rule if you make a mistake that the users versus you bring in a cake torque want ice cubes back and I had to have a little the documentation up very nicely paper read the docs I encourage everyone to support the principle of self improvement really leads to trying to break things when you know the penalty is apologizing to everyone by bank so that's what we knew
about us was really easily the other really interesting bit in in my opinion of what we do is this trip of venerating across linux-based positive systems really longing to be around the class for a few years you know there's been multiple attempts within the Australian community they hey when they talk about one week to use a name that's that one password even so that all about writing accounts and sounds are causing advancements that log in anywhere you don't have to remember about my passwords this is really nice it will be lovely to be able to authenticate once and access for clusters or even just happier identical authentication keys but but it turns out nobody wants to really stay up on there's a lot of technical work necessary ever make that happen so we actually decided to work around it now we have done not a point here but our clusters we try to give my data environment to try and make sure our software is installed everywhere and one of the things we're leveraging of that for that at the moment still out we're finding this is a really awesome technology more effectively non-root privileges on same things which bothered us the nuts and bolts end for our
Federation when you access a desktop root web browser what happens is the the application which is all actually miss tasks Java and angularjs the application goes and asks for access to a cluster you've been login by AF and because all our clusters do have different user that we've configured at some of the AAF service custom look up your username for a given cluster do live in your web browser school provider or through permission to allow the strudel web web service to access last one normal path and that web service never sees your password for the pasta and only gets a short-lived credential which is going to expire after a while so this is my way of saying really and fastest guys in Oxford not going to lose out in an hour by it anyway this is really one of the ways of the client yet I am very much a really low barrier to entry from the secure bank of the same law it's a secure service you don't have to go or get along order to figure out this is going to be okay
we really felt that someone's getting by and building out their actions to allow each cluster to keep control of they use mountain roads each service provided to people probably use native groups and guarantee that just because you signed up and CDL and we perhaps giving you resources on m32 to be able to do some basic studio look don't necessarily have resources to do work in any other part of the country on any other cluster that's after you talking to those class for administrators to ask for resources now in the future we might get enough with people buying to be able to say yes you signed up this video well clearly Australian characterization that you could get anywhere and are their
generation Federation I'm actually thinking about and developing this right now actually and have our web application asked for access to all of the clusters at once and we'll hole around and see which clusters more able to do access routes all of this is hopefully in our new web UI dependable option to the company between massive and speedy only double-a or copy between masters and right there won't be any age passwords plan currently we've already got shortly the dimensions would just make that happen for you