Sustainable Scientific Software Development

Video in TIB AV-Portal: Sustainable Scientific Software Development

Formal Metadata

Sustainable Scientific Software Development
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Sustainable Scientific Software Development [EuroPython 2017 - Talk - 2017-07-12 - PyCharm Room] [Rimini, Italy] In the experimental Sciences, new theories are developed by applying the Scientific method to produce results which are accurate, reproducible and reliable. This involves testing the experimental setup to show that it is working as designed and thoroughly documenting the progress of the experiment. Results will not be trusted unless the experiment has been carried out to a suitable standard. In computational Science, we should aim to apply the same principles. Results should only be trusted if the code that has produced it has undergone rigorous testing which demonstrates that it is working as intended, and any limitations of the code (e.g. numerical errors) are understood and quantified. The code should be well documented so that others can understand how it works and run it themselves to replicate results. Unfortunately, this can be quite challenging. By their very nature, scientific codes are built to investigate systems where the behaviour is to some extent unknown, so testing them can be quite difficult. They can be very complex, built over a number of years (or even decades!) with contributions from many people. However, even for the most complicated of codes there are a number of different tools we can use to build robust, reliable code. In this talk, I shall look at techniques and tools you can use to build more sustainable scientific code, including testing, continuous integration and documentation
Explosion Area Type theory Software Software developer Sound effect 3 (number) Student's t-test Endliche Modelltheorie Spacetime
State observer Building Machine code Distribution (mathematics) Entropiecodierung INTEGRAL Direction (geometry) Multiplication sign Source code Execution unit Solid geometry Explosion Data model Mathematics Different (Kate Ryan album) Endliche Modelltheorie Data conversion Area Source code Collaborationism Curve Software engineering Software developer Computer Physicalism Bit Special unitary group Control flow Computer Measurement Orbit Degree (graph theory) Type theory Message passing Frequency Phase transition Computer science Website Endliche Modelltheorie Software testing Code Automation Arithmetic progression Resultant Reading (process) Row (database) Link (knot theory) Computer file Theory of relativity Collaborationism Web browser Mass Field (computer science) Supercomputer Wave packet Sound effect Revision control Frequency Operator (mathematics) Software Spacetime Software testing Turbulence Data type Machine code Continuous integration Expression Machine code Explosion Radius Software Personal digital assistant Revision control Formal grammar Computational science Library (computing)
Randomization Context awareness Machine code Linear regression Source code Execution unit Conservation law Damping Error message Physical system Collaborationism Building Software developer GUI widget Computer simulation Parameter (computer programming) Mereology Unit testing Control flow Computer Category of being Message passing Physical quantity Order (biology) Interrupt <Informatik> Quicksort Hacker (term) Spacetime Point (geometry) Web page Random number Numbering scheme Sine Random number generation Algorithm Disintegration Numerical analysis Control flow Similarity (geometry) Branch (computer science) Average Entire function Frequency Latent heat Goodness of fit Nichtlineares Gleichungssystem Quantum Maß <Mathematik> Scaling (geometry) Information Chemical equation Coma Berenices Line (geometry) Limit (category theory) System call Equivalence relation Chaos theory Error message Personal digital assistant Function (mathematics) String (computer science) Infinite conjugacy class property Normed vector space Revision control Building Entropiecodierung Multiplication sign Plotter 1 (number) Set (mathematics) Parameter (computer programming) Mereology Mathematics Bit rate Gravitation Square number Cuboid Series (mathematics) Geometric quantization Area Algorithm Linear regression Complex (psychology) Physicalism Price index Perturbation theory Entire function Gravitational wave Computer science output Software testing Code Physical system Resultant Game controller Functional (mathematics) Collaborationism Limit (category theory) Theory Revision control Average Operator (mathematics) String (computer science) Software Spacetime Software testing output Noise (electronics) Execution unit Machine code Axiom of choice Interactive television Event horizon Charge carrier Conservation law
Random number Numbering scheme Functional (mathematics) Randomization Constraint (mathematics) Sine Multiplication sign Equaliser (mathematics) Letterpress printing Limit (category theory) Function (mathematics) Average Limit (category theory) Mathematics Average Function (mathematics) output Pulse (signal processing)
Point (geometry) Area Execution unit Algorithm INTEGRAL Divisor Algorithm Image resolution Plotter Computer simulation Price index Line (geometry) Plot (narrative) Error message Different (Kate Ryan album) Quadrilateral Order (biology) Software testing Nichtlineares Gleichungssystem Simulation Error message
Functional (mathematics) Machine code Algorithm Equaliser (mathematics) Continuous integration Total S.A. Machine code Circle Software testing Error message Scripting language Machine code Mapping Continuous integration Cellular automaton Graph (mathematics) Branch (computer science) Approximation Numerical analysis Error message Computer configuration Circle Personal digital assistant Software testing Simulation Resultant Flag
Functional (mathematics) Installation art INTEGRAL Multiplication sign Virtual machine Mereology Machine code Mathematics Graphical user interface Hash function Repository (publishing) Software testing Source code Computer icon Machine code View (database) Building Concurrency (computer science) Continuous integration Computer file Line (geometry) Bookmark (World Wide Web) Event horizon Circle Revision control Software testing Physical system Electric current
Fraction (mathematics) Machine code Functional (mathematics) Machine code Circle Continuous integration Software testing Machine code Flag
Web page Laptop Reading (process) Complex (psychology) Wavelet Functional (mathematics) Machine code Vapor barrier Maxima and minima Denial-of-service attack Control flow Branch (computer science) Plot (narrative) Field (computer science) Variable (mathematics) Goodness of fit Computer hardware Software testing Physical system Machine code Standard deviation Block (periodic table) Line (geometry) Variable (mathematics) Computer Benutzerhandbuch Personal digital assistant Function (mathematics) Ideal (ethics) Automation Fingerprint
Laptop Machine code Table (information) Maxima and minima Content (media) Machine code Tangent General relativity Frequency Machine code Execution unit Logical constant Suite (music) Web page State of matter Physicalism Coma Berenices Food energy Term (mathematics) Bookmark (World Wide Web) Statistics Component-based software engineering Hooking Benutzerhandbuch Velocity Personal digital assistant Function (mathematics) Sheaf (mathematics) Convex hull Nichtlineares Gleichungssystem Software testing Electronic visual display Simulation Physical system Resultant Laptop
Presentation of a group Context awareness Group action Machine code Video projector Distribution (mathematics) Entropiecodierung Multiplication sign 1 (number) Insertion loss Run-time system Mereology Disk read-and-write head Different (Kate Ryan album) Data compression Cuboid Endliche Modelltheorie Office suite Website Proof theory Source code Electric generator Software developer Open source Digital object identifier Proof theory Repository (publishing) Website Code Quicksort Video game console Arithmetic progression Writing Resultant Reading (process) Point (geometry) Web page Vapor barrier Identifiability Overhead (computing) Open source Link (knot theory) Patch (Unix) Electronic program guide Digital object identifier Online help Rule of inference Field (computer science) Twitter Hypothesis Revision control Term (mathematics) Internetworking Software Integrated development environment Software testing Machine code Distribution (mathematics) Standard deviation Information Run time (program lifecycle phase) Projective plane Line (geometry) Limit (category theory) Integrated development environment Software Intrusion detection system Object (grammar) Pressure Tuple Library (computing)
hi everyone and ceremonies of and I'm a 3rd year phd student at the University of Southampton and I will say a 2017 Fellow of the Software Sustainability Institute and say I'd like to talk to you date today about sustainable scientific software development and so that's not my motivation for wanting to talk about this and said area in my research I model explosions in space and all he won a more technical title it's the effect of including general activity in models of type 1 x 3 best neutron star but basically explosions in
space and say a bit more detail
and say I'm looking at modeling these explosions which happened on neutron also and say if you look at the yeah top left here that's kind of an artist impression of a neutron star the the things that fold when big stars explodes as a supernova and read down remnant say something that's about 10 kilometers in radius about T times the mass of the sun so there is really crazy physics happening there and the Saturn and Titan each of stars they're in orbit around stars like the Sun this means a kind of whole matter of these companion stars and it's matter builds and builds and eventually you get an explosion and unease explosions they actually happen every few hours and we have about 59 sources of these say we have lots and lots of observations and unlike most astrophysical phenomena and but polishing neutron stars a region we don't really know much about them so it is really hard to try and understand what is causing these explosions and kind of were different and phenomena we see within these expressions say I'm trying to occur including generated here models to try and understand further and so is that I saw them my K she about 3 years ago and I did a physics undergraduate degrees are in this lecture learn about experiments and stuff and I kind of fun of a crazy when I started my PhD in solid reading all these different papers about really exciting case and I look them up and I found not on different souls say I can't find one-to-many many much download them and when the myself and they have next to no documentation so the really big carried in my field and and the 1 on the website and only instructions for like like a supercomputer in Germany I had no idea how to install my own computer and had defined appears Dr. sit down next to me and tell me how to get it to install and is also kind of some questionable approaches to testing and so is very common in my area to you have kind of standard to results and people will show that code can replicate the standard results but is not a real unit testing all and integration tests or anything like that and and passing I think this is very good science I mean it kind of goes against the scientific method say and yeah and kind of I got that out of this and phylogeny my supervisor he's quite clear in seasonings and he pointed me in a direction the Software Sustainability Institute and this is institute and that was set up to be promoted and better research software and say I'm kind of helping to spread their message today hopefully and so here's an overview of all I'm gonna talk about for the rest my talk and success and it took about 1 is software sustainability and why you should care about it and I'm going to go over why scientific software is different from other types of software and then I'm going to presented kind suggested and scientific software development what phase and so kind of allayed different steps you can take to really make sure that your code is sustainable the and so what a scientific software and sustainability and said area so I was thinking when my colleagues about this like he doesn't do any coding and he was like a resistance like printing your code on recycled paper and but it's itself was its inability to me is so will your code still work in 5 10 20 years time and serve you publish a paper with results record chances are people organize look-up your paper in a few years time and again wants run your code and data results again and so kind of those 2 main issues here like can you still find your code say keep something on the internet websites often rearranged you need to really make sure that someone's so then we have to click on a link all TyCor your into the browser and also we have to find your code and there's also the issue of can they still running OK and said to if you write your code and and it depends on the library will the likely still be there in 10 years time and and kind of things like this and I still like if someone called final carried over 1 in 10 years time the not harm scientific progress because people have to kind of this the antiwar not gonna be up to and and based their research on previous results also going have to kind of recreate the wheel every time and I don't think this is very good and it is so what makes scientific software different and said area with lots of software it's made to solve a problem where you know what the solution is going to be you kind of have a good idea of what we're trying to and create when right OK it and that was scientific software you don't necessarily know what the result is but you probably writing the software because you can't solve this problem analytically and you can't do it using normal mathematical techniques and the of scientific codes are often developed over long periods of time say is a bit carried my field again mention and and it's been developed since the early 2 thousands and as curves which go back to the seventies and hence why there's still a lot of Fortran 77 floating around and and they can also involve a lot of collaboration say it could be worth hundreds of people across the world and then finally that built by scientists not software engineers and signal also people writing research software you don't have any formal coding training these appear here self taught and say the kind of maybe a not aware of profit trading practices and and I think this is 1 of the main reasons why you get kind of issues of avidity and in science and they say what so the scientific method this is something that is used in the experiment science community and it's a principle that you don't trust a result and unless there is for testing of the operators to give using experiment and its full documentation of the method that you have used in your experiment and when you do this to show your results or accurate their reproducible and reliable so someone is reading your paper you've done everything you can possibly due to convince them that they can trust the results year of gains all accurate and probably true and in computational science you essentially doing experiments but the computer is your apparatus so to me I think that it means you should you should still follow the scientific method and we really shouldn't be trusting carried star must have been properly tested and properly documented and the so this is a nice picture of an experiment and in experimental sciences say is quite well laid out loud and like that she's probably for following proper scientific principles make have experiment is as reproducible reliable as possible and we get a crack something that's more common in conversational science and this was correct from PhD Comics and they got like a million of different versions of a file it's probably very disorganized I doubt this person is using version-control yeah this is not right they it's say now I'm going to try and suggest a development work that you can use when building research software to try and make it as easy as possible to implement some good so for sustainability practices and support what you know lots of tools out there and the World Cup has also make things a remake is easiest possible frost yourself against the best present show I expect almost everyone if alive 1 here uses but draw and from a sizes respective I like to think this is kind of a conversational science measurement molap and the kind of it is a way to keep a log of all changes to OK it's a kind of all the things you've tried out maybe the works actually they didn't work and this is all
about Caesar alike mice picture from wikipedia I found here and I think this is 1 of my readers and say this is kind of like think lot but I now and if we look at and it kind of connects is maybe not the best commit messages but the kind of really shared the different things that kind of show the changes of the carrot and how it's like developed at a time and I had said there's another thing about those ensures a really aids collaboration and silicon up over right and your collaborators changes and and it helps you track without fear say you can develop everything on a branch and there's no danger the overnight the best the break everything and which I Stephanie did many times before I was implementing provisions interrupts A's and mn it is a testing I said that we shouldn't be trusting our results unless we've got proper testing and Swedish should operators on a method and have really been demonstrated to work properly and we need to make sure we understand any limitations of all experiments say in experimental science always measured the systematic carriers and random errors and all these sorts of things more trying understand all the possible errors that could occur in an experiment and incantations I've kind equivalent of this is understanding where we could get numerical errors due to like floating point computations all because we using a low order algorithm and In our simulations and say all this is that cost here but I expect most of you probably heard about the like your experiment so this was the gravitational waves tection experiment and that you might have heard about last year and and when they published the results for this experiment they publish a whole series of papers giving really detailed information about this discovery and on 1 of the papers they publish was characterization of transient noise in evolves like irrelevant gravitational-wave signal GW 59 I 14 and catchy title and but in this paper they really go through the and they characterized all the possible sources of noise at all the possible limitations that were in the experiment and its so careful it's like there's so many pages in this and they have a lovely plots really detailing all the possible sources of noise an error in our experiment and is just 1 of like 10 because they have honest and the say we really should be doing something similar in computational science to understand I errors like that and so foolishly scientific case can be quite hard to test because of a very complex solving some really difficult equations and and their investigation names we we don't really have a benchmark quite often to 10 test mn against we don't know what the solution looks like and what this does not mean that we should come really excuse ourselves and give up the like arts hot contest and the a case there is a series of steps we can go through to kind of and do our testing so 1st thing that we can break hunkered down into unit test and say you can't really trust your entire could works unless you know the individual parts work and is also makes testing your context is a lot more manageable and and when you're building these tests and really importantly you cover your entire parameter space and you also check the OK breaks when at us and say have a little example here and so a very simple function here fine just taking in that's on time swearing them and then I'm using unit tests just to build a simple set of tests here and so my 1st test is just a very simple ones so this is maybe a common normal value that we'll get when we costs and when we use this function and then got an edge cases like that of really large number in there to check that my code is still working and I'm checking and at rates when it's also if I pass in a string but it breaks property and this and this tells me off at an error say was my error and if we look here we can see on this line the square of minus 2 is not 25 say James upfront again it works through I can must and again said yeah sorry the if that is the next step and so once you checked the individual parts what you then need to check that they will work together and so an example I like to think of as if someone had got all the parts for in a shows you that they test them populate work individually and any comments you not see to build a car the you might spend most people as Rome had never bill call from scratch say if they try to put all the pieces together the cost for the noggin what property and so just because you got unit test does not mean that your code the work when all these parts interaction together and phylogeny testing a lot more difficult here because the code is a lot more complex and that this is a really important step the and then finally and you need to monitor the the development of your code with regression tests answer regression tests away you get individual versions of your code and you compared the results against each other and so ideally the performance of your code should always be improved over time or he's not getting worse and if it is getting worse and this is a really good indication of what was developed and and it can kind of this is a good way to kind of catch box where bidding completely breakthrough coast they kind of introduce some more software as and a good bayonets there's is if you have lots of people using OK why you developing it in this can help you enforce backwards compatibility for your users which is always a nice plus and against a some science specific issues with testing and so I said that you can count on main behaviors you don't know what the results of been look like and so good thing that we can do here again with inspiration from experimental science is we can use controls and Sir control is where you put in really simple input data into your cage and you check that it's still producing the correct solution and see if you think a really simple example we do if we have a balanced set of scales and and you wanna check your scales and read 0 a when you have nothing put on them they don't read theory then you have some kind of a systematic errors you have kind of some offset in your scales and so we can do this with period as well so I quote from built simulations with systems of over time and is quite often a solution where the system to just stay static had nothing to change and and if I could in the input data it does stop changing and that's a really good indication I've errors that and the lots of simulations and take that this is something that we can do an unknown a big problem that we can get in science codes is there can be some randomness say we could be persistent data where some random noise and that and we could be looking at something where as the square of a chaotic system and so says that that we can do here is we can try and isolate the random parts seeking use unit tests to check all the other parts work and and then I think tech around parts individually and also we can do things kind of by looking back at the equations with solving themselves with them and use this to test averages and we can check limits and In particular physics we can check conservation of physical quantities and say another example for that and they say hey I'm just gonna define an array of some random numbers and then then apply function to squaring and multiplying it by the sine and so if plot
out and we can see the gets some random stuff of fervent again it changes
the say got some random data and so if we look at this function and we can kind of we can put some constraints on what we think the answer is the answer is gonna change every times we can't check that all the numbers in this array are equal to another array and
but we can check so we know that the impact is between 0 and 1 say we can check that the output is between 0 and sign off on and we can also check that the mean value here is equality around 0 . 2 to 3 and so if we run this then hurray it works so just tested that we got the limits we expect and checked that the average is what we expect and so that's the way we can deal with randomness but in a way that we can do with randomness in arcade and yet
said the simulations and you off from the simulations what we often do is we take something is continuous and we discretize it's we solve it instead of solving it infinitesimally small points we solve on a grid and but this leads to errors in our solution and it means if you make the grid finer fine age you should be getting a more more accurate solution and and this the accuracy of the solution should improve the order of the algorithm that you've used to solve these equations and and if it's not then it's a good indication the or algorithm has not been implemented correctly and if we look at another example of
this so I'm going to use something called the Trapezium will to solve this integral and severe member you might remember from like sectaries collapse and UPC immoral and this way you kind of divide up the area under into go into related to museums and a new some their areas together and say 0 minus so this integral here an honor array of different resolutions that is then iterated about and then I'm going to calculate the error compared to mining solution say or not and then by then he here I'm going in a plot the errors and see what they look like so here so the errors of the b . 7 unlike cottage in this red line here is i've plotted be squared so I know that each of PC-MOS an algorithm which is order H squared accurate and say because my errors have the same slaves as H squared is the really good indication of my algorithm has been implemented correctly and can and say yes
we could also have issues with numerical errors say because we doing floating-point computations and because the China prox makes me exact maps using and non-exact graphs and it the is likely that will get error is building up and say if we use simple equalities in our testing then is probably not gonna work very well say we can do things here and by kind of doing approximate equalities say functions like case and on light will place and for the race again it if you tests and stuff and not so great the need to make sure you run them frequently and to make sure the continues to be well tested and and we can do this by using and tools such as Travis EI and circles CI and and there are many of the cell that is just a couple I quite like for Python and so these continuous integration tools will readily vinyl tests for you and then report back to results which is really great and say here's a little
example for my code say just very simply Miranda a riddle scripts having it what I wanted to see Ron I said I wanted to run these Python convergence and then is run on a
virtual machine for me and it's told me
right at the very end that way paths say really great I can
set up these tests and pretty much forget about them and every time I pitch changes to get up these will be executed that's really great and
say continues integration is really good because you can
out when but occur much sooner than and therefore we can fix them when they occur rather than waiting months down the line and when the parts that likely to be much more entrenched in arcade and say this is important because and biases the danger here and served I said we can pretty much set up the continuous integration and forget about it and but if you're developing your carried a began add new functionality to OK all the time and and outdated has a pretty
much as is this is no tests and because the 10 you and where the OK is correct duty that well compared to old and benchmark and and so this is not great and the and look and the answer is your testimony cover 20 % of your code there is no guarantee the other 80 % is going work an answer tool that we can use to really check that we are testing all of arcade and not just a small fraction of it is and code coverage tools and someone I quite like his kickoff by a and so
here's lkuko by looks like and for my code here and so if we look into the One I functions and we can see
a highlights all the lines and being executed during the tests and it shows me here there's a little branch in my tri-accept block here which has not been executed in testing and this is important because it is such a was probably written for an edge case and mn and edge cases tends be way OK breaks most often say is a really good way for me to find out and what what tests I should write in high should make sure that all of my code is being tested properly setting and it's
a documentation and so we need to make sure that someone else in our field can set up a page and run it without us having to physically sit down next to them and explain it for them and this is a massive barrier if you have me people entering the village and because it means then are going to be up to run might get started on from experience I can say is like quite alienating and the so we may need to make sure we include comprehensive installation instructions not just instructions for a couple of it see the computer systems but instructors will run on standard Linux systems say anyone with the right hardware run OK and we need to make sure he adds to the code itself should be documented so we use sensible function names variable names comments and so we have self-documenting code and so this means that if you got a really complex page people come look at the code itself and understand without having to be 2 pages and pages of documentation and a really good thing in science fairs and is having user guides with examples which demonstrate usage and scissormen I found for my Python creates is really that's really great this is having Jupiter notebooks and say
something that I did when I write like a Lagrange using as a demonstration here and we very Egyptian notebooks which kind of explain what the physics behind the code is and then we then run the period and we show you what results should look like from arcade and and then you can actually use these end with the docks documentations you can use in the Sphinx and to convert this into and money down
and then you can them render it in adults and this is a really great way to kind of build up user guides and with example cases for your users can
so funny distribution say as at the at the beginning you might remember that we need to make sure that our code is still and we can still find in 5 10 20 years time and say offensive to me he could findable was making a difference rules in the 1st place and this is little caveat here understand if you're working in a sensible sensitive environments may for working in and like the medicine industry of something like that then you might have part proprietary information you can't share but if your modeling black holes it for me open a get rich off black code so it doesn't really make sense much sense to me for this code not to be open source and you need to make sure they will always be able to be found say having at DOI and seduced object identifier is a way to provide a common link OK at which point disappear with time and we also need to make sure that we can the produce the runtime environment for arcade and say this is some stuff like making sure that and when people run your care they and have the right libraries and schools on the right versions of the library so the decoding anyone's and he . 70 to make sure that people are aware of that and say an easy way that we can do this and Fry uses by packaging of cage and subtracting Ching and things like docker containers for using corner environments and and things like that and basically your installation should be as painless as possible so it shouldn't be that people have to write lines and lines in the online and they should just be up to like a bright make and everything builds and and also it's really great to try and limit reliance on non open source libraries material this is again crates of barrier new users and so is not really very good practice and also you have no idea that this morning cancels material still going to be there in 5 years time and again so to conclude we need to make sure lock software is future proof and because if it's not future-proof this been Hong scientific that progress and we need to apply the scientific method to software development and Cook competition tational science is the science so we should really treated like this and and code should only be entrusted if they've been demonstrated to be reproducible so anyone can get anyone in your field can go online you collocate instill that when you run it themselves and the use of results and the need to test it to prove to you that any new results they get are actually to be trusted and you should document your page so that people actually understand what it's doing and contend she happen and develop themselves and so if anything I've said here is interested you can I really recommend checking out the southwestern EBT Institute website and because they have those of guides up there and which really as a stuff and a lot more detail b and said thank you and thank
you lose so we have time for a few questions of interested asking something Hey good thank you thank you for the talk was very interesting I am worried how are you convince people to adopt this especially the older generations because for example my advisor doesn't care of sharing the group or doesn't even care of shit reading might give up locks so how how long have you been you say you're elected that a lysosome of which do you of how has been your experience with all the people yeah that is like a really challenging things there I think just nagging them and really just trying to put pressure on them I mean it's very hard deafening as a less senior person to try and encourage the more senior people that these practices all good but putting pressure maybe on things like journals to make sure they have higher standards say so that they only accept codes and maybe a console so which have proper testing to show that the codes of working properly and but I think awareness whelks I think a lot of these people because their self tall and have always done it this way that is not aware that such things all really bad practice and the yeah it's definitely a challenging to be convinced that may be more senior people that they should change the base any other questions and I do for a book on how much overhead to think you talk to your project this approach I mean in terms of time and I mean for me I feel like actually doing this sort of thing saves me time because by having tests like from catch the box early on an and so I mean my master's project for example and I have heard of the neutral or anything like this and and I was working on a carried on for months and months they didn't work and 2 weeks before I had to hand in my thesis I found heart created a single value when testing and if I had some kind of testing a intron and now I would have called I would save so many months and so I think it was an initial outlay of time definitely by fill in the long run this will save time Hey what would be your advice for nurturing could or things like that and that we can actually use in research articles the and for research is when you write a paper and how would you share Kudo would you put according to the projector shows the pets so would you want and I mean yeah definitely having a DOI and to make sure that you all and you curves like it will always be on to be found in the future internet Stephanie and has I mean yeah having a gap of repository is great but we should really rely on the fact that help still gonna be there in 20 years time 1 last question but the it congratulations on very good presentation in I would like to know about the US sometimes when I I read 1 2 papers the Kompas scientist and uh go to show the court in the data in the quality of the paper using a good deed a reduced for isn't but the is not possible to reproduce the coded into that people in heavy the at any place places trouble to all parts of the data into the tuple to integrate with the source code using it to have the but about the the the low infant about this several of the question go out when uh ways possible to put the data into the results about debated because sometimes delta region in the data is not possible to reproduce the research for a yeah office of my head I can't remember I'm afraid I didn't really produce by vast quantities of data and I know definitely I could definitely look it up and if you ask me of on and they're every websites out there for scientists to dumb the detail on and make it and so forth so I know that the ACM is starting an effort to try to you have to reproduce polar code from for the papers so you can get a patch if you submit to the coach in working the started but there at the start generally used to get to of the wounded unity this is that the selling trend in you'd think you'd and lose fj