ETF testing framework: past, present and future
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69011 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 2022233 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Statistical hypothesis testingSoftware frameworkTransformation (genetics)Process (computing)Open sourceMetadataWeb serviceUsabilityRevision controlLibrary catalogShape (magazine)Local GroupChannel capacityFeedbackPublic key certificateSoftware developerPhysical systemInteractive televisionPlanningRepository (publishing)Java appletNumbering schemeHausdorff dimensionObject (grammar)Self-organizationPartial derivativeClient (computing)CASE <Informatik>Statistical hypothesis testingArchitectureServer (computing)Device driverOrientation (vector space)DisintegrationComputer networkSuite (music)State of matterOnline helpSpatial data infrastructureProbability density functionBuildingEndliche ModelltheorieSet (mathematics)Standard deviationConsistencyStatistical hypothesis testingProbability density functionWeb serviceTraffic reportingStatistical hypothesis testingEndliche ModelltheorieProjective planeUsabilityConfiguration spaceDirection (geometry)Presentation of a groupExecution unitLevel (video gaming)Representational state transferMathematicsPlanningCartesian coordinate systemSoftwareWeb 2.0Revision controlBuildingState of matterRepresentation (politics)WhiteboardSet (mathematics)Software bugServer (computing)Library catalogMetreMappingStructural loadSoftware frameworkSuite (music)View (database)MultiplicationSoftware engineeringProcess (computing)Group actionInstance (computer science)Interactive televisionUser interfaceStandard deviationOpen sourceValidity (statistics)Attribute grammarComputer animation
05:21
Standard deviationConsistencyStatistical hypothesis testingArchitectureRevision controlStatistical hypothesis testingDevice driverPlug-in (computing)Control flowQuery languageSuite (music)Singuläres IntegralData modelElectric currentBasis <Mathematik>Visual systemIdentity managementComputer configurationLocal ringSelf-organizationInstance (computer science)MetadataGeometryRun time (program lifecycle phase)Windows RegistryWeb serviceSocial classUser profileSeries (mathematics)Set (mathematics)Computer networkBeta functionUniform resource locatorGeneric programmingParameter (computer programming)UsabilityObject (grammar)Latent heatBuildingProcess (computing)Error messageMereologyBuildingWeb serviceRevision controlRepresentational state transferRun time (program lifecycle phase)Suite (music)User interfaceSocial classStatistical hypothesis testingError messageInstance (computer science)Windows RegistrySet (mathematics)AuthorizationStatistical hypothesis testingOcean currentValidity (statistics)View (database)Web 2.0Query languageDevice driverDatabaseTraffic reportingProduct (business)WebsiteGeometryPlanningResultantBlock (periodic table)Line (geometry)MultiplicationPolygonSoftware frameworkEndliche ModelltheorieObject (grammar)Generic programmingUsabilityProjective planeMereologyAdaptive behaviorStudent's t-testMetadataComputer animation
10:41
ExistenceRevision controlObject (grammar)Latent heatBuildingStatistical hypothesis testingStatistical hypothesis testingProcess (computing)Run time (program lifecycle phase)Error messageMereologyTopologySurfaceCore dumpPhysical systemCodierung <Programmierung>String (computer science)Object (grammar)Line (geometry)Characteristic polynomialTerm (mathematics)FehlererkennungGoodness of fitTopostheorieProjective planeRun time (program lifecycle phase)Plug-in (computing)Different (Kate Ryan album)Arc (geometry)Statistical hypothesis testingComputer animation
11:22
Revision controlFile formatFunction (mathematics)Statistical hypothesis testingLinear regressionoutputJava appletControl flowStatistical hypothesis testingMethodenbankDatabaseWeb serviceState of matterInclusion mapSoftwareSoftware frameworkMetadataWeb pageWikiPerformance appraisalProcess (computing)Sample (statistics)Open sourceArchitectureThree-valued logicCore dumpView (database)Library catalogInterface (computing)Device driverServer (computing)Operations researchInformationRepository (publishing)Suite (music)Expert systemTelecommunicationSoftware developerFormal languageData modelData structureTestbedPublic domainLatent heatExplosionOpen setPlug-in (computing)Visual systemCodeStatistical hypothesis testingSoftware developerProgramming languageSuite (music)MereologyMethodenbankAbstractionExpert systemInterface (computing)Integrated development environmentLink (knot theory)MathematicsServer (computing)Computer programoutputContext awarenessCodeInstance (computer science)Repository (publishing)Revision controlDevice driverTraffic reportingLinear regressionTelecommunicationJava appletFormal languageDescriptive statisticsMeta elementDistribution (mathematics)Cartesian coordinate system2 (number)WebsiteProblemorientierte ProgrammierspracheInformationSampling (statistics)Statistical hypothesis testingProjective planeVisualization (computer graphics)Computer fileResultantExpected valueWeb pagePerformance appraisalGame controllerQuery languageGeometryPublic domainLatent heatComputer animation
16:01
Suite (music)Statistical hypothesis testingData structureProcess (computing)Revision controlCore dumpUniform resource locatorImplementationStandard deviationMereologyAbstractionDependent and independent variablesLocal GroupElectronic mailing listCodeContent (media)Data typeWeb pageLanding pageOpen setLandau theoryAreaEmailIntegrated development environmentPoint cloudLink (knot theory)Formal languageDevice driverWeb serviceSelf-organizationLibrary (computing)Client (computing)Software developerVisualization (computer graphics)Statistical hypothesis testingView (database)Group actionPattern languageCASE <Informatik>PreconditionerMethodenbankLevel (video gaming)Software developerWeb serviceSuite (music)Variable (mathematics)Integrated development environmentFormal languageSheaf (mathematics)MetreProjective planeDevice driverProcess (computing)MereologyRevision controlINTEGRALLink (knot theory)Context awarenessComputer fontMenu (computing)Validity (statistics)Client (computing)ResultantCodeLibrary (computing)GeometryTerm (mathematics)Video gameQuery languageComputer animation
Transcript: English(auto-generated)
00:02
Okay Hello, everybody. My name is John Herman. I'm software engineer At interactive instruments, and I'm here today on behalf of the ETF steering group these are the colleagues from JRC and interactive instruments and
00:22
I like to present some projects that we realized in the past with the ETF testing framework Features that we added recently and our future plans so If you imagine a simple process where data is collected data is processed and distributed
00:44
What could possibly go wrong? Well, there are a lot of places where bad things can happen for instance you received incorrect data You created an incorrect mapping or configuration You have software defects
01:00
or you have other infrastructure related issues and The ETF test framework is a tool which may have you finding these issues So it's an open source testing framework, which has been developed since 2010 the current version is 2.1 which has been released one week ago and
01:27
You can use the test framework through an HTML view Or you can use the rest API Then you can start a test run the test framework loads the executable test suites and executes tests
01:42
against the servers or a data set and It's capable of testing multiple resources like data sets meter data view service downloads and catalog services It's an open source test framework, and it's governed by an steering group which currently consists of
02:02
Representatives from the JRC and interactive instruments and the steering group members can propose new changes to the software through ETF improvement proposals and The steering group is complemented by a technical committee Which does the technical stuff like merging pull requests doing releases and so on?
02:26
so There is an predefined process for the ETF improvement proposals This is a publicly available GitHub project board and in the first column the EIP is a draft
02:40
That is created by the steering group members. Then the EIP is discussed in the next column and Then the technical committee analyzes the EIP Proposes changes and then the EIP goes Through next columns in the release process Until it's implemented
03:02
So we have some important milestones of ETF In 2011 it was first used in the European spatial data infrastructure network project In 2015 it was used in the test suite for testing GMA data city GMM data
03:20
Then there was a big redesign for version 2 for the inspire well-known inspire reference belly data Then in 2020 there was the ADV test suite which brought a lot of new changes that were merged into the new version 2.1 and I talked about the OSU community project application and our future plans
03:44
at the end of the presentation So the European spatial data infrastructure network While the project to help the EU members states to prepare for the inspire directive There was a command line tool that the member states could use and execute tests against their
04:06
feature and web map services and at the end they received a report at J unit XML report that they had to style on their own and in another German project Which was used by the federal states of Germany, they validated their city GMM 3d building model
04:28
Data But that were only a triple TIF data tests There were no geometric tests and we added for that project and web user interface and
04:42
At the end you received a PDF test report So important to note here is that we had two different build configurations one for these service tests and one for executing data tests So we came up with three main design goals for the new version 2 for the inspire validator and
05:06
We wanted to have user-friendly self-explanatory test reports for the user new PDF or J unit XML reports We wanted to be consistent with the standards Which also means we want to be consistent with the standard that the team engine from OGC uses
05:22
Uses an internal model that is structured like there are test suites test model test models and so on and We wanted to be capable of testing all resources in an SDI And this also means that we want to do geometric tests and we want to test data sets and services in one instance
05:42
So we came up with a new approach with the approach of test drivers Which adapt us in the test framework which control monitor and transform the results of Test engines and as the test engines we reused existing building blocks like we had
06:04
We have an base X XML database, which we used to execute X queries and getting results which are started to result report and to do geometric tests we used
06:21
We implemented gmlg weeks, which is a plug-in to execute geometric queries in the XML database and We have soap UI for testing web services Which is based on the soap UI web API testing tool and we also reuse the team and so we execute the
06:43
Team engine test remotely through the test driver All this is included in the current Version tool line and also in the inspire inspire reference validator the inspire reference validator uses its own custom UI because it needs to be in line with the
07:03
sites of the European Commission So it uses the rest interface And it's built on top as a known Custom UI it's a central deployment and it runs about 200 tests a day and there are about
07:22
850 tests that are implemented Please note. That's the production instance still uses version 2 and there will be There is an update plan for January next year The ADV test suite was used by the surveying authorities of the German lender to validate their data sets and
07:45
it executes about 600 tests including topological tests and the lender also have some own tests that they only execute
08:01
In In the lender, so it's a decentralized deployment and in Germany was an central registry with the tests and For this project. We really had challenging runtime requirements. So a lot of the new features That are merged in version 2 are
08:20
performance improvements Okay, so for version 2.1 you will present some of the new features If you know the inspire really data in the old user interface there were a lot of executable test suite that you had to select if you want to
08:45
Test a whole theme like the meter data. So We came up with a new approach with the test classes Which allows the user to execute a lot of executable test suites in in one run by just selecting the test class and
09:02
This improves the usability and also improves the reusability of executable test suites Because they are generic executable test suites That are parameterized over values and for example, you can have one executable test suite That is parameterized with an with a see a schema validation URL
09:29
so Another new concept is the isolated view on defective objects and This isolated view lets you filter out
09:41
objects that have defects And would lead to a lot of errors in the report. So the idea is to report this error only once and an example is That You have a building that has building paths and you know that the building paths
10:04
Geometry is not valid Because multiple polygons for instance are not closed You do not need to execute geometric tests on that so You report this error only once but also are capable of doing tests like
10:25
checking the references so for the reference tests you include both views and The advantage is that you keep the report clean and you also improve the test run time
10:43
Beside gmlg x there's a new plug-in for base X which is called topics and it's used in the German project to Validate a lot of cadastral parcels, so It has really good run time
11:01
Characteristics about 20 gigabyte of data are tested in eight minutes Including gma arcs and lines from segments and it detects intersections and overlapping objects exclaves holds and about 13 different error codes that you can use for your tests
11:24
For the version 2.1 we also added the ETF result checker to the ecosystem and it's a tool for comparing the actual result with an expected result and it's designed to Use in used in in a pipeline for regression testing
11:43
So you're at the JSON control file where you say I want to test this executable test suite Then you add in file with the expected result upload the data automatically in the pipeline and receive a result for that okay, so we want to be part of the OC community and
12:04
we submitted the application for the OS geo community project and As part of this project we also created a page on the OS geo Website which is accessible through this link
12:23
So and the application is currently under evaluation We also submitted a pull request which is already merged for the OS geo live distribution and
12:41
ETF version 2 Will be included in the OS geo live version 15 It contains related documentation and some sample ETS like GML tests or GC team engine execution and I think WFS test
13:02
our future goals To close the gap between just testing the server interface We also want to test the surf data and the idea is that we have one executable test suite for Checking the interface and one executable test suite For checking the data and then the executable test suite for the interface passes the data to the to the other downstream
13:25
ETS, so it's reusable for instance if you don't want to use or test your WFS But OGC API features you can reuse the second downstream ETS for that to test the data
13:44
Another future goal is to facilitate the test development and Before let's let's do it one step back. So how our executable test suites are implemented Basically, there are subject matter experts
14:04
That specify abstract test suites, that's a abstract Description of what should be tested and then They write it to an abstract test suite repository and a test developer reads this description and
14:24
Exchanges information with the subject matter expert and Then he implements the executable test suite in a programming language like like Java X query and Uploads the implemented executable test suite to the executable test suite repository
14:42
so we want to improve the communication between the subject matter expert and the test developer a subject matter expert without deep programming language knowledge should implement simple tests and change existing tests and
15:02
In this context we also want to improve modularization of the tests and we want to provide a ready-to-use IDE And it is assist the test developer and the subject matter expert so in OGC testbed 17
15:20
We came up with a new approach with the new to Testing language, which is a domain specific language, which is tailored to To the requirements that we have for developing tests we Implemented an executable test suite for OGC API processes for the standard
15:42
implemented a plugin for Visual Studio code a new test driver and Gave our input to the engineering report, which can be attacks accessed through this link So the DSL is based on the ISO 19 105 which also the test engine uses and
16:03
This is an example of an test suite for OGC API processes it looks familiar to similar to to YAML, but it has its own syntax and At the top there are some meter Meter data for the for the test suite then there is the executes
16:24
Section where other test modules are referenced and there's a defined section just for using or reusing variables across all Downstream on the lower levels the items so the test module looks quite similar to the test suites and
16:46
on the next level There are test cases and test cases have validation steps and the validation steps are structured with a When on pattern the given when then pattern?
17:02
So the given part describes the precondition then the when part that's the action that is executed and in the then part The assertion is executed on on the action that from which the result has been received So here are some examples
17:21
of the what's Can be asserted with the assertions It's in this case, it's these are JSON assertions for the OGC IP API processes and If you if you're asking yourself do I learn We have to learn a new language
17:43
in some parts you can reuse your knowledge of X query or JSON path and use it in the In the assertions and for the rest For all other things the IDE will assist you with the context menu
18:02
So if you if you add the JSON as a JSON path assertion You get two tips from the from the IDE and the problem support view in Visual Studio code and outline view to Have an overview of all your tests and it runs on get pot dot IO so you can start the IDE and
18:26
Can start writing tests there, so okay? We have a new release which is available on github and
18:41
Post your life 15 will use the version 2 the inspire reference really data Update is planned for January the new new to test language will facilitate writing tests and will replace in long term the soap UI test driver
19:03
the cross driver execution will allow the tests services be tested and Services tested including the surf data and This project is Important for for JRC, and we think that it's also a good
19:24
helps really the OS geo community So if you have ideas or want to get involved in the development, please contact us So here's some resources which can be accessed through this link
19:40
like contacting the ETF steering group we have an Java ETF client library which can be used by integrators and their installation manuals and so on Thank you very much, and I'm happy to answer your questions