We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

ETF testing framework: past, present and future

00:00

Formal Metadata

Title
ETF testing framework: past, present and future
Title of Series
Number of Parts
351
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
ETF is an open source testing framework for validating data and APIs in Spatial Data Infrastructures (SDIs). It is used by software solutions and data providers to validate the conformity of geospatial data sets, metadata and APIs. For example, ETF is the underlying framework used by the INSPIRE Reference Validator to validate INSPIRE metadata, datasets and services against the requirements of the INSPIRE Technical Guidelines. ETF is also used extensively in Germany by the Surveying Authorities of the Laender to validate their datasets. This includes Real Estate Cadastral data, Topographic data, Control Points, 3D Building Models, House Coordinates and Building Polygons. In the test environments of the German Laender, a comprehensive series of attributive, relational, geometric, and topological tests are performed on the data, in addition to interacting with APIs and checking for errors in the interface contracts. Other European Union (EU) Member States are also reusing ETF to allow their data providers to test resources against national requirements. Finally, some software tools such as GeoNetwork open source include validation based on the ETF API in their workflow. Goals in designing the ETF software were to create test reports that are user-friendly and self-explanatory as well as to be able to validate large amounts of data, which can be several hundred GB in size. In order to cover different validation tasks and present them in a unified report, the architecture is modular and different Test Engines can be used. Currently the following Test Engines are supported: SoapUI for testing web services, BaseX database for testing XML data, Team Engine to validate WFS and OGC Web APIs using the OGC CITE tests, NeoTL Engine for testing WFS, OGC Web APIs and datasets. As a horizontal and reusable tool, which could be extended to satisfy the needs of different communities and domains, ETF is currently considered as a component of the so-called Common Services Platform under the new Digital Europe Programme of the European Commission. Within this context, activities are planned in 2022 to: i) include the ETF in the OSGeo Live v.15, and ii) submit the ETF as an OSGeo Community Project. In this talk, we will introduce the ETF testing framework, the deployment scenarios and address the current and future technical developments.
Keywords
202
Thumbnail
1:16:05
226
242
Statistical hypothesis testingSoftware frameworkTransformation (genetics)Process (computing)Open sourceMetadataWeb serviceUsabilityRevision controlLibrary catalogShape (magazine)Local GroupChannel capacityFeedbackPublic key certificateSoftware developerPhysical systemInteractive televisionPlanningRepository (publishing)Java appletNumbering schemeHausdorff dimensionObject (grammar)Self-organizationPartial derivativeClient (computing)CASE <Informatik>Statistical hypothesis testingArchitectureServer (computing)Device driverOrientation (vector space)DisintegrationComputer networkSuite (music)State of matterOnline helpSpatial data infrastructureProbability density functionBuildingEndliche ModelltheorieSet (mathematics)Standard deviationConsistencyStatistical hypothesis testingProbability density functionWeb serviceTraffic reportingStatistical hypothesis testingEndliche ModelltheorieProjective planeUsabilityConfiguration spaceDirection (geometry)Presentation of a groupExecution unitLevel (video gaming)Representational state transferMathematicsPlanningCartesian coordinate systemSoftwareWeb 2.0Revision controlBuildingState of matterRepresentation (politics)WhiteboardSet (mathematics)Software bugServer (computing)Library catalogMetreMappingStructural loadSoftware frameworkSuite (music)View (database)MultiplicationSoftware engineeringProcess (computing)Group actionInstance (computer science)Interactive televisionUser interfaceStandard deviationOpen sourceValidity (statistics)Attribute grammarComputer animation
Standard deviationConsistencyStatistical hypothesis testingArchitectureRevision controlStatistical hypothesis testingDevice driverPlug-in (computing)Control flowQuery languageSuite (music)Singuläres IntegralData modelElectric currentBasis <Mathematik>Visual systemIdentity managementComputer configurationLocal ringSelf-organizationInstance (computer science)MetadataGeometryRun time (program lifecycle phase)Windows RegistryWeb serviceSocial classUser profileSeries (mathematics)Set (mathematics)Computer networkBeta functionUniform resource locatorGeneric programmingParameter (computer programming)UsabilityObject (grammar)Latent heatBuildingProcess (computing)Error messageMereologyBuildingWeb serviceRevision controlRepresentational state transferRun time (program lifecycle phase)Suite (music)User interfaceSocial classStatistical hypothesis testingError messageInstance (computer science)Windows RegistrySet (mathematics)AuthorizationStatistical hypothesis testingOcean currentValidity (statistics)View (database)Web 2.0Query languageDevice driverDatabaseTraffic reportingProduct (business)WebsiteGeometryPlanningResultantBlock (periodic table)Line (geometry)MultiplicationPolygonSoftware frameworkEndliche ModelltheorieObject (grammar)Generic programmingUsabilityProjective planeMereologyAdaptive behaviorStudent's t-testMetadataComputer animation
ExistenceRevision controlObject (grammar)Latent heatBuildingStatistical hypothesis testingStatistical hypothesis testingProcess (computing)Run time (program lifecycle phase)Error messageMereologyTopologySurfaceCore dumpPhysical systemCodierung <Programmierung>String (computer science)Object (grammar)Line (geometry)Characteristic polynomialTerm (mathematics)FehlererkennungGoodness of fitTopostheorieProjective planeRun time (program lifecycle phase)Plug-in (computing)Different (Kate Ryan album)Arc (geometry)Statistical hypothesis testingComputer animation
Revision controlFile formatFunction (mathematics)Statistical hypothesis testingLinear regressionoutputJava appletControl flowStatistical hypothesis testingMethodenbankDatabaseWeb serviceState of matterInclusion mapSoftwareSoftware frameworkMetadataWeb pageWikiPerformance appraisalProcess (computing)Sample (statistics)Open sourceArchitectureThree-valued logicCore dumpView (database)Library catalogInterface (computing)Device driverServer (computing)Operations researchInformationRepository (publishing)Suite (music)Expert systemTelecommunicationSoftware developerFormal languageData modelData structureTestbedPublic domainLatent heatExplosionOpen setPlug-in (computing)Visual systemCodeStatistical hypothesis testingSoftware developerProgramming languageSuite (music)MereologyMethodenbankAbstractionExpert systemInterface (computing)Integrated development environmentLink (knot theory)MathematicsServer (computing)Computer programoutputContext awarenessCodeInstance (computer science)Repository (publishing)Revision controlDevice driverTraffic reportingLinear regressionTelecommunicationJava appletFormal languageDescriptive statisticsMeta elementDistribution (mathematics)Cartesian coordinate system2 (number)WebsiteProblemorientierte ProgrammierspracheInformationSampling (statistics)Statistical hypothesis testingProjective planeVisualization (computer graphics)Computer fileResultantExpected valueWeb pagePerformance appraisalGame controllerQuery languageGeometryPublic domainLatent heatComputer animation
Suite (music)Statistical hypothesis testingData structureProcess (computing)Revision controlCore dumpUniform resource locatorImplementationStandard deviationMereologyAbstractionDependent and independent variablesLocal GroupElectronic mailing listCodeContent (media)Data typeWeb pageLanding pageOpen setLandau theoryAreaEmailIntegrated development environmentPoint cloudLink (knot theory)Formal languageDevice driverWeb serviceSelf-organizationLibrary (computing)Client (computing)Software developerVisualization (computer graphics)Statistical hypothesis testingView (database)Group actionPattern languageCASE <Informatik>PreconditionerMethodenbankLevel (video gaming)Software developerWeb serviceSuite (music)Variable (mathematics)Integrated development environmentFormal languageSheaf (mathematics)MetreProjective planeDevice driverProcess (computing)MereologyRevision controlINTEGRALLink (knot theory)Context awarenessComputer fontMenu (computing)Validity (statistics)Client (computing)ResultantCodeLibrary (computing)GeometryTerm (mathematics)Video gameQuery languageComputer animation
Transcript: English(auto-generated)
Okay Hello, everybody. My name is John Herman. I'm software engineer At interactive instruments, and I'm here today on behalf of the ETF steering group these are the colleagues from JRC and interactive instruments and
I like to present some projects that we realized in the past with the ETF testing framework Features that we added recently and our future plans so If you imagine a simple process where data is collected data is processed and distributed
What could possibly go wrong? Well, there are a lot of places where bad things can happen for instance you received incorrect data You created an incorrect mapping or configuration You have software defects
or you have other infrastructure related issues and The ETF test framework is a tool which may have you finding these issues So it's an open source testing framework, which has been developed since 2010 the current version is 2.1 which has been released one week ago and
You can use the test framework through an HTML view Or you can use the rest API Then you can start a test run the test framework loads the executable test suites and executes tests
against the servers or a data set and It's capable of testing multiple resources like data sets meter data view service downloads and catalog services It's an open source test framework, and it's governed by an steering group which currently consists of
Representatives from the JRC and interactive instruments and the steering group members can propose new changes to the software through ETF improvement proposals and The steering group is complemented by a technical committee Which does the technical stuff like merging pull requests doing releases and so on?
so There is an predefined process for the ETF improvement proposals This is a publicly available GitHub project board and in the first column the EIP is a draft
That is created by the steering group members. Then the EIP is discussed in the next column and Then the technical committee analyzes the EIP Proposes changes and then the EIP goes Through next columns in the release process Until it's implemented
So we have some important milestones of ETF In 2011 it was first used in the European spatial data infrastructure network project In 2015 it was used in the test suite for testing GMA data city GMM data
Then there was a big redesign for version 2 for the inspire well-known inspire reference belly data Then in 2020 there was the ADV test suite which brought a lot of new changes that were merged into the new version 2.1 and I talked about the OSU community project application and our future plans
at the end of the presentation So the European spatial data infrastructure network While the project to help the EU members states to prepare for the inspire directive There was a command line tool that the member states could use and execute tests against their
feature and web map services and at the end they received a report at J unit XML report that they had to style on their own and in another German project Which was used by the federal states of Germany, they validated their city GMM 3d building model
Data But that were only a triple TIF data tests There were no geometric tests and we added for that project and web user interface and
At the end you received a PDF test report So important to note here is that we had two different build configurations one for these service tests and one for executing data tests So we came up with three main design goals for the new version 2 for the inspire validator and
We wanted to have user-friendly self-explanatory test reports for the user new PDF or J unit XML reports We wanted to be consistent with the standards Which also means we want to be consistent with the standard that the team engine from OGC uses
Uses an internal model that is structured like there are test suites test model test models and so on and We wanted to be capable of testing all resources in an SDI And this also means that we want to do geometric tests and we want to test data sets and services in one instance
So we came up with a new approach with the approach of test drivers Which adapt us in the test framework which control monitor and transform the results of Test engines and as the test engines we reused existing building blocks like we had
We have an base X XML database, which we used to execute X queries and getting results which are started to result report and to do geometric tests we used
We implemented gmlg weeks, which is a plug-in to execute geometric queries in the XML database and We have soap UI for testing web services Which is based on the soap UI web API testing tool and we also reuse the team and so we execute the
Team engine test remotely through the test driver All this is included in the current Version tool line and also in the inspire inspire reference validator the inspire reference validator uses its own custom UI because it needs to be in line with the
sites of the European Commission So it uses the rest interface And it's built on top as a known Custom UI it's a central deployment and it runs about 200 tests a day and there are about
850 tests that are implemented Please note. That's the production instance still uses version 2 and there will be There is an update plan for January next year The ADV test suite was used by the surveying authorities of the German lender to validate their data sets and
it executes about 600 tests including topological tests and the lender also have some own tests that they only execute
In In the lender, so it's a decentralized deployment and in Germany was an central registry with the tests and For this project. We really had challenging runtime requirements. So a lot of the new features That are merged in version 2 are
performance improvements Okay, so for version 2.1 you will present some of the new features If you know the inspire really data in the old user interface there were a lot of executable test suite that you had to select if you want to
Test a whole theme like the meter data. So We came up with a new approach with the test classes Which allows the user to execute a lot of executable test suites in in one run by just selecting the test class and
This improves the usability and also improves the reusability of executable test suites Because they are generic executable test suites That are parameterized over values and for example, you can have one executable test suite That is parameterized with an with a see a schema validation URL
so Another new concept is the isolated view on defective objects and This isolated view lets you filter out
objects that have defects And would lead to a lot of errors in the report. So the idea is to report this error only once and an example is That You have a building that has building paths and you know that the building paths
Geometry is not valid Because multiple polygons for instance are not closed You do not need to execute geometric tests on that so You report this error only once but also are capable of doing tests like
checking the references so for the reference tests you include both views and The advantage is that you keep the report clean and you also improve the test run time
Beside gmlg x there's a new plug-in for base X which is called topics and it's used in the German project to Validate a lot of cadastral parcels, so It has really good run time
Characteristics about 20 gigabyte of data are tested in eight minutes Including gma arcs and lines from segments and it detects intersections and overlapping objects exclaves holds and about 13 different error codes that you can use for your tests
For the version 2.1 we also added the ETF result checker to the ecosystem and it's a tool for comparing the actual result with an expected result and it's designed to Use in used in in a pipeline for regression testing
So you're at the JSON control file where you say I want to test this executable test suite Then you add in file with the expected result upload the data automatically in the pipeline and receive a result for that okay, so we want to be part of the OC community and
we submitted the application for the OS geo community project and As part of this project we also created a page on the OS geo Website which is accessible through this link
So and the application is currently under evaluation We also submitted a pull request which is already merged for the OS geo live distribution and
ETF version 2 Will be included in the OS geo live version 15 It contains related documentation and some sample ETS like GML tests or GC team engine execution and I think WFS test
our future goals To close the gap between just testing the server interface We also want to test the surf data and the idea is that we have one executable test suite for Checking the interface and one executable test suite For checking the data and then the executable test suite for the interface passes the data to the to the other downstream
ETS, so it's reusable for instance if you don't want to use or test your WFS But OGC API features you can reuse the second downstream ETS for that to test the data
Another future goal is to facilitate the test development and Before let's let's do it one step back. So how our executable test suites are implemented Basically, there are subject matter experts
That specify abstract test suites, that's a abstract Description of what should be tested and then They write it to an abstract test suite repository and a test developer reads this description and
Exchanges information with the subject matter expert and Then he implements the executable test suite in a programming language like like Java X query and Uploads the implemented executable test suite to the executable test suite repository
so we want to improve the communication between the subject matter expert and the test developer a subject matter expert without deep programming language knowledge should implement simple tests and change existing tests and
In this context we also want to improve modularization of the tests and we want to provide a ready-to-use IDE And it is assist the test developer and the subject matter expert so in OGC testbed 17
We came up with a new approach with the new to Testing language, which is a domain specific language, which is tailored to To the requirements that we have for developing tests we Implemented an executable test suite for OGC API processes for the standard
implemented a plugin for Visual Studio code a new test driver and Gave our input to the engineering report, which can be attacks accessed through this link So the DSL is based on the ISO 19 105 which also the test engine uses and
This is an example of an test suite for OGC API processes it looks familiar to similar to to YAML, but it has its own syntax and At the top there are some meter Meter data for the for the test suite then there is the executes
Section where other test modules are referenced and there's a defined section just for using or reusing variables across all Downstream on the lower levels the items so the test module looks quite similar to the test suites and
on the next level There are test cases and test cases have validation steps and the validation steps are structured with a When on pattern the given when then pattern?
So the given part describes the precondition then the when part that's the action that is executed and in the then part The assertion is executed on on the action that from which the result has been received So here are some examples
of the what's Can be asserted with the assertions It's in this case, it's these are JSON assertions for the OGC IP API processes and If you if you're asking yourself do I learn We have to learn a new language
in some parts you can reuse your knowledge of X query or JSON path and use it in the In the assertions and for the rest For all other things the IDE will assist you with the context menu
So if you if you add the JSON as a JSON path assertion You get two tips from the from the IDE and the problem support view in Visual Studio code and outline view to Have an overview of all your tests and it runs on get pot dot IO so you can start the IDE and
Can start writing tests there, so okay? We have a new release which is available on github and
Post your life 15 will use the version 2 the inspire reference really data Update is planned for January the new new to test language will facilitate writing tests and will replace in long term the soap UI test driver
the cross driver execution will allow the tests services be tested and Services tested including the surf data and This project is Important for for JRC, and we think that it's also a good
helps really the OS geo community So if you have ideas or want to get involved in the development, please contact us So here's some resources which can be accessed through this link
like contacting the ETF steering group we have an Java ETF client library which can be used by integrators and their installation manuals and so on Thank you very much, and I'm happy to answer your questions