Interoperability Rules for an European API Ecosystem
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44888 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2018117 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
Rule of inferenceSoftwareDigital signalSoftware frameworkScalabilityArchitectureWeb serviceFatou-MengeJava appletSoftware frameworkSoftware architectureSystem administratorFeedbackPhysical systemWeb serviceStandard deviationComputer animation
02:03
Software frameworkEncapsulation (object-oriented programming)Gateway (telecommunications)Duality (mathematics)Enterprise architectureWeb 2.0Web serviceSystem administratorSoftware frameworkTelecommunicationGroup actionBus (computing)Computer animation
02:55
Software frameworkEncapsulation (object-oriented programming)Gateway (telecommunications)Process (computing)Error messageStructural loadCAN busMilitary operationSoftware maintenanceTelecommunicationWeb serviceVapor barrierCommunications protocolComputational physicsArchitectureMessage passingSemantics (computer science)EmailData managementConditional probabilityRange (statistics)Latent heatEncapsulation (object-oriented programming)Web serviceEnvelope (mathematics)Row (database)Error messageSemantics (computer science)Message passingIntegrated development environmentData managementGateway (telecommunications)Directory serviceBefehlsprozessorQuicksortPhysical systemPoint (geometry)TelecommunicationRoutingVapor barrierFile formatCodeOnline chatEmailWeb 2.0SoftwareCommunications protocolStaff (military)LastprofilFiber (mathematics)Computer wormComplex (psychology)Software frameworkComputer animation
09:15
Numbering schemeStrategy gameSoftware frameworkStandard deviationFatou-MengePhysical systemStandard deviationArithmetic meanWindows RegistryRow (database)Digital electronicsPattern languageSoftware frameworkWeb serviceStrategy gameSystem administratorImplementationPoint (geometry)Representational state transferOpen setComputer animation
12:13
AuthorizationQueue (abstract data type)AuthenticationRoutingEstimationComputer data loggingCodeOntologyCommunications protocolWeb serviceInternet service providerAuthorizationWechselseitige InformationDifferent (Kate Ryan album)Wrapper (data mining)State of matter10 (number)Variable (mathematics)Validity (statistics)Physical systemSystem administratorPersonal digital assistantSerial portSet (mathematics)Library (computing)Uniqueness quantificationComputer data loggingSelf-organizationBinary codeMessage passingInterface (computing)Unit testingTelecommunicationOpen setSystem callOntologyData conversionOpen sourceSoftwareComplex (psychology)Multiplication signMereologyImage registrationParsingLevel (video gaming)Physical lawInformationCausalityTransport Layer SecurityEncryptionAuthenticationCommunications systemStandard deviationException handlingCivil engineeringNumberDaylight saving timeInternetworkingComputer animation
17:40
Software frameworkData managementDisintegrationContinuous functionFatou-MengeSelf-organizationWeb serviceBit rateLimit (category theory)EmailExecution unitMeta elementRepresentational state transferCommon Language InfrastructureError messageSoftware frameworkWeb serviceTime zoneResultant2 (number)Data recoveryTimestampAnalytic continuationDifferent (Kate Ryan album)TelebankingMathematicsDependent and independent variablesSystem administratorSystem callMultiplication signEmailImplementationStructural loadServer (computing)Queue (abstract data type)INTEGRALReal-time operating systemLimit (category theory)Office suiteStrategy gamePlanningData managementBit rateStandard deviationDatabase transactionNumberPhysical systemReading (process)Self-organizationSoftware architectureGoodness of fitCASE <Informatik>Error messageObject (grammar)MultiplicationStaff (military)Point cloudLevel (video gaming)Remote procedure callData centerTelecommunicationNetwork topologyComputer animation
25:20
Pressure volume diagramAbsolute valueMaß <Mathematik>Error messageMetric systemPrice indexBasis <Mathematik>Public key certificateSoftware frameworkStandard deviationEncryptionCommunications protocolElectronic signatureSign (mathematics)EmailAxiom of choiceFunction (mathematics)Electric currentFingerprintObject (grammar)RSA (algorithm)Key (cryptography)Structured programmingAxiom of choiceWeb serviceEmailError messageCartesian coordinate systemEncryptionElectronic signatureMetric systemKey (cryptography)Sign (mathematics)Subject indexingSoftware frameworkData structureSystem administratorInformation securityEmbedded systemPublic key certificateElliptic curveStandard deviationResponse time (technology)Perspective (visual)Object (grammar)Web 2.02 (number)Bit rateBasis <Mathematik>Dependent and independent variablesGroup actionQuicksortBinary codeMultiplication signAbsolute valueMereologyWordRepresentational state transferCommunications protocolEllipseStorage area networkComputer networkField (computer science)BitExecution unitDigitizingTelecommunicationConnected spaceComputer animation
31:33
Software frameworkSystem callSoftware frameworkInteractive televisionWeb serviceDifferent (Kate Ryan album)Similarity (geometry)DigitizingComputer animation
32:52
Digital signalTransformation (genetics)Goodness of fitWeb serviceEmailData conversionRepository (publishing)Computer animation
33:53
Sign (mathematics)Queue (abstract data type)System administratorDifferent (Kate Ryan album)Slide ruleCodeProjective planeSoftwareComputer configurationValidity (statistics)Web serviceObject (grammar)BuildingEmailSelf-organizationWechselseitige InformationBinary codeSerial portMessage passingRepository (publishing)PlastikkarteForm (programming)Cartesian coordinate systemPositional notationQuicksortProcess (computing)AuthenticationMultiplication signEncryptionOpen sourceMobile appLatent heatPhysical lawSeries (mathematics)MereologyRepresentational state transferCentralizer and normalizerPerformance appraisalCommunications protocolProduct (business)Meeting/Interview
Transcript: English(auto-generated)
00:04
Hi everybody. Today I will show you the Italian work toward an interoperability framework that aims to go beyond SOAP.
00:26
Okay, we will start spotting the main issues we had with the actual SOAP framework. Then we will see what changed in the HTTP world that could enable us to move from SOAP to REST,
00:43
and finally the standardization and reliability features we had introduced in the new framework. Finally, I will show you some ideas on which your feedback is very welcome. But before everything,
01:01
I work in the Italian digital team. If you want to ask, working for the Italian government is wonderful, at least until you are in the digital team. We are 30 people, quite skilled,
01:21
and we strive to make public services more accessible for citizen and business, introducing an approach to reliability and to reliable architectures in the public administration world. That is, it's very tough work,
01:40
and everything we want to be based on API, because it's a nice way and a machine-readable way to communicate. Who I am, I'm taking care of the API ecosystem in the digital team.
02:04
At first, essentially the government world have always been a closed world, an enterprise world, but now we want to move this approach from an enterprise closed approach
02:21
to a broader web approach, a distributed approach. This was the old SOAP framework. You see it, it's complex. It has a lot of layers.
02:44
Administration created services, and then they had an enterprise service bus taking care of communication, internal communication, while external communication should transit to a very custom, ad hoc and specific SOAP gateway
03:01
that takes care of a further encapsulation. Then it communicates on a private network, private governmental network, to another sub-gateways that then encapsulate and so on. It's complex, it's costly,
03:20
and moreover, it has problem with reliability. Because it's complex to do service management in this kind of environment. Why this complex? Because SOAP does communicate issues with only one HTTP status code that is 500,
03:44
and everything is into an envelope. It's not XML. So errors are serialized in XML on the SB, then deserialized and eventually re-serialized
04:03
on the SOAP gateway, which then create a further envelope and transmits it abroad. So the issue is that there is no universal semantic for communicating errors. When does things go wrong?
04:20
Well, when you are on peak loads, because you cannot do service management, and you have a system that is overloaded with requests. For example, the city of Rome is roundly three million people. On the day, on the tax paying days,
04:41
you have nearly three to nine million requests only on the city of Rome. And imagine what can happen when you have a 60 million people country on tax paying days, then when people just remember on Friday morning
05:04
that they have to pay taxes, and they all make requests between 9 a.m. to 12 a.m. All charging the ecosystem. Those kind of issues, I mean, everyday issues,
05:25
and you mean you have millions of requests that should be serviced in four hours, and when this is going to break during peak loads, because processing error is costly. Okay, so you add further CPU and RAM work
05:48
even when there are peak loads. And moreover, all these became a barrier. This worked well for 13 years, but now is a barrier for the creation of new services.
06:03
It is very expensive, but on set up and an operational cost. It complicates communication with the private sector, because if you are a private company that want to interact with all the public health government staff, you need the SB and customs of gateway and so on.
06:25
Moreover, the IT world was moving toward Seoul. So okay, we now started to, to think if the IT world moved forward,
06:43
so why can we? Soap is old, but again, it was born because there were many weak points in the format HTTP specs, and it added essentially one layer
07:02
on the underlying protocol that is not necessarily HTTP. You can send SOAP messages on HTTP, on SMTP, whatever, and as SOAP is based on messages, it is virtually asynchronous.
07:21
But today, we have this brand new, nice, wonderful semantics that is RFC 70 to 30 to 39 actually. There are even a lot of work on HTTP now. Almost all services, not only web services,
07:42
but even mail services and directory services now are inherently based on HTTP. And now we found that the network is reliable enough for having synchronous exchanges. So it does not always sense to transport messages on SMTP.
08:04
It is fine, I mean, but it's not always a requirement because our system messages are always online. So we can go synchronous. We can have ASAM messaging. We can have chats and so on.
08:21
So we can actually move beyond SOAP. These new semantics allow us to route requests based on path and method. We don't necessarily have to process messaging to know how to balance them from idempotent
08:42
or non-idempotent that you can just read, like read only or read write or when I do have to cache or not. I can use a status and headers that is a sort of reading the payload, but it's not actually reading the whole body of all the records body.
09:02
So I can just limit to status and headers. Moreover, I got a lot of semantic for caching, conditioner, and even range requests. So this new framework wants to standardize API without SOAP.
09:22
That doesn't mean we just can't sell SOAP, but we expect new services will be provided through REST API. We want to enforce an API-first approach based on OpenAPI v3, that is the standards
09:44
formerly known as Zwoger, and we added a team standardization based on standards and a new availability strategy based on distributed circuit breaker and throttling patterns.
10:00
That means we enforce all API released by the government agencies to implement throttling and circuit breaker. And this is the shiny new ecosystem we want to implement where every agency implements API
10:23
that could be meshed up and we could create new and new services, mixing national registry and PHR, that is your personal health record, they're provided by certain trial government.
10:41
Together with services provided by schools and town, and the school can inquire to the revenue system or if you are eligible for grants, and the hospital can contact the town or your personal health record to see, for example,
11:04
if you are allergic or intolerant to something, and this is our actual goal, provide and mesh up services for the citizen because the main goal for the administration,
11:21
and this is sometimes when you speak with the administration, you have to repeat to them, your goal is to serve citizen. Usually administration think their goal is to create documents, and once they have provided documents, they're done.
11:41
But you have to remember it, and sometimes you'll find somebody that understands, and this will become a turning point because he will remember and he will be somebody that follows you
12:01
and understand the actual goal of the administration, and magically they understand the new ecosystem, and then they understand we can get rid of soap. So standardization, let's come on the tech stuff. Okay, only HTTPS, we banned HTTP and binary message.
12:24
That means if your administration has a new Kafka, JMS, AM could be shining cluster, simply they cannot expose it on the internet, but they have to provide an HTTP wrapper
12:42
that should provide authentication, authorization via mutual TLS, or open API, or open ID connect, or OAuth. This is not because Kafka is not good, but because you have to standardize things,
13:01
and Kafka actually is not a standard. Kafka allows you to implement a binary communication system, but while HTTP enables to write specs where you communicate in a standard way, so you can move from Kafka to AMQP or JMS,
13:22
and the other government agencies, just they doesn't need to know, and let me say, they don't have to know if you use Kafka or AMQP. They just have to know which is your interface with the external services.
13:41
And then we want today to leverage status, method, and path, because it is important not only to provide fast services, but we need auditing and routing, because usually when you are in the government staff, you need usually to treat people,
14:05
data, personal data, and so it's better to trade off some speed for the ability to have a unique and trustworthy system of encryption, and authentication, and authorization
14:22
based on well-known protocols. Then, this is easy. Just stop logging in that way, but just log in RFC 5424, that is, syslog, because so you don't have to care about New Year's Day,
14:44
you don't have to care about daily saving time, and if I have to cross-check logs from two administration, I don't have to pay external services or external company to write expensive log parser,
15:04
but I have to simply just send you the logs, and that's fine. Well, this ontology-based schema, what's that? Essentially, in the Italian law,
15:22
states that, except for some exceptions, public organization should release open source in the wild on GitHub for everybody to use their software but then one of the issue you have is that
15:43
their web service just serialized the given name in tens of possible different ways. While in Italy, we have an ontology that is a well-known and established schemas
16:01
that say that the given name is named, that is, given name and not name nor first name. The same for the tax card, the same for the what number, so we expect in a couple or two or three years to get rid of all those serialization variables
16:26
and to have a, converge to a simple and unique serialization stuff that will save a lot of time of unit testing when two organization communicate, because if I say, let's say,
16:43
the revenue service say, call given name and the health service calls first name, you have to provide a converter and you should test that all exchanges work. So this is a standardizing name,
17:02
it's probably the most complex part because everybody wants to retain their names but on the other parts, we seem to save a lot of money in two or three years and to eventually provide for public administration a set of library
17:21
for personal information management that could provide validation and so on at the central level, so if you have to implement or to manage personal information, you can just reuse the future to come library.
17:41
So another part, reliability, this was the main goal behind the new framework. We started from the European interoperability framework. If you are in the architecture world, it's a good read, it's actually a very good read. It doesn't go too much in the tech
18:01
but it goes too much in requirements and if you want to work in the government world, you have to motivate that the change you are introducing came from the European Union because this is your passport too for motivating administration to change
18:22
and actually I think it's a well-written document. So what does he say? He says that if you provide government services, you have to plan, you have to write up a business continuity plan. It means a few but it could mean a lot.
18:41
For example, if you provide your services to a public cloud, it means you have to implement multi-availability zone strategy or if you provide the service in your data center, it means that you have to provide a disaster recovery in two cities or at least in two different data centers.
19:06
So it means for us on the interoperability stuff and not on the architectural stuff that you should provide an integrated management on load and failures. That means if your infrastructure is overloaded,
19:22
you should communicate it in a machine-readable way to the others because this avoids cascading failures which are actually one of the main issues in governmental services, especially when you have pay infrastructure behind
19:44
because, for example, you want to pay a tax. You talk with your home banking. Your home banking will contact both the service payment system of the government
20:02
and the remote public agencies. So it's a three-level, it's a three-level architecture between different both public and private agencies.
20:24
It's something like that, but the payment service is too complex. So for example, if the tax office API is overloaded, you still contact your home banking
20:42
but your requests are going to go in timeout because the public infrastructure that is behind cannot service the request. And this puts a load on your home banking. They have actual transaction that are failing.
21:01
So you have, for example, a transaction ID on PayPal and then they tell you that the transaction number X failed. The new framework provides a way to, in the case the service is not available or if it is overloaded, okay, so if it's up and running
21:25
but its queue, its service queue, its API queue call is too long, you can use an API management that return a saturation response that is just plain 503, service unavailable
21:43
plus retry after, that could be retry after one hour, five minutes tomorrow. In this way, the payment service, your home banking can know that it doesn't have to still issue more payment services, but it can just say,
22:04
okay, this payment service is overloaded. I want issue new calls for five minutes, for 10 minutes. Or for example, if the service is a real time service that works only from 9 a.m. to 6 p.m.,
22:23
it will service requests all in a given timeframe but now the service C that relies on service B knows the time when it can start and issue new requests. And this is very important because it gives hints
22:43
on the organization one to communicate to the user that actually his request, his services is up but his request cannot be serviced in five minutes, one hour, 12 hours.
23:00
So how can you do this? Okay, you have to tell people to implement this but you haven't even communicate which are the HTTP headers because as you can see on the left, there are at least 12 different possible HTTP headers
23:21
that you can use to implement this. And if you make a standardization server application relying on external services can just check those three, that is, rate limit, limit remaining, and result. And just don't have to guess
23:42
and check all those possible headers. So the important thing is to reduce and be clear on which are the supported headers and on the last one, the late limit reset
24:00
tells you essentially if you are over quota when you can start again issuing new requests, just tell them how many seconds they have to wait because if you put a timestamp or a date, if you have to rely on your NTP server
24:24
or on their NTP server, while in this way, even if the clock is queue, they just wait for enough seconds. Moreover, this approach is semantically consistent
24:40
with another header that is actually in the HTTP standard, that is the retry-after header. So always tell if they are going to saturate their queues. Moreover, when they are over quota or the service is overloaded
25:01
or out of service, communicate how long before the new request has to arrive. Then this is wonderful, this is easy. There is a standard for JSON error response, just use it.
25:22
And then this was almost all. Then future steps, standardized metrics, use rates, not absolutes, because when you have to communicate service status, you probably want to communicate your metrics.
25:45
So which is the fragrance, which is the base Unix, which is your availability. The old framework used more than 20 different metrics, some was the higher the better, some other was rates, other was absolutes.
26:04
We say just use rates, not absolutes, use basic units, just like Prometheus, that is bytes, seconds. Use availability, expose success rates, not error rates.
26:22
For example, availability, the service was up for the 95% of the time or success rate, the service responded nicely to the 95% of the request. They must pick a response, expected response time.
26:44
And we are actually thinking if to use a responsiveness that is quite tricky metrics to evaluate or an aptX index that just use the target response time to create an index that is from zero to one,
27:03
and it's quite readable and quite usable. If you have suggestion on that, please, those are welcome. Then the hard part, the tough part, is signature and encryption.
27:21
Signing an exchange with a digital certificate is the basis for a non-repudiation framework. That is, if I send something to a public administration, I have a sort of receipt that can guarantee me that the counterpart cannot deny the exchange.
27:43
SOAP has a well-established standard, that is, well-established but criticized standard for signing an encryption that is WS security. And REST has some standard, still criticized, that is just on web signatures and encryption,
28:04
namely JWS and JWE, that are used by OpenID Connect, for example. We are investigating, still working on all that stuff.
28:21
The possible choices we have are, one, leave the signature and the encryption to the application protocol, for example, to the administration. Sign just the body, that is what WS security does, so we can just extend just an object with claims
28:45
or adding HTTP headers with a signature. Or another approach that is undergoing on Amazon, there are a couple of drafts,
29:00
is to create a signature of request header and body and put everything in the HTTP headers. There are some proposals and there are many issues, but it's an interesting research stuff.
29:20
Further discussions are on digital certificate. I'm lurking on this web work group. They consider RSA a legacy, but it's not that you say,
29:44
just don't do RSA. I mean, you should think well, because it's something we have to think at least with a 10-year perspective. The advantage of considering RSA a legacy
30:01
is that elliptic curves keys are very short and they are easily embeddable in HTTP headers or in claim. On others, there are something new in HTTP2,
30:20
are those structured headers. You can see the second stuff enclosed with stars is actually base 64 encoded binary. So the advantage of structured headers is that you can embed binary stuff into headers
30:44
in a standard way. So when you get stuff that is enclosed, it's just base 64 encoded. Well, another discussion, but it's really too much for a conference, whether to duplicate or adopt the digest header.
31:03
If you use it, please take care of reading carefully both the digest header RFC and all the proceedings of a couple of HTTP work group conference papers,
31:21
because it seems nice, but could be tricky. Well, I think that I made a quite nice run. I hope I didn't bore you enough. If you're interested on the new data framework, actually it is in Italian,
31:41
but I'm working with the GDS, that is the United Kingdom Digital Services. I started some preliminary talk on this work. There are some similarities, some differences,
32:01
but we just started last month, but we made a lot of calls to try to find some common grounds, and we even start some first interaction with the French government.
32:22
And so there is the opportunity to create really a European framework for API. There will be a European conference in October, I hope to make it there.
32:41
And if you're interested in this stuff, please let me know, and I will provide all the English documentation I can. And well, thank you very much. Again, you can write me on Twitter or on my institutional email.
33:06
Again. If there are questions? Please for questions, come up here. Any questions?
33:20
So thanks for the talk, it was very interesting. I'm glad to see that also Italy's kind of progressing on this, and I'm glad to see that there is some conversation with the UK government. I think they are quite good in digitalizing all the services. They have a very good GitHub account with a thousand repositories, so I think it's good.
33:42
So the question is, I have a couple of questions. So what are the first services that a citizen will use using these new technologies? If you have targeted some specific service. And the second one is all these work is open source.
34:04
What's the process of accepting pull requests or contribution from the community? Okay, actually the main services are related to data. For example, the queue, if you go to the ER
34:25
in the hospital, all those data are usually exposed by API, so you can know, for example, if you're in a big city, you may know which are the hospital with the longest queue or the lowest queue.
34:44
And a lot of work for the citizen is still based on data. Our first target instead are some
35:00
frequently used services provided between administrations that would be based on REST API, but with mutual authentication with TLS, essentially, because those are a backbone where local and smallest public agencies
35:29
like towns can build services on top. So the first goal to me is to unlock some services just like the tax code validation
35:43
that it seems simple, but it is not because, for example, a tax code of a dead person is not valid. So there is a service that provides a natural validation to check if the tax code is attached
36:00
to a valid person, a real-life person or a real company. And this is used billions of times, and it could be used, for example, to provide a real validation of all forms of the citizen provided by the citizen
36:21
that are usually, when you fill up a form and you specify your tax code, they usually validate just the mega syntactical validation. Or, for example, if you're driving licenses valid, those are core services that are very frequently used
36:41
that could be easily scaled, and if you move them with this approach, it will gain a lot of benefits for them. About the open source stuff, actually, we have almost 190 repositories
37:03
in the Italian government account, and we are working hard with the central administration because through the law, tells that they have to release in the open source,
37:23
there are no sanction for not doing that. So you have to check the project, you have to help them because releasing is an open source, it's not just throwing out whatever it is.
37:43
I mean, you should provide a quality software, you should provide software that could be documented enough for the organization to get contribution. But now there is one of the main project, it's a new project we are working on,
38:01
is messaging service for people, where through a mobile app, you could receive payment notification, you could receive fines, for example, or announcement on your healthcare data,
38:25
and every application is backed by an API service that can receive and deliver messages to the citizen on their phone,
38:40
and they can, through the application, register a wallet to the, in Italy we have a PAGO API that is sort of a peg of UK that started in almost four years ago.
39:01
The nice thing is that you can use this payment service to register your credit card and payment services to pay taxes and so on. And this app is fully open source, the infrastructure is fully open source,
39:20
you can get it and use it, it is very well documented, it's on GitHub, and the documentation is marked down, like mostly all the new Italian government project. So we have time for one more question. I just wanted to quickly mention,
39:41
there's a project called COSE, COSE, which is RFC 8152, and that's a signing and encryption protocol, which might be useful to you. That's one of the slides you mentioned. You were looking at different options for that, and I just wanted to mention that.
40:01
Just let me know. RC? RFC 8152. Yeah, it's based on CBOR, I think it's used as another way
40:25
of serializing the object. It's very interesting. There is one of the Senate exchanges that partly use CBOR and I think,
40:45
I don't know if actually all these, specification, because it seems, I will show you. This CBOR stuff is actually very interesting. And essentially, the CBOR is binary serialization
41:06
of JavaScript. Using the header notation I showed you before, the star one, you can just create and serialize a JavaScript object in a binary one and serialize and put it in a header in a standard way.
41:25
Well, thank you very much. It would be interesting to have a talk with you on that matter. So, and then, thank you very much for all your patience. I hope it was interesting enough,
41:43
even if it's a public project, but public and government can be good and fun. Thank you, Roberto. Thank you to you. Goodbye.