Successfully retrofitting extensibility into established software products
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 150 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/51433 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NDC Oslo 2013107 / 150
3
4
5
6
8
11
12
15
17
22
26
27
31
32
39
40
41
42
44
47
51
53
56
57
59
60
61
63
64
66
67
68
69
71
72
79
80
81
82
83
85
87
89
90
93
94
95
97
98
99
100
101
102
103
106
108
109
110
114
118
119
120
122
125
126
130
132
133
134
135
136
137
138
139
140
141
142
145
00:00
SoftwareProduct (business)Client (computing)Computing platformSoftware frameworkMultiplication signCodeValidity (statistics)CodeProduct (business)Electric generatorCategory of beingPhysical lawComputer simulationGroup actionLoop (music)Extension (kinesiology)MereologyMoment (mathematics)Physical systemHypothesisTerm (mathematics)Visualization (computer graphics)AreaSoftware maintenancePrototypeIndependence (probability theory)Process (computing)Thomas BayesMetropolitan area networkFood energyAbsolute valueMoving averageData storage deviceStability theoryTraffic reportingClient (computing)Imaginary numberEndliche ModelltheorieDomain nameComputing platformUniform resource locatorPlug-in (computing)ECosRight angleWeb 2.0VolumenvisualisierungPosition operatorSoftware developerSimulationSoftwareStructural loadOcean currentDirection (geometry).NET FrameworkMobile appComputer animation
06:22
Visual systemHost Identity ProtocolSoftwareComputing platformProduct (business)Software developerForceSoftware frameworkContent (media)Software development kitCodeInsertion lossComputer programmingWindowExtension (kinesiology)Physical systemOperator (mathematics)Data storage deviceCartesian coordinate systemRight angleSmartphoneMobile appCodeSoftwareNetwork topologyRewritingProduct (business)Level (video gaming)Fiber bundleAxiom of choiceContent (media)MereologyMoment (mathematics)SubsetTerm (mathematics)Visualization (computer graphics)NumberMenu (computing)Software maintenanceRevision controlGoodness of fitMassProcess (computing)Strategy gameAdditionPoint (geometry)Chemical equationBoilerplate (text)Software frameworkGraph coloringSound effectClient (computing)Domain nameComputing platformPlug-in (computing)Service (economics)Operating systemAssociative propertySoftware developerUsabilityDampingForm (programming)Server (computing)PlanningSocial classCuboidCrash (computing)Insertion lossStability theoryFrame problemGreatest elementWritingComputer animation
15:17
Product (business)Insertion lossOperations researchCodeSoftware maintenanceConsistencyData managementArchitectureProcess (computing)BuildingSelf-organizationSingle-precision floating-point formatRevision controlPersonal digital assistantClient (computing)Mechanism designDisintegrationInformation securityAttribute grammarCodeImplementationComputer architectureSelf-organizationExpert systemProduct (business)Category of beingFlow separationDecision theoryGroup actionExtension (kinesiology)MereologyTerm (mathematics)ConsistencySoftware maintenanceCovering spaceCASE <Informatik>Process (computing)Spectrum (functional analysis)Point (geometry)Poisson-KlammerCartesian coordinate systemSoftware frameworkSurfaceDifferent (Kate Ryan album)Block (periodic table)Computing platformSingle-precision floating-point formatRule of inferencePlug-in (computing)Optical disc driveSpacetimeService (economics)Mechanism designSoftware developerMultilaterationPhysical systemRevision controlServer (computing)PlanningStrategy gameShared memorySocial classData storage deviceObservational studyStudent's t-testStability theoryReflection (mathematics)Greatest elementComputer fileECosCodecComputer animation
23:37
DisintegrationTelecommunicationInformation securityArchitectureTime domainLengthCodeVideo gameSoftwareTelecommunicationSemiconductor memoryType theoryProduct (business)Category of beingFlow separationExistential quantificationMereologyDevice driverVisualization (computer graphics)Data modelSystem callConfiguration spaceInternetworkingProcess (computing)Instance (computer science)Information securityPointer (computer programming)Point (geometry)Social classReading (process)Cartesian coordinate systemWeb browserChemical equationReflection (mathematics)Software frameworkSurfacePlastikkarteBuffer solutionComputer fileClient (computing)Endliche ModelltheorieDomain name.NET FrameworkComputing platformLoginCycle (graph theory)LengthProxy serverMultiplication signBoundary value problemVulnerability (computing)Plug-in (computing)VolumenvisualisierungDemoscenePosition operatorVolume (thermodynamics)Mechanism designMobile appData structureComputer architectureUser interfaceCausalityTotal S.A.Structural loadExtension (kinesiology)Slide ruleSampling (statistics)NumberScaling (geometry)Revision controlFamilyServer (computing)PlanningMassCASE <Informatik>Interactive televisionStability theoryGreatest elementScalabilityBuffer overflowUniform resource locator2 (number)SpacetimeRight angleFigurate numberPattern languageComputer animation
31:57
Information securityTime domainLengthArchitectureRevision controlRule of inferenceCodeFlow separationShared memoryNetwork topologyExpert systemElectronic program guideConsistencyCodeData managementTelecommunicationNetwork topologyExpert systemBuildingProduct (business)Flow separationConstraint (mathematics)BitData loggerMultilaterationPower (physics)Extension (kinesiology)MereologyProjective planeTable (information)Visualization (computer graphics)ConsistencyNumberData modelQuicksortConfiguration spaceMachine visionRevision controlDivisorServer (computing)Process (computing)NamespaceGastropod shellAdditionPoint (geometry)Social classControl flowCuboidSet (mathematics)Cartesian coordinate systemShape (magazine)Film editingSoftware frameworkGreatest elementClient (computing)Different (Kate Ryan album)Domain nameComputing platformMultiplication signPlug-in (computing)Business objectVirtualizationSoftware developerMathematicsStaff (military)PlotterDimensional analysisArithmetic meanDivision (mathematics)Forcing (mathematics)Physical systemMilitary baseGoodness of fitPlanningBasis <Mathematik>CASE <Informatik>Presentation of a groupShared memoryInsertion lossFrame problemComputer fileView (database)Object (grammar)Electronic program guideSpacetimeRight angleMobile WebComputer animation
40:18
Software frameworkExpert systemElectronic program guideConsistencyChi-squared distributionWeightAutomationSoftware testingBinary fileStability theorySource codeCodeCompilerLinear regressionExecution unitPersonal digital assistantFault-tolerant systemException handlingProcess (computing)CodeData managementMathematicsFunction (mathematics)Fault-tolerant systemProduct (business)Level (video gaming)Perturbation theoryINTEGRALSoftware testingBinary codeBitSheaf (mathematics)Linear regressionExtension (kinesiology)MereologyExecution unitVisualization (computer graphics)ConsistencyAngleAreaSystem callRevision controlException handlingCASE <Informatik>Process (computing)AdditionPoint (geometry)Poisson-KlammerCartesian coordinate systemStability theorySoftware frameworkSource codeRegulärer Ausdruck <Textverarbeitung>Patch (Unix).NET FrameworkUnit testingComputing platformMultiplication signRule of inferencePlug-in (computing)Overhead (computing)Figurate numberSoftware developerSelf-organizationComputer programmingBuildingForm (programming)Group actionMoment (mathematics)QuicksortSoftware maintenanceMachine visionGoodness of fitServer (computing)PlanningCompilation albumCAN busInsertion lossGreatest elementSign (mathematics)Client (computing)Website40 (number)Different (Kate Ryan album)Uniform resource locatorRight angleComputer animation
48:38
Software testingProcess (computing)BuildingClient (computing)Software developerBeta functionComputer to plateStability theoryInternet forumVisual systemSample (statistics)Data managementArchitectureProduct (business)SoftwareCodeVideo gameSequenceComputer architectureFeedbackProduct (business)Electric generatorIntegrated development environmentBitGroup actionLetterpress printingExtension (kinesiology)Projective planeSampling (statistics)Visualization (computer graphics)ConsistencyNumberConfiguration spaceGoodness of fitProcess (computing)Boss CorporationRoundness (object)Direction (geometry)Electronic mailing listStability theoryChemical equationAlpha (investment)Client (computing)Coefficient of determinationDifferent (Kate Ryan album)Representation (politics)Multiplication signBuffer overflowPlug-in (computing)Stack (abstract data type)Right angleFigurate numberAndroid (robot)Design by contractSoftware developerMobile appAttribute grammarComputer clusterSelf-organizationSpeech synthesisDressing (medical)Level (video gaming)Category of beingReal-time operating systemMereologyPhysical systemMachine visionPlanningUltraviolet photoelectron spectroscopyField (computer science)ExistencePoint (geometry)Social classControl flowData storage deviceGreatest elementSign (mathematics)Computer fileBit rateECosUltimatum gameComputer animation
56:58
Product (business)ArchitectureSoftwareComputer animation
Transcript: English(auto-generated)
00:01
Should we start? All right, thank you. You're welcome. After lunch, again, who missed that? I intend to make it interactive. So ask questions, make me pause, and if we run out of time, I'll tell you and we'll run till the end. So my name first, Jan.
00:20
I work for Schlumberger. You will find a booth with the blue and orange balloons there in the lobby, in the hall. My role is lead architect for extensibility in the company. In the past experience, I did work for this company actually all my working career. Great employer in great places where we do cool software development from Paris to Oslo nowadays for me,
00:46
and I've been working on the web and desktop, Java, .NET nowadays. So in that current role I'm in today, I am taking lessons learned from one of the biggest software product we have, and it's a big software product in the oil and gas domain.
01:02
And we did actually retrofit extensibility after the product was already successful, and we are now taking those lessons learned to other products in-house, and I want to share some of that to you because I think it's very generally applicable. So there we go, why extensibility?
01:22
I already asked the question, but for me, there's two ways how to solve that. The first one to your business owners or product analysts or basically business-minded people, because they think of your software most likely as a platform that delivers on some scenarios, workflows of your clients.
01:45
And imagine the top one is yours, and you have some steps your clients like to perform with your software. Step one, typically load in data. Step two, process it. Step three, validate it. Step four, export it or report on it.
02:00
And oops, for your client, there is no step three, there is no validation. They would like to do validation, or you have validation, they would like to do it in a different way. So they find another product, your competitor, and they ship the data over, and they don't like it because it's copy and paste or export import and always some manipulation you have to do by hand.
02:21
Nobody likes that. And oops, your competitor happens to have the reporting as well, perhaps. So then your clients will not really be willing to come back and do all the copy and paste back again. So you are already getting slowly out of business, and you are out of business once your competitor completes the workflow before you do. Perhaps they are better positioned to do that, and you just lost a client.
02:45
So your incomplete workflows is what typically takes your clients to your competitors, and they may not have the energy to come back. The other way how to sell extensibility for more of the technical people who know what the technical data is and they intend to manage it, if you think of your product from the day one,
03:05
you start perhaps with two developers in the garage, and it grows, and you make first release there. Then you keep going, and you make releases every year, and it keeps growing, and it's a success. And somewhere along the way perhaps you get acquired by something bigger.
03:20
And then towards the end, this is an imaginary product I'm talking about, but you may face a decline in your productivity. This might be because your maintenance dollars are growing. You did leave some technical debt back in the code base, and now it's hurting. Now it's too late to go back and re-architect, and it's slowing you down.
03:42
You can't be adding features that fast anymore as you were in the beginning. So perhaps you had a bright moment, and somewhere in the process of growth, you did establish your extensibility. You built, on top of what you had, APIs that were meant to decrease your technical debt,
04:05
decrease the tight coupling between your subsystems, make your subsystem work through API that has more stability to it than the general code in your code base. And if you did that, well, surely sooner or later, your other colleagues in your company would like to add some more to your product. Now they can. Now you have APIs.
04:23
So you will start duck-fooding most likely those APIs with your coworkers and groups that you work with in your company. And then if that works, then your clients are going to like that as well. And this is one of the big lessons learned on the business side. In oil and gas, which is the domain we are in,
04:42
our clients don't like to share their intellectual property much, not even with the software vendors like us. They really like to keep it, and they really then like to implement the plugins with their custom algorithms, custom simulation software, custom visualization, whatever is it, but they love that they can do that themselves.
05:04
What we end up with is that perhaps we even sell more copies of the basic product. And then if you are even more successful and your ecosystem is growing, you will have independence of the vendors coming and picking up your APIs and being part of the ecosystem, delivering plugins to your app store,
05:24
and research brings in cutting-edge research science, and you shorten the loop between when the science is available on somebody's test machine and between the moment when somebody can ship a commercial plugin with something bright and new
05:43
because they started the prototyping on your platform directly. It's much easier and much faster to commercialize research work when it was done already, not in, say, Matlab, but it was done inside your product. And then academia, of course, if your software becomes used in classrooms
06:02
or in master thesis, then, of course, you are growing the next generation of your clients, customers, users. All that is good. All these are the things why we do extensibility. Then let me refine the term extensibility as well.
06:23
Any of you have a smartphone? What do you think of that as being an extensible system? Of course, you get apps from the app store. The apps are written, if you wrote an app for something like that, are written against some APIs, but so is Windows programs.
06:42
They work against Win32 and WinRT libraries, but for the purpose of this talk, I'm leaving that out of the room. Sorry for your smartphones. And I want to focus on the rich extensibility of applications that work together in a very deep fashion.
07:08
So this is a map of Visual Studio extensibility. A lot of entry points, a lot of extensibility points for application developers and plugin writers. You can plug into the IDE, to the team server, to whatever you want.
07:22
Very rich, very broad, and your plugins can actually implement any possible subset of it and communicate with what's already there. So I'm assuming you all know Visual Studio. That's why I've put this on, and thanks to Microsoft for this picture. But for the oil and gas domain where we are in, let me show one screenshot here of the product we ship and love.
07:44
And the extensibility for us is similarly rich, and you can bring in your custom data items into the data tree. In here, this is a data tree. You can have your custom menu items and ribbon bar buttons and toolbars
08:01
and extend the workflow that way, or you can add custom processes to enrich the workflow with additional workflow steps which can be automated. You can get your data visualized on this fancy 3D canvas, or you can implement your own visualization canvas, bring in a custom window into the picture.
08:22
And we let you do all that, and we let you do bundle of plugins, and you can have them richly interacting with each other. And we do have, in the effect of that, plugin writers that offer services to other plugins because it's all playing together, and it's in the interest of the end user, the client who's using this application.
08:45
And just like you are sitting here, you wouldn't be able to recognize what came from a plugin or what we shipped in a box, which is really the purpose, the first-class citizenship. And I did mention the term plugin, sorry, platform. I want to disambiguate that as well.
09:01
What do we mean by a platform? What do you mean by a platform? If somebody says platform, what's the association in a software sense? How do you distinguish something being a platform, not just a product or application? Operating system is a platform for you, okay? And you could call it a framework as well,
09:24
if you can call it a platform, so it has a development framework then. I'll try my two ways to put that, and this is thanks to a couple of my very smart coworkers. One of them said, platform is a successful product,
09:43
which is, at the same time, relevant and attractive to both end users and developers. So relevant and attractive to both end users and developers. There's a lot of conflict built into this sentence, and it's very hard to deliver on that, on maintaining the balance.
10:01
What it means in practice is that your developers would say, I really like that. It's simple to learn, it does what I need, and it removes a lot of the other boilerplate coding I would have to do if I did this from scratch, say on top of just the operating system or on top of just Excel, which is a general purpose, very extensively too by itself,
10:21
but there is nothing in it for your particular domain perhaps. So the platform is then very relevant to these developers who have some code to port and enrich the workflows. For the end users, of course, it then becomes very usable,
10:41
and it does what they need. In plain terms, that's what we mean by being business relevant. And you can be very good, you can have zero technical debt, but if you're not relevant to your business, you're not going to make money, full stop. Second definition, it is a product,
11:02
platform is a product where 80% of the content, say a number of the buttons and UI features and features in general, comes via the extensibility framework. So in the mental picture of that, I have this bar where roughly the 20% is the platform, is the shrink trap product,
11:20
including your extensibility framework in it, and there is a mass coming through the API. So if I join that with the picture I was showing just a minute ago, all that, that's orange that corresponds to the 80 or 20%, and everything in the colors are the plugins.
11:42
And you want to have most of the content coming via the SDK, that's when you become a successful platform. Why 80-20? It's obviously because everybody uses 80-20. But also let's not think about it as a number to achieve, so the arbitrary choice here, but it is something to keep in mind,
12:03
that there is always going to be the 20. It would be foolish to think that you can move with your extensibility framework the coverage of your APIs in other terms, as fast as the product evolves. In fact, if you did, perhaps that would slow down the agility of the main product of these 20% in here,
12:22
and you don't want to do that either. So keep in mind that the product grows, just like in the picture on the corner, and allow for the additional goodness to come.
12:40
As you do that, what do you think can go wrong? And if you've done it, this is the moment when I really hope to hear some war stories that we can all learn from. What is it that you should keep in mind from day one, or what is it that likely you can encounter as a setback?
13:01
Any ideas? Yeah, definitely.
13:31
So the comment is that what can go wrong is that if you don't design part of your API really well, and it gets used, it is very expensive to change it,
13:42
not only for you, but for all the consumers, and it costs you your reputation. The perception of the framework being stable is gone if you make that kind of a mistake. So dogfooding is a good mitigation strategy for that. I agree. Anything else? Any other idea you had?
14:01
It's actually the same. It's the same, okay. With your plugins not running on more than just a single build of your product, kind of, or which way?
14:24
Okay, so when you upgrade the main product, the plugins are worthless? They just plain don't run anymore? Or there's too much maintenance to make them run again? You need to rewrite something? Yeah, definitely.
14:41
All right, any other? Okay, yes. Have you all seen my slides? Because the first one that I came up with was the loss of agility,
15:02
and I was alluding to it when I was showing you the growth of the typical product that becomes extensible along the way. You can't afford that. The plugins may be less stable than the product itself, and if a plugin crashes the product,
15:22
what does it look like to the end user? They don't care what crashed it, right? They'll still call you. You'll have to pick up the phone, and you'll have to take the blame, and it may take you a week before you realize it was not your code that crashed it, right? Plugins can do conflicting things to share things. One plugin can undo something.
15:40
Some other plugin has worked hard to make. The API can change too quickly, and the maintenance cost just plainly may be too high for some of the ecosystem participants to still be within the ecosystem, and they will step out, and they'll take their plugins off your store, and you'll have less on your store. The API consistency may not really happen
16:02
if you didn't establish your rules early, perhaps, right? And if you don't cover sufficient surface of your application scope with your APIs, then your extensibility framework is not really relevant for many workflows, and the plugin vendors
16:22
will only have very limited niche to contribute to. They will not be able to bring additional broad, spanning workflows that we, at least in the industry we are in, we are really after the end-to-end workflows. Compatibility. Plugins do not run or do not compile anymore.
16:42
And there we go. For that problem space defined thus far, here's my six lessons learned, how to cope with them, what's the mitigation strategies, what to do, what not to do. Let's go through them.
17:01
Manage your scope and prioritize. So I'm sorry this is an architecture talk, as it was labeled, but I'll have to start from the very beginning. If we don't start there, then the architects are not going to be able to do their job anyway. So one of the lessons learned is to be clearly transparent to your community
17:21
and know your entire community. If you have plugin writers recruited from research, as well as high school students, as well as Uber expert developers, then that's a very broad spectrum to cater to. Then you will have to prioritize your parts of the community
17:43
that you cater to first and what's not that important to you. But you will have to know it. You will have to know whom are you writing the APIs for. Be very transparent about why you're not doing certain things yet and share back the roadmap. When you say we'll do these things now,
18:02
next year we'll do those and a year after that, that's how you make your planning, how you spend your resources, how you invest into the technology. Keep in mind that, obviously, your API consumers need to do exactly the same thing. They need to plan their investments into your technology
18:23
and you want them to be committed to your technology. So you have to provide the openness and be transparent about what you're going to do when and stick to it. Here it gets a little more architectural. One of the lessons that did work for us
18:43
was that the extensibility framework was put into the same code base. It wasn't kept totally isolated on the side, like let's see if that works out. It was in the code base, but yet still separated to separate organization. First of all, when you are prioritizing your requests
19:00
for features and scenarios and use cases, it's probably a good idea in the first phase, at least initially, to prioritize your UI features and the visible things for the end user, actual end user, hands-on user,
19:21
separately from your API use cases. If you put it all into one bag and let a single person decide what's important, the odds are that that person is going to value the most important things for today or tomorrow or the next year, and that's typically buttons. Still, the end user's buying the product.
19:44
They need buttons. They need the features. Otherwise, they can't do things, and it's kind of thinking step ahead to establish the extensibility so that anybody else can put in the buttons. It's kind of obvious when you think about it this way, but day to day, it happens.
20:01
It's very hard to prioritize for a typical business owner, product owner in the agile sense. So, positively discriminate your extensibility. The extensibility is the new kid on the block. You need to make sure it happens.
20:21
Suddenly, basically, you prioritized your architectural quality attributes of your product differently. You bumped up the priority of extensibility, and it skips over stability, skips over speed, perhaps, and you have to put your money where your mouth is. You have to find ways in your organization to positively discriminate the new thing, the extensibility.
20:44
And you can do that easily by avoiding the conflict of interest. If your organization is broad, if the product is already big, then your product owners, you can really split the ownership of these people, manage the extensibility backlog, and these people manage the UI features for the end user.
21:04
I'm putting there in brackets initially. It's going to work for first phase. Depends on your product how long that will be, but at some point, you will want to merge it back because you will realize that the platform keeps evolving,
21:21
those 20% and off where you want to be. You're by far not at 20% when you begin. You're at 100% native platform and nothing coming through the API, and you want to bite into that and kind of reverse the ratio there.
21:42
So at some point, you're going to come to a situation where you will realize, I can't keep up with the native development. If I have a separate group of people doing the APIs for those new features in the platform, they have to learn it, they have to wait until it's finished and commercial
22:00
before they can put in the API. So your APIs might be one release behind your main product. Not good in the long term. Join it back then. Last point, be diligently, not API request driven, but scenarios driven.
22:20
Make your stakeholders formulate use cases, scenarios what they do and what they want to achieve because most of the asks are, if we only had this property on this class, make that public. I'm sure you have it, just make it public. Or if you only provided this service to give me access to that, if you don't ask why,
22:42
you're going to miss the important bits because for many of these requests, the answer has actually been, oh yeah, but you can actually do it this way, and you already have the API for it when you look at that entry point over there. Or you can implement it, of course, and then they'll realize, no, but that's not really what we meant by it.
23:02
We really needed to achieve that, and then you know, and then it's too late. Of course, again, once you put the APIs down, it's very costly to change your mind later. Second, now I'm stepping into design and architecture, the vital mechanisms.
23:20
This is what you'll probably find on textbooks or Wikipedia or wherever you go to see a definition of extensibility. These mechanisms you typically need to figure out very early or up front, so they are very architectural. Discovery, how is the product going to discover the plugins at startup? You're going to just throw them into a folder and run the reflection over all the DLLs
23:41
to find out if they have plugins inside, or you're going to register them in some kind of a file or the .NET config file and read through there and locate those DLLs and load them to the main process. Whatever fits, it's typically what you have to figure out very early. Soon after, you'll have to make sure those plugins
24:01
are packaged and deployable as well in a commercial sense. XCopy doesn't really work for your end users. First of all, they don't really do XCopy. And second, if something goes wrong, it's not really diagnosable. Then when it's deployed, the life cycle kicks in. When you start up your product,
24:21
the plugins need to be located, loaded, integrated into the platform, find the entry points in the platform, get the instance of this and instance of that, integrate into the user interface, so put their buttons wherever those are supposed to appear, and perhaps even integrate between each other.
24:43
As I said, we did see many plugins offering APIs to each other, basically plugins having compile time dependency. If that's a driver for you, you've got to allow that in your life cycle mechanism to have the plugins make a handshake between each other.
25:02
Of course, that digs into separation and isolation, but I'll leave that for next slide. And at some point, the plugins should be unloaded and the application saved to complete the cycle. Isolation and security would be a full slide, and communication between plugins is what you definitely have to figure out early.
25:21
Communication between plugins and the platform and plugins each other. The chattiness, the kind of data structure, the data volumes that go between the boundaries of the plugins and the platform, that's going to most likely hint how you should architect the extensibility, typically how you should isolate your plugins.
25:43
I'm adding a few more that we discovered on the way, and totally you said that, backward compatibility, forward compatibility of plugins, and versioning, right? Absolutely, right? If you don't define those, then you're just going to get surprises. Your end users are going to get surprises.
26:01
You don't want to make a deployment mess for your clients because you made your product extensible. That was not the purpose. Resource governance. If you are successful, you get a lot of plugins. Let's hope that happens, or was happening for you. It is probably going to come to a point where
26:20
the product becomes sluggish, and it will be good to know who's to blame, right? Some plugins consume an enormous amount of memory. Some plugins may hold onto the CPU and do too many things on the UI thread, and some plugins may abuse the GPU, the GPGPU, and kind of slow down your visualization perhaps
26:42
because they do calculations. At some point, you're going to have to do things you know as an end user. If you run a browser like an Internet Explorer has from time to time at the bottom, the orange bar, saying a plugin is running too slow, do you mind disabling it? Because we can start up much faster without that plugin.
27:01
And it takes you to a place where you can see how much each plugin has cost you in the startup time, and you can disable those that are offending. Diagnoseability and testability make not only your code, but the plugin code testable and diagnosable when something's going wrong during deployment.
27:22
Have logs for that, that you can tell people, go and fetch that log and send it to me, then I can tell you what's going wrong. Otherwise, it's endless hours on the support line. Isolation. I guess this is the most crucial point, most architectural, the one that if you set it up one way,
27:43
it's most likely going to stay that way for the entire duration of your product life cycle. Excuse me. What you should think about, it's going hand in hand with security. Security in the sense of plugins misbehaving
28:04
and either exploiting vulnerabilities or opening up your platform for more vulnerabilities to be exploited by somebody else. Or taking client data and shipping it over in the network somewhere else, as they are not supposed to.
28:20
Already mentioned the plugin-to-plugin interactions, the communication patterns, the data volumes, the chat in S, the non-functional requirements, going down the performance and scalability, it's right there, and the breadth of your API coverage. If you take an example of application that was designed
28:43
in such a way that the whole data model was decomposed to just data entities and property bags and relationships, like entity framework thinking, then the entire API to get any part of the data can be done in maybe five methods, one class with five methods,
29:01
and you can get access to anything in the data model. But that typically gets you to late-bound code in your application, and that's not very pleasant either. So most products have rather rich APIs, specifically having concrete types for concrete things,
29:24
not generally handling everything as an entity. And if that's the case, then you know that you are going to have to instrument a lot broader API surface when you go across the isolation boundary in practice.
29:41
So consider one product that basically has APIs to access the visible things and the non-visible things. I'm really dumbing things down for the purpose of the mental exercise here. And you have plug-ins plugged in, loaded into the process, into the same memory space, limited or no isolation.
30:06
What's that going to do good? Which of the quality attributes is going to actually make it shine? Any positives about this plug-in model? Very interactive, right?
30:21
You could have the end users spin numbers, or if you have the 3D scene like we have, then it's going to flow, and all the renders are going to be rendering because they're right there. They're loaded to the same memory space. They have access to all the data instantaneously. Do they even hold a pointer to it, right? Where is it going to be not that advantageous?
30:46
When it crashes. When any plug-in has a buffer overflow, let's say, it crashes your application, and it makes it security vulnerable at the same time. You don't want neither of that. Strike your balance.
31:00
Another model where you can afford the isolation vis-a-vis your performance and scalability, for example, and all the questions on the left. You can put in a plug-in host process, or in .NET you could use app domains and restrict the privileges in the app domain
31:20
to isolate your plug-ins to just give them the least privilege, and handle all the communication across. So when the plug-in starts up in the plug-in host, it's running on top of some kind of an API proxy that takes all the calls and makes the magic behind the scenes to do the RPC call or the cross-up domain marshaling
31:42
to the main host. Well, that's going to cost, right? You have to engineer some smartness into that to make it performant again, but it's totally possible. Third model, if you happen to have client-server, two-tier, three-tier application architecture, then your plug-ins may actually need to be
32:01
both on the client and the server, right? If it's a desktop application that talks to a backend, then you may have UI kind of extensions installed in to the client, as well as configuration or additional parts of the data model on the server. And you're going to have to sort out the things,
32:20
user versioning. The versioning between the plug-ins on all the clients connecting to the same server, right, are all the clients, have all the clients been upgraded to the latest version of the plug-in so that the server works for everybody. Quite a few things to think about. That's why this is very architectural, very difficult to change too late.
32:42
A few other things to remember. You still strive for first-class citizenship. So from the end user's point of view, they are not supposed to know which buttons or feature came from a plug-in and which came with your shrink wrap product to start with. Same for, say, custom domain objects. If you have pieces of data that can play a role
33:03
in the platform in a generic sense, you should not make it constraints for the custom API-based custom pieces of data that they do not have some privilege, that they cannot visualize or they have to do additional jumping through hoops to achieve the same thing
33:20
that the native parts have. None of that. Strive to have the first-class citizenship for all of these custom things. That's the only way to make it work. Compatibility rules, again. And if you are out of process, isolation, then you can even have plug-ins running 32-bit
33:40
while your main process is 64-bit or vice versa. We see this with the number of third parties that are not available in 64 or 32-bit. Well, you can just build your plug-ins for where you have the third party and sort it out in your communication between the main host and the plug-in host.
34:05
Kind of look behind the corner, right? Not only mind your own code, but just like you want to separate concerns in your code, make sure that you don't really make your API in such a way that separation of concerns in the plug-in code would be impossible.
34:21
And it's kind of easy and seductive, and it happens that the APIs will guide you. As a developer, you will have no other choice than implement one class that does persistence, presentation, and all the other concerns, all in one class, because the API, the SDK just thought it might be convenient, right? But yeah, you grow your code in your plug-in,
34:42
and then you're stuck with a tiny, small class. I mentioned diagnosability and log files. Mental picture that I would like you to walk away with, perhaps, is that you should design for success and design for growth. So even when you start with the two people in the garage, you probably have a vision
35:02
to where you want to take your product. So I'm putting up a picture here which has kind of an L blue shape of the bits of the framework that are common, and perhaps the application shell and the UI, the visible pieces of the UI are surfacing all the way up to the user or the plug-ins, and then there is a shared part of the L shape at the bottom.
35:24
That's your framework in general, right? And then you have your subsystems or domains. In our case, this can be drilling, reservoir engineering, geology, you name it.
35:40
It's a good idea to think about it this way and avoid these peer-to-peer couplings between these subdomains. Rather, spot these and harvest down the API, make the internal API better and usable for everybody by putting down the coupling through the shared layer,
36:01
and it cleans up your build dependencies and cuts some of your technical debt. Plus, in addition, that API you just moved down there, you can name that public, right? If you did the right job with it, then perhaps that's your perfect example for making the same API available for the plug-ins. That's one aspect of this picture.
36:22
The other is that once you kind of divide it and you are ready to conquer, each of these domains perhaps have a sub-architect, and they can take the same mental picture. So this, in fact, is a fractal, right? Each of these domains has an L shape of their own local framework and their own sub-subdomains,
36:41
and they divide and they conquer. And again, the couplings between, say, these two boxes in here should go through this blue one here, should not go directly, and there is no reason to have the API for that down there.
37:03
That's too much, too visible. Why this is designing for growth and why this is important is that this goes hand in hand with how do you segregate your code to DLLs, and even worse, how do you divide your namespaces, basically.
37:24
Adding a different reference to your Visual Studio project is not that bad, but changing your namespaces all around in your code and in all the plug-ins, that's a huge cost. And as a developer, you know that changing namespaces is nothing easy or simple.
37:41
Many ways how you can make it wrong. So if you are set up this way mentally, then your bottom namespace, say, bearing the name of your company or your product, then has the subnamespaces in the subdomains, plus perhaps the big bar on the left can be your .ui subnamespace or something like that. And then further, this breaks into the subnamespaces,
38:02
all of these subdomains then govern. So if you are in this mental picture from day one, it can avoid you the huge cost of changing your mind later. So think of your namespaces in the tree as to where you want to take it.
38:20
Now you've started writing code. It's architected. Now we have developers drafting APIs and reviewing each other's APIs, I hope. And as they do, they may sit around the table like that, and one of them may argue about how ergonomic and easy to find and easy to learn
38:41
and easy to use the API must be. The other one saying, but that's not consistent. We've done similar things there, and it looks completely different. Nobody's going to be able to apply the learnings from there in here. And there will be people arguing about structs versus classes and public virtual methods and sealed classes and serializable classes in the framework.
39:04
And it's a good idea to kind of brute-force that and make them agree, make them write up a document so that it can be enforced. Even when some of these people no longer work for you, or they moved on to other roles and other jobs and they are managers now,
39:22
the knowledge needs to live on for the sake of the consistency, first of all. I already mentioned that, and this is how it then gets used in practice, that you need to know your audience, and you need to cater to all of them. So you need to guide the novice to use your API the right way,
39:44
and you need to empower your expert at the same time. If that takes two different sets of APIs that are easy to find for the novice, but the expert can find the advanced API somewhere and achieve much better performance or greater things
40:01
when they understand the power of the tools that they are given. You should do that. Last, but that's already been said so many times, Scott Myers definitely yesterday said that, prefer consistency to coolness. I'm out of my job now.
40:25
Last thing here, there is a book actually in the bookstore that I definitely recommend all of you to read, especially if you are on the .NET platform. We made it mandatory for all developers and managers.
40:41
That's the only way how managers can appreciate, and the business owners can appreciate, why does it take more to develop good APIs? Why does it really take more time to deliver the extensibility comparing to delivering the actual buttons or the features of the application? There is actually a fair amount of overhead.
41:01
The overhead around 40%, 50% goes to delivering good APIs. So it's good to have a good start by picking a book like that as your starting point, and then customizing the learnings from there to your own platform.
41:27
Quality is your reputation. Who do you want to be compared to? If you were in sales for a moment, or marketing actually, and you had your framework used out there,
41:41
what kind of thing do you want them to think about when they think of your framework? What other frameworks you want to compare yourself to? If you are on the .NET side, do you want to compare yourself to the .NET framework itself, and not the .NET 1.1 that had a huge leap to .NET 2? You don't want to be compared to that,
42:00
but do you want to be compared to the .NET 3.5 to 4 stability, level of documentation, quality? So for all the things that you do, there is a lot of quality in it, and you don't have to go too far, and this is usually not considered quality, but your documentation is a big part of your code base,
42:21
or a big part of your product, and you can totally help yourself by installing some additional tools into Visual Studio to get spellchecker on what's going to end up in the developer documentation, in the CHM, through St. Castle perhaps. Spellchecking, consistency checking, run regular expressions on the outputs
42:41
to see that you are calling the same things by the same names, or when your marking piece of API is deprecated, and you want every developer to write the deprecation the same way, so it's understood the same way, you may want to write a regular expression to find that everybody is following the rule that this method has been obsolete in 2013 version of this product,
43:05
and it will be physically removed in 2015. Then it's crystal clear for every consumer what to expect. If you have a rule like that, and I recommend you do, test it. This is totally automatable, and you should. There's code snippets in the documentation, the code angle bracket section in the XML of the summary.
43:23
You can actually take that out surgically while you're building, paste it into a mini program, and compile. And if it doesn't compile anymore, it means that your API changed and your documentation didn't. It's very easy to refactor something and leave your documentation behind,
43:40
and then the poor API consumer is going to be looking at that and not believing. These are easy API changes, but what about API behavior? When your framework starts behaving differently, well, that can crash the plugin just as well
44:02
as when you renamed the method and didn't tell anybody. So what you can do about that, though, is you totally automate a regression suite, but not in the sense of isolated unit tests. That's not going to help you here. Totally encouraging.
44:20
I guess I would be crucified if I said something against unit testing. That's encouraged, but that's not going to help you all the way here. You really need to have something worth of integration tests where you have pseudo-plugins or plugins running on top of the entire stack. You're going to load your product. You're going to run the plugin in an automated way,
44:42
make it do things, and then make those calls go all the way down through your product, through all the layers. You should not isolate anything in that sense. It's not going to give you a test for behavior changes. That's the only way to detect behavior changes.
45:02
Test your supported use cases as well as non-supported. Make sure that the exceptions that you said you throw in case something is passed in, you do throw, and that's totally testable. Graceful failures. Test your source code and binary code compatibility. Now I'm biting into something that's very big
45:21
in the area of extensibility, frameworks, development, the stability promise. You have to tell your consumer what to expect, and that goes into the compatibility for most part. You're going to tell them that, for example, and what we do is we tell them, your plugins, once they are compiled against
45:43
this major version of this product, they're going to run on all the minor updates and patches and hot fixes or whatever you call it. We guarantee that the plugin will work. And that goes down to we must not change or remove any APIs or change behavior. We can add, that's not going to hurt anything.
46:02
Or we can't remove or change. Source code compatibility then means when the new major version of the product comes out, say in 2015, the plugins will not run anymore. You can't install the 2013 plugin on the 2015 platform,
46:22
and that's typically the case for other platforms. But you can cross-compile. You can recompile it. And it is going to work. It's going to compile. It may give you deprecation warnings, because we did mark some APIs as obsolete because we intend to remove them later,
46:41
but it's still going to compile. So with zero maintenance, a single plugin can work for two major releases. That is a stability promise that we give, for example. And depending on how agile is your business or whether you are shrink-wrap desktop application or long-lift server-side product,
47:03
your stability promise is going to have to be different. Figure it out. But that's for it. Coming to process, and process is architectural as well, as architects nowadays have to really be very organization-minded and technology-minded and business-minded.
47:21
Process is part of it. Be test-driven. So before your developers start drafting the API, make them think through what's the scenario this is going to enable. And we use successfully the product called SpecFlow to draft the APIs in plain English and have it compiled by Visual Studio into unit test.
47:43
Simplifying. Systematically review all the APIs. So every new bit of API or every change to an API to go out, I wholeheartedly recommend you have a very rigid process, actually, to go through. And have a sign-off where all the guys that were arguing back there
48:02
of which way is the API supposed to be done, that they all now agree and they all uniformly accept the API the way it looks. They all collectively said it's consistent, it's usable, learnable, all of that. Because it's kind of obvious, right? But it's done this way and at this particular time of the API development
48:26
because that's the last point when it's relatively inexpensive to change it still. If you did ship it as public API, then yes, you're going to have plug-ins built against that and then good luck convincing everybody to change their code.
48:42
If you stick to your stability promise, as you should, then it's going to take you one year, two years, ten years, whatever your stability is, is what you promised, to deprecate and get yourself out of where you didn't really want to be if you had a very clear and collective sign-off.
49:03
Why is it a good idea to have the sign-off collective? Why is it... Oh, there's really two ways, right? Can you have a dictator saying, yes, this is the right way? Or should you gather more people, like representatives from if you grow the development, take a representative of each of those groups
49:21
participating in the API development and make them agree? What do you prefer, actually? A dictator or a democracy? Who's for dictatorship in API development? That's all right, yes. Two. Who's for democracy?
49:43
Not all the time. Okay, democracy doesn't work out. What? Balance. Yes, I think it takes a balance, you're right. I think it takes the collective wisdom for people to really feel ownership of the consistency.
50:02
Nobody likes to do things because they have to. People like to do things because they believe that's the right way to do things. But it sometimes takes a dictatorship to cut the discussions. And if there is really no productive outcome and there is two different ways how to do things and none of them is clearly better, then somebody has to choose.
50:23
All right? Yes, it depends. So when you dog food your API and it's inside the company, you have a lot of consumers,
50:42
there is no intellectual property, nothing else you need to take care of when you disclose. So you can invite anybody from within your company, typically. When you have real clients out there and our clients are big oilfield companies, we invite some of them after we've been through
51:01
a round of IP or non-disclosure agreements. But yes, it's a good idea to get the feedback as early as possible, especially when you have a client that's willing to give you feedback. You should cherish that. And I think that's really where I'm going, so thanks for that question. You've got to help the ecosystem to be born
51:22
and to be built up. And it's really costly, actually. The items here, they do not come cheap at all. But I'm convinced you have to do that if you are to hope for an ecosystem. And by ecosystem, I mean, again, we can compare ourselves to the app stores, the Apple and the Microsoft and Android.
51:44
They did help a lot, the ecosystems, to be built up. They had prizes for developers and conferences and giveaways and all that. What you can do, other than giveaways, is to give pre-commercial snapshots of your bits.
52:01
We do monthly throughout the alpha phase. Give more clear CTPs, customer or client technical previews where you say, this is the new bit, please look at that, we want your feedback. And alpha, beta, as usual. Why is that important?
52:20
That's because you want to get early feedback, obviously, but also you want your plugin writers to have their plugin ready when you're releasing your product. There is no use to have a phone out there on the market if there are no apps nowadays. Nobody buys a phone if there is less than 10,000 apps for it,
52:44
making the number up, but that's roughly what it is. So this is a way to help everybody along to be ready when you are shipping. Value their time, so of course, if there is ugly things that everybody would have to go through, automate it for them. Visual Studio visitors to generate well-played code
53:02
and get people started and do the right things in a very short amount of time and show their bosses how productive this new environment is and that the fact that they bought three licenses for you was a great idea. How much time is that saving? Make samples ready to run, so no additional configuration that you have to go through
53:21
to make the samples show up. That makes it successful. And the life after F5. So when the developer develops a plugin, wouldn't it be nice when they hit the F5 in Visual Studio that the product starts up with the plugin integrated and it stops at the first breakpoint? Yeah, that's productive, right?
53:42
So that takes some wiring into how you generate your projects, project files, Visual Studio project files, perhaps from a wizard. Make your stability promise or insert what we mean by that and you can have any flavor of that, but that's your contract between you and all of the plugin writers.
54:04
Do not break it. It costs. Been there. And make it social, of course. If you don't have a Stack Overflow tag, you don't exist. Heard the keynote, right? So that's my six lessons learned.
54:22
The takeaway from here, I did show you what we are doing internally, quite honestly, and successfully. So taking the lessons from the one product to other commercial products. So we did formalize this process and we measure how much each of these other products,
54:43
how close they are in their process, in their way they do things, in the way they test things, in the way they build things, and they have ways to improve basically, clear directions where they need to invest. We do that for other major internal development. Software, we don't ship.
55:01
We use it internally. What we've just been through is general software development practices that will make your code better no matter what. Reduce your couplings, for example, right? Obviously, by making clear APIs. So you can apply that there as well. But what you need to do most of all,
55:21
and I think I said it about five times, is that every product is different, every business is different. You of course have to tailor this to your needs. Fine print and disclaimer at the same time. Because, architecturally speaking at the end here, every product has different purpose,
55:42
drivers, business drivers, that translates to different architectural qualities, quality attributes, whatever you call them, and they will be differently prioritized. So again, extensibility may not be in the list at all, but reliability would be there. If you are in, say, medical or in the real-time processing,
56:03
real-time control, right? It has to be totally reliable, and it doesn't need to be really extensible, right? So figure out where you are, what is the priority of your architecture qualities to deliver on? Because that is what defines your architecture
56:22
and the trade-offs you have to make. If it then happens that extensibility, and I hope so, rises up, bubbles up through the sequence of the priorities, that means you're going to have to trade off something else. That means you're going to have to trade off something else
56:41
in your architecture. It's not just like buttons that will not be there anymore. Some other architecture qualities will have to take a hit. There's different trade-offs to make in every product. With that, thank you. And you can find out more by either asking questions now
57:02
if we have time, no? Or sticking around or coming to the booth. Thanks.