How to get a feature committed?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 35 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/48287 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
PGCon 201912 / 35
1
3
4
10
12
13
16
17
19
20
24
28
29
33
34
35
00:00
Computer animation
00:52
Computer animation
03:46
Computer animation
05:10
Computer animation
06:00
Computer animation
10:40
Computer animation
15:14
Computer animation
16:52
Computer animation
21:37
Computer animation
22:58
Computer animation
26:59
Computer animation
30:46
Computer animation
31:58
Diagram
34:33
Computer animation
38:42
Computer animation
45:32
Computer animation
47:03
Computer animation
49:03
Computer animation
50:23
Computer animation
51:42
Computer animation
53:36
Computer animation
56:49
Computer animation
Transcript: English(auto-generated)
00:06
Hi, good morning everybody. So it's in front of an extremely large audience that I'm going to talk to you about the PostgreSQL development process today. This is not a talk about hacking, this is a talk about personal
00:26
experience, about how we work within the community and how we get things done in it based on my own experience and some input I actually have to offer. So the title of the talk itself is how to get a feature committed
00:45
because when you send a patch your final goal is actually to get something merged into Postgres. So why would you want to do that? Why? Sorry, I forgot that one. Normally I don't have ten minutes between two slides so it should be fine
01:03
some automatic switch off. Anyway, so you want to get something merged. Why would you actually want to do that? Why is it worth it? So I'm going to try to explain to you this morning why and a couple of reasons why it's actually worth it. So let's keep going. So about myself that's just a small
01:25
presentation slide. My name is Michael Pacquier. I am French based in Tokyo and I'm contributing to PostgreSQL since 2009. I'm actually
01:42
contributing to Postgres since I live in Japan because I'm in the country there for 10 years. I got in a job that was for NTT back in the days and I got introduced to Postgres and since that point in time I'm continuing just to work on it and I just keep enjoying it and still do what I like. So nice. I do
02:07
patch reviews, I send patches to hackers. You may have seen my name from type to type. I try to do a couple of things. I don't know how many patches I have contributed to Postgres. A lot of small things mainly, a lot of features.
02:22
To my surprise there has been a feature I was working for Postgres 12 on which I have been working on since 2012 for a project that I basically gave up in 2014. Somebody picked it up and somebody else committed it. It happens that I perhaps have to maintain this thing now but it's really a cool feature.
02:43
It's called Reindex Concurrently. So I do some blogging, patch review, I work with the community and PostgreSQL has named me a committer of the project since 2014. So this talk also includes my own experience in being a committer, what it
03:03
means and what actually are the consequences about trying to do anything involving Postgres. I'm working for VMware where I'm doing a couple of things like anything related to packaging, PostgreSQL integration support for customers, as well as support for people using Postgres and
03:22
trying to integrate it as well. So anything touching Postgres usually comes into, oh you are the Postgres guy, and requests just come in and it's just the point of taking care of that and try to make people happy as much as possible or at least to extinguish fires as much as I can. So that's
03:47
about me. So about the community of PostgreSQL itself, so you're here, so it's a database, core database engine in charge of passing the data, receiving the thing, in charge of the protocol, planning, executing and it's here to
04:01
make sure that anything that you put in Postgres you can actually get it back because that's the whole point of the database. It has a steady design, its development process is perhaps slow for some enterprise class people but we take it slow because we want it to think about Postgres in the long term
04:23
as a very very long-term project. We still want the project to be around in 20-30 years, so we take it in a steady pace and we try to design features in a shape that they would still serve us to extend more PostgreSQL even in the future. It's a worldwide investment, we have
04:42
contributors from all over the world. The release notes include the name of the individuals who have worked on that, I don't actually know how many names we have on that, but we have a couple of hundred people contributing into the hackers mailing list where the PostgreSQL features are
05:01
Since 2018 we have a code of conduct that defines how the PostgreSQL people should actually work around that. PostgreSQL is, who knows actually about the DP engine ranking which is a kind of reference about database engine references around, so it basically says the higher your
05:27
score the more popular you actually are. PostgreSQL is number four into that, the first three you most likely know about them. This is around I think for something like 10 years now and PostgreSQL is keeping a huge growth and
05:43
within the last couple of years it's the database engine listed at least by that which has the highest growth rate of growth, so it's actually growing much on all the other ones. So I like to think about PostgreSQL as a product,
06:05
it's an open source community, we try to work together. It's open source, it's BSD licensed which means that you can just take the code, compile it by yourself, modify it as you want, you can redistribute it, resell it if you want, but I'm just going to say that folks have a
06:25
cost. You fork something, you maintain it and actually you have to maintain it and people usually in my own experience tend to really underestimate the cost it takes to maintain a fork in itself. So if you like what you do, perhaps the
06:43
community does not, you may still want to fork it, but you need to think also in terms of long-term prospects, which means that you actually, what you get into PostgreSQL itself may not be at first what you actually are looking for, but in my own experience you actually finish with something which is
07:04
actually much better than what you thought first because you get input from people you are not usually working with but work on Postgres for a very very long time so they may have input to offer to you which gives more value to what you are looking for. So some people who are also in this
07:24
audience are able to live with this model and they have actually with the time a huge customer base that allows them to maintain this model till they try to, I guess, catch up with PostgreSQL as much as they can and
07:42
they try to, at least I can say in the last 10 years or so, that people tend to see that, okay that was nice to do that in the first days, but it's harder to contribute first in Postgres, but if I think about that in like 10,
08:01
20 years it's going to be kind of difficult to maintain something which is more like a fork like that. So it's really hard to contribute in PostgreSQL for the first time, for the first patches. Most likely you are going to get your first patch rejected. Most likely I would say 30 or 40 percent
08:26
of the time Tom is going to pop in and say no sorry minus one, there is a famous c-shot on that, in short minus one Tom Lane. It happens that it's something which is very very difficult to say, it takes
08:43
experience and time but usually if somebody who is working on Postgres for a very long time tells you so, there are very good reasons that it's actually the case. So the initial time to do the development or to get used to it requires an initial investment, really an initial investment
09:02
from the company doing that. But if you think about that in terms of years and not short-term perspectives, it's actually you have a lot to gain from that. Speaking from my own experience, in my company we have been maintaining a fork before I joined of PostgresQL using 9.0, 9.1, 9.2. We have
09:24
decided to switch to 9.3 and we have deeply cut the amount of time we actually had to maintain our custom patches. And at the end we were kind of oh okay it's fine. So if you can use upstream, try to plug into upstream, you
09:46
can break Postgres in many awful and a lot of ways. Just for example use some of the hooks, try to plug into the planner hook or do anything. Most likely the first attempts of doing anything are going to break somewhere
10:01
Postgres, most likely. And the PostgresQL community is managed by a set of many individuals, be they committers, hackers, reviewers, people working on extension on the community ecosystem that have, I don't know, I'm just writing dozens of manures, but I think it may be actually way more than that. Perhaps a
10:24
couple of hundred manures or I'm not going to say a thousand, but I have absolutely no idea. But if you gather the total time of everybody working on that, you can most likely have a very, very huge number of that. So there is an experience behind that and people who have worked on Postgres. So the
10:42
database engine itself is an ACID compliant thing. ACID is atomistic consistency isolation and durability, as you may know about that. It's MVCC based. If you are here you most likely know about that. It's free to use as much as free is defined, which is that you can actually use it, but the cost of
11:05
running it, maintaining it, doing DBA work on it is actually up to you normally. It's open engineering, open discussion, open to new and good and even sharp ideas. It's even open to bad ideas because it's good to
11:20
actually, even if you think you have a good idea, somebody shows that and says oh actually it's not a good idea. It's good that you actually mention your idea on the list because somebody else has a look at it and say oh actually somebody has tried to propose that in the past and you have a past preference about that. So usually you keep around the reasons. The PostgreSQL
11:41
archives hold a trace of the exchanges happening on the hackers since 1996, since the first project has actually risen up in postgres.org. So if you browse up the archives, you try to look for past ideas, you may be able to
12:01
find stuff related to what you would actually like to propose about that. It's highly pluggable, you can do many things like data types, plug-ins, we have an extension ecosystem, you can do a lot of fancy stuff on that and we try to make PostgreSQL more and more pluggable such as not only we have a database engine but we have what I would perhaps say a database platform
12:26
which is that we try to make PostgreSQL more and more pluggable, more and more usable, so this has also a danger that perhaps PostgreSQL core itself is BSE but we may see the rise of more and more property, closed source which tries to plug in top of Postgres. Andreas mentioned that yesterday about
12:44
table AM. Table AM I think is my cool feature for Postgres 12. I'm actually playing with it and it's really cool from a hacker point of view but you have risks also behind that. Risks about closed source means that it's also up to you to maintain it and actually to run it, so if you know
13:04
it's a balance, you may be able to pay for that or perhaps not, it depends on what you actually want to do. The code quality is really extremely high based on some past expenses. Enterprise internal code
13:21
usually has a lot of hacks, xxx to-do, stuff like that about things which are actually not documented at all, under-documented, sometimes magic. In Postgres we try to avoid that. If you have a command there is a good reason behind that. The commit logs also have an extremely high quality
13:42
these days. If you take back the commit logs from perhaps 15 years ago, they were not as good as now, but it means that people have gained in attention and gained in quality about what they contribute into Postgres. So likely the criteria is really nice to see. So the people in charge of merging code into
14:04
Postgres, I would say the kind of guardians of Postgres, are a set of committers, people who are allowed to push code into the main repository itself. So the distribution is really worldwide. Based on the counts of this
14:21
week, there are 29 people listed as committers of the PostgreSQL project. Seven people have been newly named, including myself last year. We have one new committer this week and actually the new guy, I didn't know that, he's in New Zealand as well, so we have actually now two people in Oceania, which
14:42
is actually more or less in the same time zone as I do. I think I'm listing myself in the Europe one, I don't think I'm in the Asia one, anyway, but we have people freely from a lot of places, mainly in North America, US, and Europe
15:00
has the main attention, but I see also a lot of people from Asian countries or even Oceania or even other continents trying to contribute back and contribute directly to Postgres. So PostgreSQL has a yearly, basically a yearly
15:21
release schedule, so we try to release a new major version of Postgres every year, which happens basically in September, and we have a development schedule where we basically have a period where we discuss and integrate new features. It's
15:41
in this other stability period, but we try to have things divided basically in two pairs of time. You have the feature submission discussion that happens from July to April based on the latest trends, and after April we have something called a feature freeze, which is now actually a process
16:05
handled by the release management team, which is a release of a couple of individuals in charge of making sure that the release is able to come to pace and that we are able to release things in time. Depending on the issues, there could be a couple of months in between, but we try to push things
16:24
such as we keep things stable and on time for a release such as we are able to keep some pace, at least for the last couple of years. So after the feature freeze we have a period really focused on stability, roughly from April to September, where in September we try to say we have a .O which is
16:43
basically ready to be used by people. And we have also some beta releases, like one, two beta releases on top of that. So once you release something, in Git, not all open source projects do that, but we have one branch which is for the
17:04
stable work. Stable branches do not have new features, they have only bug fixes. Anything that crashes, anything which is considered, I would say, as a regression, even sometimes it depends on the cases, but we try to focus on the stability of the thing in itself. Stable branch is maintained
17:23
for five years after the GA release, so Postgres 12, which is planned for this September basically, would be maintained until 2024. And this year we
17:42
will have at the same time, basically I would say it's one minor release after the five-year window, which should be I guess in November, 9.4 will get its EOL and we're not going to maintain it on the community side, so this means that if you use a given version of Postgres, you basically have a
18:03
window of five years that you can use for upgrading your instances. So this may sound short from an enterprise point of view, because normally you release a product, you have support maintenance policies, so you either have to choose
18:21
between keeping up your product to use the newest version, or you need to maintain a custom stable branch by yourself. But from the community point of view, it's five years and patching one bug fix across basically six to seven branches is a lot of work. You can believe me on this one. Even simple
18:43
things, you need to make sure that you introduce no regressions, sometimes some areas of the code change slightly in some weird way which cause actually the patch to break in some cases. So it's not as easy as it is. Applying a feature is only one single branch, so people are like, okay, it's okay, so that's fine to do it.
19:03
Bug fixes get much more tricky in this area. I got a couple of months ago a bug fix about synchronous replication, which had to go down to 9.4, I'm doing that from time to time as well. We have much improved testing frameworks
19:22
since 9.6, and so you finish by either backpacking the testing framework to test it by yourself, or do some bashing or some manual testing or anything like that. It happens that in many cases I have in my own GitHub a version of the
19:40
testing framework which is compatible with the past versions, such as I can still rely on it in case if I need to. There are a couple of hacks around that, because I'm rather lazy on that, but it works and I was able to rely on it. It was about the, if you look for that, it was last summer about the consistency points issues on standby. So we have some test cases now. Yep. Yeah, you knew this one.
20:10
Sorry? What's your name? Oh yeah, yeah, okay, I see where you are. Yep, yep. So we, you may be happy to know
20:28
that we actually have a test case for that now. So it's not going to break again, normally at least the build time is going to complain before. The test case has been added in 12, but you can always run it on the back branches as
20:43
well. That's basically what I did also for some of my stuff. So you gain more
21:03
experience with the stupid stuff I did, which is good actually for you. Yeah, that's part of the experience as well. If you don't do any mistakes you don't actually, I mean anybody is going to do mistakes anyway, so it's just being proactive about them, taking responsibility about them is something
21:23
that as a committer is very, very important. As a patch author I think it's also very important to follow up on things and to show that you are willing to help in maintaining things even if you're not the one who actually merged the patch as well. That's based on my experience. So for the development cycle
21:41
things are divided into commit tests. We have in the last five or six years four to five commit tests since PostgreSQL 10, since the 10 release cycle, we have actually five running from June to the end of March, which are one
22:02
month period where we say we will try to commit patches, review them, we have a list of patches listed in a dedicated app called the commit test app which is on this URL directly. Six, seven years ago we had up to 120 patches to
22:24
deal with. It happens that lately not only patches do accumulate but more and more people contribute to Postgres and we have lately to deal with up to 250 patches in the commit test app itself. And we have one month of break between
22:42
each, but it happens also that a feature could also get committed in between depending on the talk. It really depends on the cases, this can happen as well, but we try really to keep things into this period. I have
23:09
committed myself stuff, I'm pretty sure I have committed stuff myself for new features in between commit tests as well. I mean it's still within the development period itself so I don't think it's actually a huge big deal.
23:23
For example it's a refactoring or it's something where it's even a new feature. It happens, it depends on your time, how you can actually allocate that. So it's not a strong rule but actually it's really to say we have that and a lot of patch authors or reviewers tend to focus
23:41
really on this period and you know they need a break after because it's a huge work happening even if you send one, two patches for newcomers it's a process to dive into it, to work with it, to get used to it. It's a huge process and it's a lot to take usually for people so it's good to
24:00
have breaks actually because you can actually rest on it and just take a set of coffees for a full month until you are able to move on and battle back into the battlefield. So regarding the stability period, we
24:22
have after the feature freeze a first beta which we tried to release just before PgCon. This year we have released it actually one week ago. I'm not seeing that many issues yet, new issues. So we have a couple of things but I'm not
24:44
seeing anything new and huge. I may be wrong on that Mr. Lane. Do we have any huge new open issues since we released beta 1? Not really that much.
25:03
I don't think so. So it means that you guys here are not doing your homework which is to test Postgres and report back issues to us. If you think it's an issue please tell us. Perhaps it's not an issue but at least let's discuss about it. So please do your work. We try also to do as much as we
25:24
can but everybody has a limited time slot so if you have some spare time you know what to do. And of course during this stability period it's of course up usually to the committers to fix stuff but help in stabilizing things is
25:43
actually good because you show involvement in the community process. In my experience, in my own experience, people like writing cool stuff, fancy code because you know they are cool, they are fancy but you need still to go
26:01
behind perhaps rethink portion of them it's not as cool as fancy but working also on the stability issues working on making sure that what you have at the end is something that customers can customers users so your customers can actually rely on is also something that I think is as important as
26:21
writing cool fancy and nicely shaped code. I don't like complicated things myself I would rather delete code than adding new code if it's actually to add more value in terms of the long-term maintenance and if it actually makes sense sometimes of course it doesn't make sense but I really like simple
26:42
things that's why even complicated features sometimes can't find their way through it if you cut it into simple pieces so I think it's good also to help in stabilizing things and showing involvement in that but cool features are
27:00
also cool because you know they have big items on the release notes and you say wow that's cool and it attracts even more customers but sometimes people tend to forget about the part where you actually need to maintain the actual thing that has been committed in itself so about the commit test management as I mentioned before we have up to 250 patches lately which is
27:22
a lot to deal with based on the patterns that I can see we have few rejections meaning that we don't want that please don't submit again basically plus we could have more I think we have a lot of patches which are what we
27:42
called return with feedback which is it's cool but you could perhaps if you do it this way it's not like we don't want it but we don't want it in this shape so please rework it and resend it once again for a follow-up commit test we have also a pattern which is sad so normally if you say I have 250 patches
28:05
all my patches are going to be treated for this commit test this never happens we have a lot of patches getting bumped from one commit test to the other because people don't care about them don't have the time to treat them to have some interest in it but we have a lot of patches getting moved from
28:23
one commit test to the other and we have even patches that get into the commit tests for many commit tests and nobody actually cares much about them if we don't actually care about them usually it means that we don't want them so by default they get either written with feedback but never actually
28:42
and something is is that everybody forgets about patches so you have your daily work you have your daily life you have a lot of things that can get into the involvement or actually the huge and motivating involvement that you
29:01
have or you want to get to the committee but I can see that authors forget about their patches reviewers also forget to review and register their names for a given patch so please have a follow-up of what you submit if you say to within the commit test application that you sent a patch and
29:22
somebody has sent a review for it just send a small email sorry I don't have the time this month I'm not going to make it please I'm going to send that once again and we are able to move on with that but people tend to actually not do that say honestly I don't have the time perhaps I will but
29:41
I'm not exactly sure so it's good if you show also good and fast involvement because it's less work for the people actually doing the management work of the 250 patches trying to do the vacuuming work for 250 patches requires in my own experience something like six to eight hours of focus and
30:02
still you make a lot of mistakes and at the end of the day you need not only a good coffee but really a good double black coffee if you think that moving
30:31
I mean obtaining the status in your patch such as it's not sitting down in itself and you think it's adapted yes please do it by yourself because it's
30:41
less work for the for the individual which I think I'm mentioning on my next slide no I'm actually not but perhaps I have that but we have a community manager the community manager is here to make sure that the community finishes at some point at least that we try to make it finish
31:00
on time so if you do it by yourself it's a huge help for it because the community manager cannot really do it's not physically or psychologically possible to do a close follow-up of each patch but as a patch author as a patch reviewer you actually know about the patch and its situation usually much
31:23
better than the community manager which could a little bit misjudge a given patch so if you can move in by yourself yes please definitely yes do it even if it's a small patch a couple of people still keep monitoring all that I do it if I can I see a patch it should come in as soon as possible I try to
31:46
actually to follow up on that some other people do that Tom here does that we have other committers who do this kind of work as well but it's good to update the commit fest app really as much as you can so this is a small graph about each commit fest in the commit fest app itself commit fest are
32:05
numbered using an integer so this is actually the integer so if you do go into the commutefest.posk.org and then append the number you would actually bump into the commit fest that we've been doing and that would be the commit fest back we have data in this commit fest app back to 20 we
32:25
have 22 23 5 commit fest a year 5 6 years so perhaps 2014 right I don't have here and that's basically a trend of all the data that you have usually
32:41
patches never stay in so that they have a final status which is either committed rejected written with it back moved and we have a new status called withdrawn which is another submitted it but actually I say no it's not a good idea so he's just giving up on the patch by himself so the trend is that patches are just basically growing even if we have a
33:03
steady state for the last two years or so of 20 200 250 or so patches and there is a kind of spike about the committed portion which matches to the last commit fest of release cycle because people tend to have stuff much
33:23
more stuff committed in the last commit fest as you can see there are a lot of patches also moved from one coming test with the other meaning that we have a steady state as well but you don't have actually that many new patches coming it's either so the total is also perhaps we may want stats where we
33:45
want to compare the amount of unique things that are kept around because many entries keep being around for two three commit tests so it's not the pure total it's the total of current entries in the coming test itself in terms of unique things I don't have exact numbers about that but you can see the
34:04
historic or about how many times a single patch has been moved to the coming test but it's a little bit harder actually to track that in the contest app so it's not an exact number it's a current state of things but if you take the full total starts it doesn't mean that we have like I don't know
34:22
200 to 2,000 3,000 patches no it doesn't mean that just means that we have also a lot of things that got around and not as many patches as you as you would think by looking at them so about the patch submission we have guidelines on the wiki normally if you send a patch you send it first to
34:41
hackers for discussion you register it into the commit test app work in progress patches are fine you are working on the cool thing it's a complicated thing particularly if you are working on a complicated thing please discuss first it's very very important you may have a work in progress
35:02
it may be cool but sometimes it's good to have a discussion or even a rough draft of something working which perhaps you've been doing say in one two days but it's much better in having something growth in one two days that you are ready to show up like oh I have this cool feature what do you think and I would like to do that and begin discussing on them then spending
35:25
one or two months working on something sending it to the committee sending it to the committee just to have it rejected immediately so I think when you implement a new feature about having even hacky or rough stuff done
35:42
in a short time frame first discuss it decide its design based on the interests of people and something which matters a lot also is if people are actually interested in that as well it may not be worth moving on with a feature that nobody actually would be much interested in progress perhaps you
36:01
are if you send a patch you are actually interested but interest from other people also matters a lot for any kind of features as well so usually you send image to pjsql dash hackers in this case if you have a bug fix or if you want to report a bug sending an email to hackers is fine as
36:20
well usually those go through the pjsql dash bugs which has a form on the website if I recall correctly I've never actually used it myself but it happens that we have that if you use the form on the website it gets assigned automatically a bug number which we sometimes reuse in the commit log itself
36:44
lately we tend to have in the commit logs themselves a reference to the thread where a given problem is being discussed which I think is a new trend and it's actually very helpful when you want to track past problems and actually to follow up on the discussion about what led to what has been committed so we kept references around yes no I'm just saying that you send it
37:16
to one of them some people like sending things to hackers sometimes to bug I
37:23
think bugs is preferred because it's in the name of the list itself but I haven't seen a lot of people sending a bug related problem even hackers hackers look at bugs as well I think it's fine to do it but please no do not send to multiple mailing lists because people tend to think common please
37:43
don't do that we have also when you try to cross posts given problem on even email there may be also some moderation rules based on that I'm not exactly sure about all the details because I I try to be a good student I
38:00
don't do it I just send email to one single list but no limit yourself into one single list as much as possible in case of bugs one of them is fine I think I would send it to bugs anyway because it's a bug hackers I mean hackers is fine as well it's not I think a strong rule perhaps people are
38:22
going to scream after me if I said that just after this presentation but I think it's my personal take on that both are fine because at the end it's we want to get something fixed and hackers are on both mailing lists mainly please prefer pjsq dash bugs as well so that's my take on so about the patch
38:45
contents you have all the coding itself comments are very important please please provide documentation about that and tests so back in the days if you have replication related thing people have been using bash or
39:02
anything like that we have a more advanced testing framework which is called tap test based on Perl compatible down to Perl 9.5 sorry 5.8.9 which is very hard requirement so it's I don't know when it was released but
39:21
it was some time ago anyway so we try to make to maintain certain compatibility such as build for members do not complain about that so depending on your feature you may be looking for isolation test regression test most likely tap tests depending on the requirements of what you have they
39:40
usually are designed not to be too costly still add value depending on what you have you may even send a patch and you may develop tests that are very very costly but if you can actually prove your points it helps people actually reviewing the patch even if you send something that takes 10 20 minutes it's most likely not going to get merged into the tree but if it helps
40:03
tree to prove your point to say I have a bug this bug can only be reproduced by that given test case and people are just able to I don't know take a tap script and you actually wrote one yeah please send it because it's always helpful for anybody looking at your patch in terms of review or anything a long-running test is not going to be merged but
40:25
sometimes it could be reduced in such a way that it's brings value and it's not it does not actually cost much to introduce in the final fix that gets into the tree yes oh yeah yeah so what I tend to do I mean I have dealt
41:02
with cases where actually I had to do that myself because sometimes you want to it's more case I'm just taking a small example for example I had a couple of months ago a bug related to begin commands and I wanted to trigger errors in some specific code path to make sure that the session is still isn't in
41:23
the same state because I think it was some folks from people actually reported that and they actually for the code but we were not doing the actual operation in a really consistent way in Postgres so what I have been doing was sending a patch to the mailing list that was just C code that please apply
41:42
it on top of it apply this patch and then what I had was a simple stat check for this flag on disk and just trigger an error if you see this flag to be able to reproduce the errors like that so it depends on the case but I have been doing that as well sometimes for other cases I have been doing introducing manual slips in a couple of code paths to actually be able to
42:02
reproduce that reliably as well because sometimes you just cannot you don't have an infinite number of points that you can plug into it so hooks have a limitation anyway so sometimes you may have something which has the shape of a C patch and yes these are actually good to have
42:23
because if it's only to slow down the code to have to reproduce a race condition particularly usually you must I mean it's not must but you would most likely do that as well so if you have test cases like that I say yes please send them these are always very helpful yes yeah yeah if you say for
42:47
example if I take I've had such stuff also in the past as well just take a breakpoint here do that do that do that if you keep precise reproducible steps these are absolutely great I mean the I think that these are in my opinion
43:06
high quality reports because you actually show how to reproduce it if you show how to reproduce it you already did 80% of the work because you discovered how to reproduce it I mean it's 80% it may be actually more than that because after that it's just actually reproducing a
43:21
fixed point if it's possible to reproduce it yes please send all your steps precisely I did that that that and that if you play with a debugger yes please do it I took a break point there there precise steps are good reports quality reports mean a fast fix fast fix means that you and
43:42
potentially your customer are going to be happy much faster than what you would expect we have a lot of reports with people sending bugs and saying that oh I have that and it crashed and the first reply we do is we don't know what you did please send a backtrace at least if you have a reproducible test
44:03
case please send it we are limited to guess what it is and this costs time and we may not have actually the time to dig into trying to guess what you perhaps did wrong sometimes people even have custom extensions that cause the breakage and you see that if you look at the backtrace for example so it really
44:27
depends a lot in the things so if you send a patch a bug if you have anything reproducible yes please send it send your steps it's very good to have so we have I think I'm running out of time maybe so just to go through that quite
44:43
quickly we have coding conventions for anything like macros AR formats signal handlers and we try to keep the documentation about that quite clean and at least documented even if it's far from being complete it has a lot in
45:01
that for example if you want to do some blocks which don't get indented it's in the docs this is referred into the docs with this link so if you can if you want to write patches you can always have a look at that we have also configuration for editors you may be a VI a max or something else guy we
45:22
have also documentation that you can use to make your code easier to reproduce for Postgres following the PostgreSQL coding convention as well so for regression tests we have main regression tests in SRC tests regress which are the the oldest historical ones we have since 9.1 I think since
45:45
serializable has been introduced isolation tests to be able to test concurrent behaviors between multiple sessions we have of course extensions tests SRC test modules is more recent than that test in country modules since extensions have been introduced we have a huge improvement on
46:03
that because we have the actual framework to test that more easily since 9.4 and in more advanced way 9.6 we have more advanced recovery related tests for example for anything related to crash recovery archiving also likely
46:21
logical replication, publication subscription so we have the basic framework to do to do that I have done a lot of work on that and people use them a lot and we have them also running in the build farm automatically so this took a couple of years but we are really in a good shape I think in this area it's really cool stuff and we can do much more than what we tended to do
46:43
in the past we have a set of non-secured tests which you can specify using that I have that in my environment but it's on my own laptop and it's secure so it's fine so you can run a set of extra tests which are for SSL, LDAP, Kerberos depending on what you install and what you
47:02
configure as well for your computation stuff about the patch formats people have preferences yes no so acceptable formats is in my opinion anything that can actually be cleanly applied so GitAM tends to be very talkative and
47:25
complain quite easily on that but usually if I cannot apply it and I can just basically apply using a patch-p1 I'm basically fine with it sometimes I have seen patches that cannot actually apply but if I can read it say yeah I can be able to produce a patch based on that so using
47:47
things like Git format patch and adding a version number into the patch you send is actually very very helpful in following up how the patch versions are evolving as well this is usually what you should try to do so sometimes
48:04
splitting things into multiple commits also makes a lot of sense you have a cool feature but perhaps you can do first refactoring which makes sense these are much easier to commit because you reshape the code in the way you want in a more modular way and then you get the actual real feature getting done and you reduce the amount of global diff that your feature introduces
48:22
improving the readability of the commit and the git history happening in Postgres so that's my take to send a patch as long as I can apply it I'm fine so it's a patch-p1 as long as it goes through that I use all commands any kind of commands but as long as it's possible to produce
48:42
something perhaps it won't compile if it doesn't compile perhaps it needs some rebasing as well depending on the cases but it's usually easy enough to actually see if the patch is here and if you have a reproducible test case it's possible to still use it and adapt to it directly as well so I don't know
49:02
how many patches I have I have a couple extra more anyway so we have a commit test manager we don't have any more time a kind of deputy handling all the patches and doing a lot of work on that this is done by season developers it means mostly tracking patches poking at people please do it
49:24
please do it please we're sending emails and being kind of noisy about such matters we have a CF bot which tests automatically patches that has been something developed by Thomas Monroe with a given URL we have automatic tests on Linux or Windows people tend to send patches on Linux
49:45
few have Windows environments so if you send a patch and you see it breaks on Windows you cannot actually test that automatically I have been abusing of that a little bit to patch the test codes in such a way that some tests
50:00
which are not triggered by default would be triggered by that as well so you can also abuse a little bit of that Thomas Monroe has written that and it's very very helpful because you can see the status of all your patches in a given commit test running so you can actually follow up if you did something wrong or not if it requires a rebase you can still be done if and for the
50:22
commit test manager it's very very helpful to see so about the patch review we have exchanged between authors and reviewers a patch may be marked as ready for committer at the end and when it's written like that the committer looks at it and say I'm okay to commit it or not it depends on
50:41
cases sometimes the committer looks at it says no it's not quite ready yet please do that or rework it in this way or this way it depends really on the flow and also on the patch difficulties usually when you write one patch please review one patch of equal difficulty to maintain the balance in the force so as we don't finish in cases where we have a bunch of patch
51:09
authors sending a lot of cool features with nothing actually getting reviewed it happens that it's actually the case for a lot of things but this tends to improve within the last couple of years actually and reviews are very very
51:23
important part of the process because you get used to more code and you gain more experience in trying to break things trying to break things is actually the cool part about reviewing stuff as well because you can analyze problems you can view new code you can try to analyze new parts of the code and get more familiar with the PostgreSQL code base. Usually when you send a patch
51:45
people largely underestimate the amount of time it takes to write a patch you can most likely expect your patch to be rewritten once twice or even more so please make sure that you agree about the shape of the patch with the people you are dealing with to make sure that you don't actually send versions
52:04
that serves absolutely for nothing sometimes people just rush send a new patch and somebody comes in say no actually no it's not going to happen it still happens but you really should be careful about making sure that you have an agreement about something instead of sending 30 40 versions that
52:25
actually don't with most versions actually not mattering much so a consensus is key you are not going to have a patch in the shape you designed it first never usually not really happens like in 90% of the cases I People come in you have n persons participating into a given discussions
52:46
you are going to get at least n plus one opinions on it so make sure that you have a consensus it's an exchange negotiation with other people sometimes you need to drop the ball and accept that other people also have opinions I
53:06
mean it's actually an essential part of the process because people have reasons to say so if they say so they have usually good reasons to say so so it's good to communicate on that and to reach a state where you all
53:21
actually agree about how the thing is actually shaped and a complicated patch will never finish the way you actually want it to be it usually finishes in a much better shape than you usually proposed so the review process is actually very very important so in conclusion of that I'm
53:41
just going to say that you need to be patient investing time in the committee requires energy time sometimes you work in environments that do not provide any of them sometimes work in some cases where you have some of them you may rush into having new features a customer issue that needs an immediate look up
54:03
on that you need to work with that as well but you have also matters in life so everybody has a life so please be patient with people do not work on patches on Friday nights it's an extremely bad idea as a committer for
54:25
one year in PostgreSQL in my own experience it works not only for PostgreSQL but for anything do not commit patches on Friday night it's an extremely bad idea so when it comes to Postgres if you commit a patch and break
54:41
something people are going to show up mostly immediately saying you broke the build farm please fix us up because other people are working on it and we want it to be stable which is a very acceptable position in my opinion so it
55:01
applies this is really a general rule if you do something make sure that you have after committing something room to make sure that you didn't destabilize everything and you can actually come in and at the end perhaps do just a git revert or anything like that but this doesn't apply only to Postgres so help others take new challenges reviewing new parts of the course you
55:24
are not familiar familiar into means that you will get good with this code if you continue to have a look at it so it's very very important to have a look at new areas and even if you don't know it don't hesitate to jump in so it's not very fancy but fixing bug and doing the actual maintenance work
55:43
requires a lot of work so if you can come in and help I think it's as important as doing cool features and stuff it's even more important because you maintain a stable long-term perspective which is very very important for Postgres. Remain polite, respect others. The PostgreSQL community
56:04
people is a community that people like working with because we try to be at least we are we try to be an open community we deal with each other without any kind of naming any community or anything like that but we
56:22
try to be open to others remain gentle to each other to respect everybody's opinion and I'm really grateful for working with this community because actually you feel that you are part of something and you contribute to it for a long term and what you do now is something that you will still see in
56:41
ten years from now. In software engineering you don't see that much around honestly. So I think that's everything I have I got question in the middle so I got a little bit overboard I think.