1 00:00:02,719 --> 00:00:04,560 I'm here to talk about fixing the PR 2 00:00:04,560 --> 00:00:06,560 bottleneck and 3 00:00:06,560 --> 00:00:09,919 this is kind of a grand title for trying 4 00:00:09,919 --> 00:00:11,960 to fix the thing that most organizations 5 00:00:11,960 --> 00:00:13,800 struggle with, I think, and have kind of 6 00:00:13,800 --> 00:00:16,039 historically struggled with before AI, 7 00:00:16,039 --> 00:00:16,920 you know, 8 00:00:16,920 --> 00:00:19,039 we have always had huge numbers of PRs 9 00:00:19,039 --> 00:00:20,480 just laying around that no one's 10 00:00:20,480 --> 00:00:22,839 bothered to review and this is now 11 00:00:22,839 --> 00:00:24,440 increased massively because of the new 12 00:00:24,440 --> 00:00:27,440 strains on us because of AI imposing all 13 00:00:27,440 --> 00:00:29,039 these weird constraints. 14 00:00:29,039 --> 00:00:30,039 And 15 00:00:30,039 --> 00:00:31,519 to do this, I'm going to use the rubric 16 00:00:31,519 --> 00:00:33,640 of my skills, which is kind of heard 17 00:00:33,640 --> 00:00:35,759 about, maybe you've used them. And I 18 00:00:35,759 --> 00:00:37,719 have a couple of new skills to announce 19 00:00:37,719 --> 00:00:40,119 that are going to hopefully 20 00:00:40,119 --> 00:00:41,759 improve the way that you do PRs, improve 21 00:00:41,759 --> 00:00:43,640 the way that's or improve the speed at 22 00:00:43,640 --> 00:00:45,920 which you can review them and do them. 23 00:00:45,920 --> 00:00:47,359 Speed. 24 00:00:47,359 --> 00:00:49,159 Now, 25 00:00:49,159 --> 00:00:52,920 we've been pushed to do more with less, 26 00:00:52,920 --> 00:00:55,840 essentially, or more PRs, more work, 27 00:00:55,840 --> 00:00:57,439 more stuff. And this is the kind of the 28 00:00:57,439 --> 00:00:59,479 central promise of AI, that we're going 29 00:00:59,479 --> 00:01:01,320 to be able to use these agents to scale 30 00:01:01,320 --> 00:01:03,560 ourselves up to do more work. 31 00:01:03,560 --> 00:01:06,120 And this has resulted in the software 32 00:01:06,120 --> 00:01:07,519 factory, probably the biggest buzzword 33 00:01:07,519 --> 00:01:08,519 of the day. Everyone's talking about 34 00:01:08,519 --> 00:01:10,799 software factory that I have chance to. 35 00:01:10,799 --> 00:01:11,879 And 36 00:01:11,879 --> 00:01:13,079 I think of a software factory as 37 00:01:13,079 --> 00:01:15,200 primarily something where 38 00:01:15,200 --> 00:01:17,719 instead of the human initiating 39 00:01:17,719 --> 00:01:19,719 all of this work, we're going to pass 40 00:01:19,719 --> 00:01:21,280 some of that initiation, some of the 41 00:01:21,280 --> 00:01:22,760 initiation is going to be done by 42 00:01:22,760 --> 00:01:30,760 agents. And I think of that 43 00:01:52,200 --> 00:01:54,079 like Jeff come in and say, "Okay, let's 44 00:01:54,079 --> 00:01:55,879 turn that into a fix or turn that into a 45 00:01:55,879 --> 00:01:57,840 reproduction." Or maybe I ping people 46 00:01:57,840 --> 00:01:59,719 straight away. And maybe you have other 47 00:01:59,719 --> 00:02:01,280 things. Maybe you have Planet Scale 48 00:02:01,280 --> 00:02:03,280 hooked up, so it gives you query reports 49 00:02:03,280 --> 00:02:05,079 on the slow queries on your database. 50 00:02:05,079 --> 00:02:07,120 Maybe that then triggers a different 51 00:02:07,120 --> 00:02:08,560 thing of your software factory. All of 52 00:02:08,560 --> 00:02:12,479 this is not humans triggering it, it's 53 00:02:12,479 --> 00:02:14,360 uh deterministic code triggering it, 54 00:02:14,360 --> 00:02:16,199 right? And so these accelerate your 55 00:02:16,199 --> 00:02:17,919 software factory. They push more code 56 00:02:17,919 --> 00:02:19,120 through it. 57 00:02:19,120 --> 00:02:21,599 But then you need brakes, right? 58 00:02:21,599 --> 00:02:23,759 If you just have permanent acceleration 59 00:02:23,759 --> 00:02:25,639 pushing stuff through your factory, 60 00:02:25,639 --> 00:02:26,680 you're going to end up with a slop 61 00:02:26,680 --> 00:02:28,319 cannon, right? You're just going to end 62 00:02:28,319 --> 00:02:30,800 up with a ton of slop crappy PRs that 63 00:02:30,800 --> 00:02:32,840 you're not going to be able to touch or 64 00:02:32,840 --> 00:02:35,719 review, or even freaking look at. 65 00:02:35,719 --> 00:02:37,800 So you need brakes. These are mechanisms 66 00:02:37,800 --> 00:02:41,080 that slow down, that increase quality, 67 00:02:41,080 --> 00:02:42,639 that make sure that your code base 68 00:02:42,639 --> 00:02:44,319 doesn't turn into a software entropy 69 00:02:44,319 --> 00:02:46,599 nightmare, because code is the 70 00:02:46,599 --> 00:02:49,120 environment your agent operates in. And 71 00:02:49,120 --> 00:02:50,879 if you have bad code in your code base, 72 00:02:50,879 --> 00:02:54,039 that is going to beget more bad code. 73 00:02:54,039 --> 00:02:55,319 And so I'm going to talk about these 74 00:02:55,319 --> 00:02:57,719 three brakes in this talk, and talk 75 00:02:57,719 --> 00:03:00,039 about how we can use them to 76 00:03:00,039 --> 00:03:02,919 counterintuitively go faster. 77 00:03:02,919 --> 00:03:04,439 So what are automated checks? 78 00:03:04,439 --> 00:03:06,159 These are the deterministic checks in 79 00:03:06,159 --> 00:03:07,840 your repo that we've had for thousands 80 00:03:07,840 --> 00:03:10,159 of years, or you know, since the '50s. 81 00:03:10,159 --> 00:03:12,400 Deterministic checks, where we have 82 00:03:12,400 --> 00:03:14,599 linting and tests and type checking and 83 00:03:14,599 --> 00:03:16,519 code quality metrics. All of these 84 00:03:16,519 --> 00:03:19,039 things going together, and they all work 85 00:03:19,039 --> 00:03:21,039 the same every time. 86 00:03:21,039 --> 00:03:22,159 Layered on top of that, we have 87 00:03:22,159 --> 00:03:25,159 automated review. So we have agents who 88 00:03:25,159 --> 00:03:27,319 look at our code and say, "Okay, 89 00:03:27,319 --> 00:03:28,400 you know, these are for the things that 90 00:03:28,400 --> 00:03:30,639 the tests didn't catch, or this is 91 00:03:30,639 --> 00:03:31,879 looking at the structure of the code 92 00:03:31,879 --> 00:03:33,280 base in general." 93 00:03:33,280 --> 00:03:34,959 And then on top of that, the third 94 00:03:34,959 --> 00:03:37,239 layer, the final layer, is human review. 95 00:03:37,239 --> 00:03:39,400 So people looking at the PR. 96 00:03:39,400 --> 00:03:41,560 And these three layers form this kind of 97 00:03:41,560 --> 00:03:43,800 cake that we end up with 98 00:03:43,800 --> 00:03:45,560 when we get to human review. 99 00:03:45,560 --> 00:03:47,439 And so more speed, of course, means more 100 00:03:47,439 --> 00:03:50,239 PRs. And so the goal here is to make 101 00:03:50,239 --> 00:03:52,520 human review faster by leaning on those 102 00:03:52,520 --> 00:03:55,280 first two phases. 103 00:03:55,280 --> 00:03:56,960 And we're going to stop the slop. 104 00:03:56,960 --> 00:03:58,319 That's the first principle here, which 105 00:03:58,319 --> 00:04:00,479 is if you raise the quality of the code 106 00:04:00,479 --> 00:04:01,400 that you're shipping, you're going to 107 00:04:01,400 --> 00:04:04,000 end up doing less human review because 108 00:04:04,000 --> 00:04:05,479 it's just going to be better work, and 109 00:04:05,479 --> 00:04:07,400 so you're going to end up needing to uh 110 00:04:07,400 --> 00:04:09,680 make fewer interventions. 111 00:04:09,680 --> 00:04:10,719 So, 112 00:04:10,719 --> 00:04:12,400 automated checks. 113 00:04:12,400 --> 00:04:15,039 Now, automated checks are cheap. That's 114 00:04:15,039 --> 00:04:16,759 the cool thing about them is that they 115 00:04:16,759 --> 00:04:18,560 don't cost tokens like automated review 116 00:04:18,560 --> 00:04:20,798 does, they don't cost human effort, they 117 00:04:20,798 --> 00:04:23,360 just cost CPU cycles. So, these are for 118 00:04:23,360 --> 00:04:24,639 instance, you know, you run your tests 119 00:04:24,639 --> 00:04:26,560 on every code change. Maybe those tests 120 00:04:26,560 --> 00:04:29,839 do incur some tokens because, let's say 121 00:04:29,839 --> 00:04:32,279 um an agent uh creates a bug and the 122 00:04:32,279 --> 00:04:33,918 test catch it, then you need to spend 123 00:04:33,918 --> 00:04:36,000 some tokens to go and fix it. But, those 124 00:04:36,000 --> 00:04:37,439 are tokens pretty well spent in my 125 00:04:37,439 --> 00:04:39,519 opinion. So, checks are cheap. That 126 00:04:39,519 --> 00:04:41,720 means you can layer on loads and loads 127 00:04:41,720 --> 00:04:43,480 and loads of them on your repos, and 128 00:04:43,480 --> 00:04:45,759 you're probably not using enough of them 129 00:04:45,759 --> 00:04:47,399 or not being creative enough with your 130 00:04:47,399 --> 00:04:49,000 use. 131 00:04:49,000 --> 00:04:53,000 But, checks can lie. 132 00:04:53,000 --> 00:04:54,560 Does a green CI mean that the code is 133 00:04:54,560 --> 00:04:56,800 ready for merge? No, it does not. And 134 00:04:56,800 --> 00:04:58,680 so, we've always needed, on top of these 135 00:04:58,680 --> 00:05:01,879 checks, some extra layer to figure out 136 00:05:01,879 --> 00:05:03,439 if there's anything catastrophically 137 00:05:03,439 --> 00:05:05,800 wrong with the code before we ship it. 138 00:05:05,800 --> 00:05:08,240 And so, all of the other phases, the 139 00:05:08,240 --> 00:05:09,680 human review and automated review, these 140 00:05:09,680 --> 00:05:11,959 are lie detectors. These are for finding 141 00:05:11,959 --> 00:05:14,399 lies in the automated checks. 142 00:05:14,399 --> 00:05:15,600 Now, I want to show you some of these 143 00:05:15,600 --> 00:05:17,000 lies first of all 144 00:05:17,000 --> 00:05:18,560 because this helps when we're thinking 145 00:05:18,560 --> 00:05:19,918 about code and thinking about automated 146 00:05:19,918 --> 00:05:23,040 checks to see how bad it is and how bad 147 00:05:23,040 --> 00:05:24,360 things can get. 148 00:05:24,360 --> 00:05:27,319 The first is tautological tests. 149 00:05:27,319 --> 00:05:28,959 A test that just reasserts the 150 00:05:28,959 --> 00:05:31,160 implementation. Opus 5 got addicted to 151 00:05:31,160 --> 00:05:33,160 these. I don't quite understand why. It 152 00:05:33,160 --> 00:05:34,800 would have, for instance, 153 00:05:34,800 --> 00:05:37,120 X post character limit equals 280. Can 154 00:05:37,120 --> 00:05:40,000 anyone guess the test that was written 155 00:05:40,000 --> 00:05:41,759 to test this behavior, right? You've 156 00:05:41,759 --> 00:05:42,918 probably seen this a thousand times. 157 00:05:42,918 --> 00:05:45,199 This is real code from agents or from 158 00:05:45,199 --> 00:05:47,199 stuff that I found my agents doing. 159 00:05:47,199 --> 00:05:49,560 it said expect X post character limits 160 00:05:49,560 --> 00:05:51,839 to be 280. 161 00:05:51,839 --> 00:05:54,199 So, the implementation looked like that. 162 00:05:54,199 --> 00:05:56,680 And the test essentially reasserted the 163 00:05:56,680 --> 00:05:58,279 implementation. That is a tautological 164 00:05:58,279 --> 00:05:59,839 test. 165 00:05:59,839 --> 00:06:02,720 And tautological tests are bad because 166 00:06:02,720 --> 00:06:05,600 they're extremely structure sensitive. 167 00:06:05,600 --> 00:06:07,720 They're very sensitive to the actual 168 00:06:07,720 --> 00:06:10,000 internal workings of the system. So, it 169 00:06:10,000 --> 00:06:12,319 means I cannot change that constant 170 00:06:12,319 --> 00:06:15,160 without a test failing. But 171 00:06:15,160 --> 00:06:17,000 I cannot rename that constant without a 172 00:06:17,000 --> 00:06:19,319 test failing. Like I have to do it's so 173 00:06:19,319 --> 00:06:22,040 tied into the structure of my system. 174 00:06:22,040 --> 00:06:23,918 And I found another one which is even 175 00:06:23,918 --> 00:06:26,800 more egregious, I would say. This is 176 00:06:26,800 --> 00:06:30,240 incredible uh test. What it's doing here 177 00:06:30,240 --> 00:06:33,079 is it's essentially testing whether two 178 00:06:33,079 --> 00:06:34,800 things in the UI appear in the right 179 00:06:34,800 --> 00:06:36,680 order. So, it's checking that the pitch 180 00:06:36,680 --> 00:06:39,519 detail page uh or rather the video 181 00:06:39,519 --> 00:06:41,800 section comes after the content plan. 182 00:06:41,800 --> 00:06:43,399 What it does it doesn't render it to a 183 00:06:43,399 --> 00:06:47,639 screen, it just reads the actual file, 184 00:06:47,639 --> 00:06:50,519 the module, into its own memory, and 185 00:06:50,519 --> 00:06:52,879 then it finds the right thing, so finds 186 00:06:52,879 --> 00:06:54,959 content plan, finds videos, and then it 187 00:06:54,959 --> 00:06:57,879 expects the videos to be after it in the 188 00:06:57,879 --> 00:06:59,839 source material. 189 00:06:59,839 --> 00:07:01,439 Which is crazy if you think about it cuz 190 00:07:01,439 --> 00:07:03,480 I can just like change the way the 191 00:07:03,480 --> 00:07:05,160 source material looks and this test will 192 00:07:05,160 --> 00:07:07,040 fail. It's too sensitive to the 193 00:07:07,040 --> 00:07:09,279 structure of my code base. So, that's 194 00:07:09,279 --> 00:07:10,639 another way that automated checks can 195 00:07:10,639 --> 00:07:11,759 fail. 196 00:07:11,759 --> 00:07:14,040 And there are also Oh, sorry. Automated 197 00:07:14,040 --> 00:07:16,000 checks can lie. 198 00:07:16,000 --> 00:07:17,439 There are also tests that literally 199 00:07:17,439 --> 00:07:18,639 cannot fail. 200 00:07:18,639 --> 00:07:20,399 And this will feel familiar to you if 201 00:07:20,399 --> 00:07:22,399 you've abused mocking in the past or 202 00:07:22,399 --> 00:07:24,160 have used various things. 203 00:07:24,160 --> 00:07:26,519 For instance, here we have a use audio 204 00:07:26,519 --> 00:07:28,680 boost function. 205 00:07:28,680 --> 00:07:30,800 And this use audio boost function, uh 206 00:07:30,800 --> 00:07:33,839 its internals use the audio context API 207 00:07:33,839 --> 00:07:35,000 in the DOM. Don't worry if you don't 208 00:07:35,000 --> 00:07:36,800 know any of this. But what we're doing 209 00:07:36,800 --> 00:07:38,519 here is we're just stubbing it out with 210 00:07:38,519 --> 00:07:41,399 some fake methods. And it turns out the 211 00:07:41,399 --> 00:07:44,040 audio context has some complicated error 212 00:07:44,040 --> 00:07:45,920 modes and it will fail if you use it 213 00:07:45,920 --> 00:07:47,800 under strange conditions. And so just 214 00:07:47,800 --> 00:07:50,079 doing this means our tests cannot fail 215 00:07:50,079 --> 00:07:51,639 using those modes and we're going to hit 216 00:07:51,639 --> 00:07:53,399 strange errors in production that our 217 00:07:53,399 --> 00:07:55,360 tests can't fix. 218 00:07:55,360 --> 00:07:56,920 And so 219 00:07:56,920 --> 00:07:58,720 the question is then 220 00:07:58,720 --> 00:08:01,480 if you can cheat on automated checks, if 221 00:08:01,480 --> 00:08:04,319 you know, and even in good faith ways as 222 00:08:04,319 --> 00:08:06,600 well, the the AI isn't trying to write 223 00:08:06,600 --> 00:08:09,160 bad tests here. It's just taking our 224 00:08:09,160 --> 00:08:11,639 instructions and writing tests that are 225 00:08:11,639 --> 00:08:13,439 too tied into the structure instead of 226 00:08:13,439 --> 00:08:15,720 actually executing code. So how do we 227 00:08:15,720 --> 00:08:18,720 make automated checks harder to cheat? 228 00:08:18,720 --> 00:08:20,319 If we can do that, then we can increase 229 00:08:20,319 --> 00:08:21,918 the quality of those checks, which means 230 00:08:21,918 --> 00:08:25,439 what's increases our quality bar. 231 00:08:25,439 --> 00:08:27,560 And the first thing I really like about 232 00:08:27,560 --> 00:08:29,600 this is code base design. So you can 233 00:08:29,600 --> 00:08:31,959 actually design your way out of these 234 00:08:31,959 --> 00:08:33,519 bad checks. 235 00:08:33,519 --> 00:08:35,879 So what does good code base design look 236 00:08:35,879 --> 00:08:37,120 like? I've talked about this before in 237 00:08:37,120 --> 00:08:39,120 previous AI engineer talks I've given, 238 00:08:39,120 --> 00:08:40,479 which are deep modules. These are 239 00:08:40,479 --> 00:08:42,559 modules that hide complex behavior 240 00:08:42,559 --> 00:08:45,480 behind simple interfaces. This is a John 241 00:08:45,480 --> 00:08:47,159 Ousterhout idea from the philosophy of 242 00:08:47,159 --> 00:08:48,960 software design. 243 00:08:48,960 --> 00:08:50,679 If we look at these two modules 244 00:08:50,679 --> 00:08:52,919 we've got A which has a large 245 00:08:52,919 --> 00:08:55,080 implementation hiding behind a tiny 246 00:08:55,080 --> 00:08:57,919 little interface at the top. Okay? And 247 00:08:57,919 --> 00:09:00,440 then B is a large interface, lots of 248 00:09:00,440 --> 00:09:02,159 functions you can call and those 249 00:09:02,159 --> 00:09:03,720 functions individually don't do very 250 00:09:03,720 --> 00:09:05,480 much. Does that make sense? 251 00:09:05,480 --> 00:09:06,600 Yeah? 252 00:09:06,600 --> 00:09:07,759 Now 253 00:09:07,759 --> 00:09:10,320 if you have a deep module like A here, 254 00:09:10,320 --> 00:09:12,120 you're going to have fewer structure 255 00:09:12,120 --> 00:09:14,960 sensitive tests because you're hiding 256 00:09:14,960 --> 00:09:17,360 more of the implementation behind that 257 00:09:17,360 --> 00:09:18,919 interface. If it's just testing at that 258 00:09:18,919 --> 00:09:20,559 interface, you're going to get better 259 00:09:20,559 --> 00:09:22,759 tests. And so that your job here is to 260 00:09:22,759 --> 00:09:25,679 force the agents to use that little 261 00:09:25,679 --> 00:09:27,919 interface instead of reaching into the 262 00:09:27,919 --> 00:09:29,480 implementation to test these weird 263 00:09:29,480 --> 00:09:32,080 implementation details. 264 00:09:32,080 --> 00:09:34,360 So I've got to go for this. 265 00:09:34,360 --> 00:09:37,480 You can have like the weirdest vibe 266 00:09:37,480 --> 00:09:39,919 coded like code base, the crappiest code 267 00:09:39,919 --> 00:09:41,519 base that you've ever set your eyes on, 268 00:09:41,519 --> 00:09:42,960 and you can run this skill on it and it 269 00:09:42,960 --> 00:09:44,879 will make it better. What this does 270 00:09:44,879 --> 00:09:46,639 essentially gives you opportunities for 271 00:09:46,639 --> 00:09:48,720 deepening modules. 272 00:09:48,720 --> 00:09:49,799 And 273 00:09:49,799 --> 00:09:51,000 kind of looks like this. 274 00:09:51,000 --> 00:09:51,879 Raise your hands if you've used this 275 00:09:51,879 --> 00:09:53,240 skill, by the way. 276 00:09:53,240 --> 00:09:54,320 Not sure how many of my folks are in 277 00:09:54,320 --> 00:09:55,960 this room. Yeah, okay. It's really 278 00:09:55,960 --> 00:09:57,919 freaking nice. Essentially, it gives you 279 00:09:57,919 --> 00:09:59,759 a HTML document. I'll get out of the way 280 00:09:59,759 --> 00:10:02,399 here. Of all of the different um 281 00:10:02,399 --> 00:10:04,799 potential opportunities it sees. So, we 282 00:10:04,799 --> 00:10:06,919 can see here we have a before and we 283 00:10:06,919 --> 00:10:09,080 have an after where we're sort of like 284 00:10:09,080 --> 00:10:10,720 reducing duplication, we're creating a 285 00:10:10,720 --> 00:10:13,240 nice deep testable module. 286 00:10:13,240 --> 00:10:14,399 And then you can go ahead and implement 287 00:10:14,399 --> 00:10:15,279 that. 288 00:10:15,279 --> 00:10:16,960 And attached to this, there's also this 289 00:10:16,960 --> 00:10:18,600 kind of language that I've put together 290 00:10:18,600 --> 00:10:21,519 for describing modules because 291 00:10:21,519 --> 00:10:23,639 like if you ever try and read up about 292 00:10:23,639 --> 00:10:25,159 how to structure a code base, you're 293 00:10:25,159 --> 00:10:26,840 going to find 20 different approaches 294 00:10:26,840 --> 00:10:28,639 and they're all going to be called DDD. 295 00:10:28,639 --> 00:10:29,639 And 296 00:10:29,639 --> 00:10:31,919 like what you need is a consistent 297 00:10:31,919 --> 00:10:33,480 language that you can use in your team 298 00:10:33,480 --> 00:10:35,039 to talk about this stuff. And so, I have 299 00:10:35,039 --> 00:10:36,360 a little code base design skill that 300 00:10:36,360 --> 00:10:38,840 defines what locality is, defines what 301 00:10:38,840 --> 00:10:40,000 leverage is, 302 00:10:40,000 --> 00:10:41,960 defines what a seam is. I was using 303 00:10:41,960 --> 00:10:43,639 seams before they were cool. 304 00:10:43,639 --> 00:10:44,720 And 305 00:10:44,720 --> 00:10:47,240 what locality means is kind of 306 00:10:47,240 --> 00:10:50,559 how uh well located together all of the 307 00:10:50,559 --> 00:10:53,360 code is. How can you change like a small 308 00:10:53,360 --> 00:10:54,960 change in one module and have it ripple 309 00:10:54,960 --> 00:10:56,080 out? 310 00:10:56,080 --> 00:10:58,279 And also leverage is what you get when 311 00:10:58,279 --> 00:11:00,320 you have a deep module because the 312 00:11:00,320 --> 00:11:01,720 caller, the person who's actually 313 00:11:01,720 --> 00:11:05,679 calling that module, gets a lot of value 314 00:11:05,679 --> 00:11:07,440 of calling a simple function. Both of 315 00:11:07,440 --> 00:11:09,399 those are very good in code bases and 316 00:11:09,399 --> 00:11:12,000 good for agents, too, it turns out. 317 00:11:12,000 --> 00:11:13,200 But 318 00:11:13,200 --> 00:11:14,559 I'm sort of describing all of these 319 00:11:14,559 --> 00:11:17,279 high-falutin coding standards, but how 320 00:11:17,279 --> 00:11:19,799 do we actually make sure the agent does 321 00:11:19,799 --> 00:11:21,919 them, right? How do you make sure the 322 00:11:21,919 --> 00:11:24,080 agent creates deep modules and creates 323 00:11:24,080 --> 00:11:26,159 good tests and doesn't write these crap 324 00:11:26,159 --> 00:11:27,799 tautological ones or structure sensitive 325 00:11:27,799 --> 00:11:32,159 tests? 326 00:11:32,159 --> 00:11:35,080 I do think most people get this wrong. 327 00:11:35,080 --> 00:11:36,799 And 328 00:11:36,799 --> 00:11:39,440 my first piece of advice is don't put 329 00:11:39,440 --> 00:11:41,440 coding standards in your implementer 330 00:11:41,440 --> 00:11:43,320 agent. Okay? 331 00:11:43,320 --> 00:11:45,240 Let me explain this. 332 00:11:45,240 --> 00:11:46,480 If you imagine the implementer agent 333 00:11:46,480 --> 00:11:47,919 kind of looks like this. 334 00:11:47,919 --> 00:11:49,919 Where this is all of the things the 335 00:11:49,919 --> 00:11:51,399 agent needs to be able to do in its 336 00:11:51,399 --> 00:11:53,200 single context window. It needs to be 337 00:11:53,200 --> 00:11:54,759 able to explore, 338 00:11:54,759 --> 00:11:56,039 like to look for the code that it's 339 00:11:56,039 --> 00:11:57,639 going to change. It then needs to 340 00:11:57,639 --> 00:11:59,799 actually change it, so make the updates 341 00:11:59,799 --> 00:12:02,159 to the files in the green. And then it 342 00:12:02,159 --> 00:12:04,200 needs some budget for actually debugging 343 00:12:04,200 --> 00:12:05,960 the thing. So if we're running those 344 00:12:05,960 --> 00:12:07,960 automated checks for actually checking 345 00:12:07,960 --> 00:12:10,720 and verifying that it works. 346 00:12:10,720 --> 00:12:13,159 Now this is quite a lot of work, it 347 00:12:13,159 --> 00:12:15,759 turns out. And if you try to impose your 348 00:12:15,759 --> 00:12:17,960 coding standards on it as well, it's 349 00:12:17,960 --> 00:12:21,279 going to perform worse. 350 00:12:21,279 --> 00:12:23,360 So implementation is overloaded. That's 351 00:12:23,360 --> 00:12:25,399 the mental model I want you to have. 352 00:12:25,399 --> 00:12:27,879 And so can we find a way to impose those 353 00:12:27,879 --> 00:12:29,639 coding standards in a way that isn't so 354 00:12:29,639 --> 00:12:30,960 overloaded? 355 00:12:30,960 --> 00:12:33,279 Well, this is my effort. This is my code 356 00:12:33,279 --> 00:12:34,600 review skill. 357 00:12:34,600 --> 00:12:36,480 And it receives a diff. 358 00:12:36,480 --> 00:12:39,120 And it reads a file inside the 359 00:12:39,120 --> 00:12:41,279 repository called coding standards, 360 00:12:41,279 --> 00:12:43,559 which you can write, you can customize. 361 00:12:43,559 --> 00:12:45,399 And then it checks if the code follows 362 00:12:45,399 --> 00:12:47,360 those standards. 363 00:12:47,360 --> 00:12:49,960 And so if you look at the reviewer agent 364 00:12:49,960 --> 00:12:52,399 here, it also crucially runs it in a sub 365 00:12:52,399 --> 00:12:54,639 agent. So it's got its own context 366 00:12:54,639 --> 00:12:56,159 window to kind of handle here. It's got 367 00:12:56,159 --> 00:12:57,679 its own budget. 368 00:12:57,679 --> 00:12:59,200 This one, 369 00:12:59,200 --> 00:13:01,399 it needs to do some exploration, right? 370 00:13:01,399 --> 00:13:03,200 Because sure, it receives the diff, so 371 00:13:03,200 --> 00:13:05,120 it knows exactly where it's located, 372 00:13:05,120 --> 00:13:06,519 where the code is. But it should 373 00:13:06,519 --> 00:13:08,080 probably do a bit of exploration just so 374 00:13:08,080 --> 00:13:09,759 it has the wider context, understands 375 00:13:09,759 --> 00:13:11,799 the code. But it doesn't need to do any 376 00:13:11,799 --> 00:13:14,120 implementation. Doesn't need to do any 377 00:13:14,120 --> 00:13:16,639 debugging. So while implementation is 378 00:13:16,639 --> 00:13:18,720 overloaded, review is actually 379 00:13:18,720 --> 00:13:20,960 underloaded. So it doesn't have that 380 00:13:20,960 --> 00:13:22,879 many jobs to do. This means you can pile 381 00:13:22,879 --> 00:13:24,879 in a bunch of coding standards to it, 382 00:13:24,879 --> 00:13:26,639 and And will do a much better job than 383 00:13:26,639 --> 00:13:29,879 if you try to do it with implement. 384 00:13:29,879 --> 00:13:32,120 So, I think of this, and this is 385 00:13:32,120 --> 00:13:34,080 uncomfortable, right? Cuz we we all want 386 00:13:34,080 --> 00:13:35,840 to be able to just get good code out the 387 00:13:35,840 --> 00:13:37,279 first time. 388 00:13:37,279 --> 00:13:39,480 But I think of this as the two-part 389 00:13:39,480 --> 00:13:41,799 process for writing good code, which is 390 00:13:41,799 --> 00:13:44,159 implement, you make it work, and then 391 00:13:44,159 --> 00:13:46,240 code review, you actually make it good. 392 00:13:46,240 --> 00:13:48,759 You impose your coding standards. And 393 00:13:48,759 --> 00:13:50,440 for, you know, the retro developers 394 00:13:50,440 --> 00:13:51,960 among us, this is essentially a 395 00:13:51,960 --> 00:13:54,240 red-green-refactor approach. We use one 396 00:13:54,240 --> 00:13:56,200 context window to make it okay, do the 397 00:13:56,200 --> 00:13:58,480 red-green, and then we do another 398 00:13:58,480 --> 00:14:00,720 context window to refactor it. 399 00:14:00,720 --> 00:14:02,200 That's at least how it works in my head, 400 00:14:02,200 --> 00:14:04,600 and it's been very successful for me. 401 00:14:04,600 --> 00:14:06,559 This means that you when you have coding 402 00:14:06,559 --> 00:14:08,840 standards, you don't put them in global 403 00:14:08,840 --> 00:14:11,720 scope. You don't put them in agents.md, 404 00:14:11,720 --> 00:14:13,480 cuz then they sort of drown out your 405 00:14:13,480 --> 00:14:15,320 implementer. It may read them, it may 406 00:14:15,320 --> 00:14:16,480 not. 407 00:14:16,480 --> 00:14:18,200 You put them in coding standards.md, and 408 00:14:18,200 --> 00:14:20,600 that way just the code review agent does 409 00:14:20,600 --> 00:14:22,559 it. 410 00:14:22,559 --> 00:14:24,039 Now, I've got another idea here, which 411 00:14:24,039 --> 00:14:25,000 is 412 00:14:25,000 --> 00:14:25,879 I've been talking to lots of people 413 00:14:25,879 --> 00:14:26,720 today. 414 00:14:26,720 --> 00:14:27,759 Lots of people saying, you know, I've 415 00:14:27,759 --> 00:14:29,279 been talking about review and automated 416 00:14:29,279 --> 00:14:31,080 review, increase, you know, fixing the 417 00:14:31,080 --> 00:14:32,399 PR bottleneck. 418 00:14:32,399 --> 00:14:34,279 So many folks say, "Oh yeah, we just use 419 00:14:34,279 --> 00:14:36,360 a third-party service. We use a cursor 420 00:14:36,360 --> 00:14:37,919 bug bot. We use code rabbit or something 421 00:14:37,919 --> 00:14:38,919 like that." 422 00:14:38,919 --> 00:14:40,639 I think that 423 00:14:40,639 --> 00:14:43,200 I've I've really tried to make a 424 00:14:43,200 --> 00:14:45,919 generic code review skill in the past 425 00:14:45,919 --> 00:14:47,879 that finds all the bugs and does 426 00:14:47,879 --> 00:14:49,759 security review and that kind of thing. 427 00:14:49,759 --> 00:14:51,799 It turns out it's really, really hard 428 00:14:51,799 --> 00:14:54,200 because you either make it too general 429 00:14:54,200 --> 00:14:55,919 and it just gives you false positives 430 00:14:55,919 --> 00:14:57,000 that aren't actually relevant to your 431 00:14:57,000 --> 00:14:59,720 use case, or you make it too specific. 432 00:14:59,720 --> 00:15:01,519 You say, "Okay, find all the TypeScript 433 00:15:01,519 --> 00:15:02,879 stuff," and then Rust people can't use 434 00:15:02,879 --> 00:15:03,839 it. 435 00:15:03,839 --> 00:15:04,879 So, 436 00:15:04,879 --> 00:15:07,399 I would say don't outsource automated 437 00:15:07,399 --> 00:15:09,559 review. Build your own. Build up your 438 00:15:09,559 --> 00:15:11,519 own coding standards over time. Share 439 00:15:11,519 --> 00:15:13,600 them across your team. And if you have 440 00:15:13,600 --> 00:15:15,559 an opportunity to impose coding 441 00:15:15,559 --> 00:15:16,799 standards, if you've got some docs 442 00:15:16,799 --> 00:15:18,600 sitting around that no one reads, this 443 00:15:18,600 --> 00:15:21,039 is the place to put them in. 444 00:15:21,039 --> 00:15:22,600 And also, 445 00:15:22,600 --> 00:15:24,960 when you have this automated reviewer, 446 00:15:24,960 --> 00:15:26,919 a really natural inclination for lots of 447 00:15:26,919 --> 00:15:28,679 people is to say, "Oh yeah, my code 448 00:15:28,679 --> 00:15:30,559 review agent, what it does is it reads 449 00:15:30,559 --> 00:15:32,039 the code and then it comments on the 450 00:15:32,039 --> 00:15:34,039 PR." 451 00:15:34,039 --> 00:15:35,440 Uh so, 452 00:15:35,440 --> 00:15:37,240 what your code review agent is doing in 453 00:15:37,240 --> 00:15:39,600 that case is it's providing more work 454 00:15:39,600 --> 00:15:41,200 for the human reviewer. The human 455 00:15:41,200 --> 00:15:42,799 reviewer then has to read all of these 456 00:15:42,799 --> 00:15:44,279 verbose comments and figure out, "Okay, 457 00:15:44,279 --> 00:15:45,480 do we implement this? Do we implement 458 00:15:45,480 --> 00:15:46,440 that?" 459 00:15:46,440 --> 00:15:49,080 The reviewer should commit. It should 460 00:15:49,080 --> 00:15:50,960 make fixes. So, it should actually 461 00:15:50,960 --> 00:15:53,080 change the things that it finds. Cuz 462 00:15:53,080 --> 00:15:55,039 then when the human comes around, you're 463 00:15:55,039 --> 00:15:57,799 reviewing a really nice artifact. If it 464 00:15:57,799 --> 00:15:59,320 finds anything that it has any questions 465 00:15:59,320 --> 00:16:01,679 over, then of course it can comment, but 466 00:16:01,679 --> 00:16:04,320 the default should be commits. 467 00:16:04,320 --> 00:16:07,000 Stop trying to one-shot good code. 468 00:16:07,000 --> 00:16:09,080 Stop trying to make the implementer 469 00:16:09,080 --> 00:16:11,120 agent the only thing that you do and go, 470 00:16:11,120 --> 00:16:12,279 "Okay, I'm going to force it to be 471 00:16:12,279 --> 00:16:13,399 amazing." 472 00:16:13,399 --> 00:16:14,759 It takes a little bit of, you know, 473 00:16:14,759 --> 00:16:16,080 thinking your way out of there, but once 474 00:16:16,080 --> 00:16:17,480 you realize it, 475 00:16:17,480 --> 00:16:19,399 it is fabulous. 476 00:16:19,399 --> 00:16:20,720 So, okay. 477 00:16:20,720 --> 00:16:22,480 With all of that process, we've run our 478 00:16:22,480 --> 00:16:24,120 automated checks, we've now run our 479 00:16:24,120 --> 00:16:25,919 automated review to make sure the 480 00:16:25,919 --> 00:16:28,039 automated checks aren't lying. 481 00:16:28,039 --> 00:16:30,759 How do we then maximize the PR's quality 482 00:16:30,759 --> 00:16:32,480 in terms of human review? How do we get 483 00:16:32,480 --> 00:16:34,960 it like working the best it can? 484 00:16:34,960 --> 00:16:38,360 So, we need a human-friendly PR. And 485 00:16:38,360 --> 00:16:40,320 this is a new skill 486 00:16:40,320 --> 00:16:41,960 coming into the repo, which is currently 487 00:16:41,960 --> 00:16:43,240 in progress, but I'll be releasing it 488 00:16:43,240 --> 00:16:45,559 soon, which is the PR skill. This is one 489 00:16:45,559 --> 00:16:47,120 I've been mulling over for a long, long 490 00:16:47,120 --> 00:16:48,519 time. Haven't quite figured out what the 491 00:16:48,519 --> 00:16:50,000 state of the art is. 492 00:16:50,000 --> 00:16:52,839 And I've realized the best way to make a 493 00:16:52,839 --> 00:16:54,480 PR skill is just to steal all the best 494 00:16:54,480 --> 00:16:57,000 ideas that everyone's got. And 495 00:16:57,000 --> 00:16:58,519 it's very nice. 496 00:16:58,519 --> 00:17:02,159 Now, what does a good PR body look like? 497 00:17:02,159 --> 00:17:03,559 How would you recreate this skill on 498 00:17:03,559 --> 00:17:04,680 your own? 499 00:17:04,680 --> 00:17:06,720 Well, the first principle is some 500 00:17:06,720 --> 00:17:09,078 reviews are more important than others. 501 00:17:09,078 --> 00:17:15,400 Not every review is essential, right? 502 00:17:15,400 --> 00:17:16,519 once you understand this, you realize, 503 00:17:16,519 --> 00:17:18,680 "Okay, that means I can focus my energy 504 00:17:18,680 --> 00:17:21,039 on the really important reviews. But how 505 00:17:21,039 --> 00:17:23,680 do we categorize them? Well, 506 00:17:23,680 --> 00:17:25,799 you got to think about the PR as using 507 00:17:25,799 --> 00:17:28,240 this AWS terminology, which is is it a 508 00:17:28,240 --> 00:17:31,480 one-way door or is it a two-way door? 509 00:17:31,480 --> 00:17:33,319 Now, most PRs that you have will be 510 00:17:33,319 --> 00:17:35,920 two-way doors. You can merge the PR and 511 00:17:35,920 --> 00:17:38,359 then always pull it back later. It's the 512 00:17:38,359 --> 00:17:40,400 glorious benefit of being a software 513 00:17:40,400 --> 00:17:42,200 engineer as opposed to a civil engineer, 514 00:17:42,200 --> 00:17:43,559 right? Mostly when you're a civil 515 00:17:43,559 --> 00:17:45,119 engineer, it's a it's a one-way door, 516 00:17:45,119 --> 00:17:46,480 right? If you get something wrong, that 517 00:17:46,480 --> 00:17:48,720 bridge is going down. But, 518 00:17:48,720 --> 00:17:50,880 if you have a two-way door, it means 519 00:17:50,880 --> 00:17:53,079 that you can easily revert the change. 520 00:17:53,079 --> 00:17:56,279 Now, that might be more um 521 00:17:56,279 --> 00:17:57,599 a bit more nuanced than you might 522 00:17:57,599 --> 00:17:59,279 expect. It might be that a very simple 523 00:17:59,279 --> 00:18:01,720 change accidentally blasts out an email 524 00:18:01,720 --> 00:18:03,440 to 60,000 people or something, in which 525 00:18:03,440 --> 00:18:05,160 case that is a one-way door. You want to 526 00:18:05,160 --> 00:18:06,799 review that very, very carefully. 527 00:18:06,799 --> 00:18:08,920 Involves expensive migrations or data 528 00:18:08,920 --> 00:18:10,960 loss, that is a one-way door. Review the 529 00:18:10,960 --> 00:18:13,079 hell out of that PR. 530 00:18:13,079 --> 00:18:15,400 But also, tied onto that, we need to 531 00:18:15,400 --> 00:18:17,880 understand radius of this PR. What can 532 00:18:17,880 --> 00:18:20,920 go wrong? And if things do go wrong, how 533 00:18:20,920 --> 00:18:23,960 bad is it? And this means I end up with 534 00:18:23,960 --> 00:18:25,759 a nice little sort of summary right at 535 00:18:25,759 --> 00:18:27,480 the bottom of all of my PRs, which is 536 00:18:27,480 --> 00:18:29,519 the merge danger. I can see this one is 537 00:18:29,519 --> 00:18:31,759 a two-way door. Its blast radius is 538 00:18:31,759 --> 00:18:34,559 localized, and so I can see fantastic, I 539 00:18:34,559 --> 00:18:36,000 don't need to pay that much attention to 540 00:18:36,000 --> 00:18:37,160 this. I'm just going to sort of review 541 00:18:37,160 --> 00:18:38,400 it a little. 542 00:18:38,400 --> 00:18:40,440 That's really important. 543 00:18:40,440 --> 00:18:42,240 Next, we need to understand what the PR 544 00:18:42,240 --> 00:18:44,559 is even doing, right? And I've tried 545 00:18:44,559 --> 00:18:46,640 lots of different ways of figuring this 546 00:18:46,640 --> 00:18:48,519 out, and the best thing I've come up 547 00:18:48,519 --> 00:18:51,759 with is using pseudo code. Now, 548 00:18:51,759 --> 00:18:54,200 a huge um 549 00:18:54,200 --> 00:18:56,039 points of gratitude here to the show me 550 00:18:56,039 --> 00:18:58,240 skill from the human layer skills repo 551 00:18:58,240 --> 00:19:00,119 by Dex Hadley. This is a phenomenal 552 00:19:00,119 --> 00:19:02,279 skill that just essentially dispenses 553 00:19:02,279 --> 00:19:05,079 with most text and shows things to you 554 00:19:05,079 --> 00:19:07,920 in images and diagrams instead. This 555 00:19:07,920 --> 00:19:10,160 makes it a lot easier to grasp actually 556 00:19:10,160 --> 00:19:12,480 what's changing and why it's changing. 557 00:19:12,480 --> 00:19:14,480 So, you get the kind of standard sort of 558 00:19:14,480 --> 00:19:16,799 set of like mermaid diagrams and UML 559 00:19:16,799 --> 00:19:18,599 for, you know, this is a sort of 560 00:19:18,599 --> 00:19:20,880 sequence of things that happened. You 561 00:19:20,880 --> 00:19:22,680 also just get these lovely simple ones 562 00:19:22,680 --> 00:19:24,160 like this, for instance. We can look at 563 00:19:24,160 --> 00:19:25,599 this and go, "Okay, we're working in a 564 00:19:25,599 --> 00:19:27,680 CLI. We can see a new command has been 565 00:19:27,680 --> 00:19:30,000 added." And we've got two new things, 566 00:19:30,000 --> 00:19:31,680 little flags up the top here. Just 567 00:19:31,680 --> 00:19:33,200 little summaries like this. It really 568 00:19:33,200 --> 00:19:34,599 does make a massive difference. It's 569 00:19:34,599 --> 00:19:36,440 hard to overstate. 570 00:19:36,440 --> 00:19:38,200 So, you're trying to 571 00:19:38,200 --> 00:19:41,400 like make understanding the why as fast 572 00:19:41,400 --> 00:19:43,680 as possible. 573 00:19:43,680 --> 00:19:45,839 And I think a third principle here is 574 00:19:45,839 --> 00:19:47,559 something you should be thinking about 575 00:19:47,559 --> 00:19:49,680 whenever you do human review. 576 00:19:49,680 --> 00:19:53,680 Because because we're not doing like 577 00:19:53,680 --> 00:19:55,359 um 578 00:19:55,359 --> 00:19:56,240 because 579 00:19:56,240 --> 00:19:59,200 our processes now are so 580 00:19:59,200 --> 00:20:01,599 sort of streamlined and all we're all 581 00:20:01,599 --> 00:20:03,559 collaborating around these same skill 582 00:20:03,559 --> 00:20:05,880 files, these same steering files. We're 583 00:20:05,880 --> 00:20:07,359 all building an environment for our 584 00:20:07,359 --> 00:20:09,519 agents to operate in together. 585 00:20:09,519 --> 00:20:11,400 You should think of the process that 586 00:20:11,400 --> 00:20:13,880 produces your code as just as important 587 00:20:13,880 --> 00:20:15,480 as the code itself. 588 00:20:15,480 --> 00:20:17,039 In other words, when you do a human 589 00:20:17,039 --> 00:20:18,400 review, you're not just reviewing the 590 00:20:18,400 --> 00:20:20,119 code, you're reviewing the system that 591 00:20:20,119 --> 00:20:21,759 creates it. 592 00:20:21,759 --> 00:20:23,920 And the theory here is that you never 593 00:20:23,920 --> 00:20:26,440 want to write the same comment twice. 594 00:20:26,440 --> 00:20:28,400 Right? You never want to catch the agent 595 00:20:28,400 --> 00:20:31,599 doing the same thing over two PRs. And 596 00:20:31,599 --> 00:20:33,440 so, what's the mechanism by which you 597 00:20:33,440 --> 00:20:36,200 can make your human review matter? 598 00:20:36,200 --> 00:20:37,640 Well, this is a new skill. This is 599 00:20:37,640 --> 00:20:38,920 called retro. 600 00:20:38,920 --> 00:20:40,920 Retro for retrospective. You essentially 601 00:20:40,920 --> 00:20:43,079 take a session that you've done. It can 602 00:20:43,079 --> 00:20:45,039 either be like a single um agent 603 00:20:45,039 --> 00:20:46,680 session, or it can be a PR plus the 604 00:20:46,680 --> 00:20:48,599 session, or you can just get it to look 605 00:20:48,599 --> 00:20:50,839 at Okay, look at all of the PRs that 606 00:20:50,839 --> 00:20:52,759 we've done over the last week, all of 607 00:20:52,759 --> 00:20:55,079 the reviews, pull them all in. 608 00:20:55,079 --> 00:20:56,759 Let's do a retrospective on them. 609 00:20:56,759 --> 00:20:59,400 And it will suggest automated checks and 610 00:20:59,400 --> 00:21:02,400 coding standards to make the next one 611 00:21:02,400 --> 00:21:04,359 better. So, this is the compounding 612 00:21:04,359 --> 00:21:06,160 effect where you're essentially 613 00:21:06,160 --> 00:21:08,480 by doing human review, you're making the 614 00:21:08,480 --> 00:21:10,720 quality of the next human review higher, 615 00:21:10,720 --> 00:21:12,680 and you're sort of saving less work from 616 00:21:12,680 --> 00:21:14,160 yourself next time. 617 00:21:14,160 --> 00:21:15,799 And Retro is a really smart skill. It 618 00:21:15,799 --> 00:21:17,880 adds a bunch of stuff. So, obviously, it 619 00:21:17,880 --> 00:21:19,799 suggests automated checks. 620 00:21:19,799 --> 00:21:21,720 It suggests updates to coding 621 00:21:21,720 --> 00:21:24,200 standards.md. It does other smart stuff, 622 00:21:24,200 --> 00:21:25,880 too. So, it looks at navigation 623 00:21:25,880 --> 00:21:28,279 pointers. How easily did the agent find 624 00:21:28,279 --> 00:21:30,119 its information? Can we provide a 625 00:21:30,119 --> 00:21:32,400 pointer inside agents.md to help it out 626 00:21:32,400 --> 00:21:34,240 next time? 627 00:21:34,240 --> 00:21:35,880 It looks at tool economy. Are there 628 00:21:35,880 --> 00:21:37,480 different tools that we're using in the 629 00:21:37,480 --> 00:21:39,720 session that can, you know, could be 630 00:21:39,720 --> 00:21:41,559 made more token efficient? It's amazing 631 00:21:41,559 --> 00:21:43,279 how many things this catches, actually, 632 00:21:43,279 --> 00:21:44,799 cuz those are often really hard to debug 633 00:21:44,799 --> 00:21:46,440 from the outside. 634 00:21:46,440 --> 00:21:48,599 It just looks at bloat, as well. So, are 635 00:21:48,599 --> 00:21:50,200 there bloated steering files? Are there 636 00:21:50,200 --> 00:21:51,920 bloated skills that contribute to these 637 00:21:51,920 --> 00:21:54,119 bad results? Can we make them more 638 00:21:54,119 --> 00:21:56,200 organized? 639 00:21:56,200 --> 00:21:57,400 So, 640 00:21:57,400 --> 00:21:58,599 that's the goal. It's to make human 641 00:21:58,599 --> 00:21:59,839 review faster. 642 00:21:59,839 --> 00:22:01,920 We do that by layering up automated 643 00:22:01,920 --> 00:22:04,240 checks. We're layering up automated 644 00:22:04,240 --> 00:22:06,720 review, and we make the human review as 645 00:22:06,720 --> 00:22:09,119 painless, as simple, and as kind of 646 00:22:09,119 --> 00:22:11,200 optional as we need it to. You really 647 00:22:11,200 --> 00:22:12,799 don't need to review every single 648 00:22:12,799 --> 00:22:14,119 two-way door. 649 00:22:14,119 --> 00:22:16,799 Every single one-way door you do. 650 00:22:16,799 --> 00:22:17,599 So, 651 00:22:17,599 --> 00:22:18,960 these are my skills. 652 00:22:18,960 --> 00:22:21,319 I'm here at dot dev \{{}slash} skills, and 653 00:22:21,319 --> 00:22:24,000 we're shipping version 1.3 this week. It 654 00:22:24,000 --> 00:22:25,519 has been glorious hanging out with you. 655 00:22:25,519 --> 00:22:26,880 It's been a really nice conference. I'm 656 00:22:26,880 --> 00:22:28,319 going to be outside in the lobby if 657 00:22:28,319 --> 00:22:30,799 anyone wants to have a chat. Uh thank 658 00:22:30,799 --> 00:22:32,480 you so much for having me. Thank you, 659 00:22:32,480 --> 00:22:35,480 Paris.