WEBVTT Kind: captions Language: en 00:00:04.689 --> 00:00:09.160 align:start position:0% everybody<00:00:06.729> thanks<00:00:07.729> for<00:00:07.819> coming<00:00:08.029> to<00:00:08.149> the<00:00:08.330> e2 00:00:09.160 --> 00:00:09.170 align:start position:0% everybody thanks for coming to the e2 00:00:09.170 --> 00:00:10.930 align:start position:0% everybody thanks for coming to the e2 client<00:00:09.920> summit<00:00:10.219> I<00:00:10.280> know<00:00:10.340> there's<00:00:10.580> been<00:00:10.670> a<00:00:10.730> lot 00:00:10.930 --> 00:00:10.940 align:start position:0% client summit I know there's been a lot 00:00:10.940 --> 00:00:14.320 align:start position:0% client summit I know there's been a lot of<00:00:10.969> e<00:00:11.330> to<00:00:12.019> talks<00:00:12.320> so<00:00:12.530> maybe<00:00:12.830> you're<00:00:13.130> sick<00:00:14.089> of<00:00:14.210> it 00:00:14.320 --> 00:00:14.330 align:start position:0% of e to talks so maybe you're sick of it 00:00:14.330 --> 00:00:16.600 align:start position:0% of e to talks so maybe you're sick of it but<00:00:14.509> it's<00:00:15.320> really<00:00:15.620> special<00:00:15.950> to<00:00:16.070> be<00:00:16.219> here<00:00:16.460> this 00:00:16.600 --> 00:00:16.610 align:start position:0% but it's really special to be here this 00:00:16.610 --> 00:00:19.810 align:start position:0% but it's really special to be here this is<00:00:16.670> my<00:00:16.759> first<00:00:17.029> Def<00:00:17.240> Con<00:00:18.400> didn't<00:00:19.400> didn't<00:00:19.759> go 00:00:19.810 --> 00:00:19.820 align:start position:0% is my first Def Con didn't didn't go 00:00:19.820 --> 00:00:21.490 align:start position:0% is my first Def Con didn't didn't go last<00:00:19.970> year<00:00:20.240> I<00:00:20.300> really<00:00:20.630> wanted<00:00:20.869> to<00:00:21.020> all<00:00:21.349> the 00:00:21.490 --> 00:00:21.500 align:start position:0% last year I really wanted to all the 00:00:21.500 --> 00:00:24.310 align:start position:0% last year I really wanted to all the rest<00:00:21.680> of<00:00:21.829> prismatic<00:00:22.160> steam<00:00:22.640> went<00:00:22.880> so<00:00:23.320> it's 00:00:24.310 --> 00:00:24.320 align:start position:0% rest of prismatic steam went so it's 00:00:24.320 --> 00:00:27.249 align:start position:0% rest of prismatic steam went so it's truly<00:00:24.590> amazing<00:00:25.630> opportunity<00:00:26.630> to<00:00:26.689> be<00:00:26.809> here<00:00:27.020> so 00:00:27.249 --> 00:00:27.259 align:start position:0% truly amazing opportunity to be here so 00:00:27.259 --> 00:00:30.940 align:start position:0% truly amazing opportunity to be here so what<00:00:28.250> I<00:00:28.279> want<00:00:28.460> to<00:00:28.489> talk<00:00:28.610> about<00:00:28.790> is<00:00:29.259> our<00:00:30.259> test 00:00:30.940 --> 00:00:30.950 align:start position:0% what I want to talk about is our test 00:00:30.950 --> 00:00:32.709 align:start position:0% what I want to talk about is our test net<00:00:31.130> we've<00:00:31.309> we've<00:00:31.759> been<00:00:31.969> running<00:00:32.149> one<00:00:32.390> for 00:00:32.709 --> 00:00:32.719 align:start position:0% net we've we've been running one for 00:00:32.719 --> 00:00:35.350 align:start position:0% net we've we've been running one for some<00:00:33.380> time<00:00:33.590> now<00:00:33.800> and<00:00:33.829> we've<00:00:34.760> actually<00:00:35.030> learned 00:00:35.350 --> 00:00:35.360 align:start position:0% some time now and we've actually learned 00:00:35.360 --> 00:00:39.160 align:start position:0% some time now and we've actually learned a<00:00:35.420> lot<00:00:35.510> so<00:00:35.750> let's<00:00:35.899> dive<00:00:36.110> into<00:00:36.320> it<00:00:37.960> first<00:00:38.960> of<00:00:39.109> all 00:00:39.160 --> 00:00:39.170 align:start position:0% a lot so let's dive into it first of all 00:00:39.170 --> 00:00:41.259 align:start position:0% a lot so let's dive into it first of all who<00:00:39.469> who<00:00:39.949> are<00:00:39.980> prismatic<00:00:40.609> labs<00:00:40.850> or<00:00:41.030> who<00:00:41.149> is 00:00:41.259 --> 00:00:41.269 align:start position:0% who who are prismatic labs or who is 00:00:41.269 --> 00:00:44.470 align:start position:0% who who are prismatic labs or who is prismatic<00:00:41.629> labs<00:00:42.019> were<00:00:42.980> a<00:00:43.370> self-organized 00:00:44.470 --> 00:00:44.480 align:start position:0% prismatic labs were a self-organized 00:00:44.480 --> 00:00:48.459 align:start position:0% prismatic labs were a self-organized group<00:00:44.839> of<00:00:46.149> just<00:00:47.149> dedicated<00:00:47.839> bakhtin 00:00:48.459 --> 00:00:48.469 align:start position:0% group of just dedicated bakhtin 00:00:48.469 --> 00:00:50.049 align:start position:0% group of just dedicated bakhtin engineers<00:00:48.890> people<00:00:49.129> who<00:00:49.370> really<00:00:49.550> wanted<00:00:49.850> to<00:00:49.909> be 00:00:50.049 --> 00:00:50.059 align:start position:0% engineers people who really wanted to be 00:00:50.059 --> 00:00:52.630 align:start position:0% engineers people who really wanted to be a<00:00:50.359> part<00:00:50.719> of<00:00:50.839> scaling<00:00:51.320> etherium<00:00:51.769> and<00:00:52.129> sort<00:00:52.579> of 00:00:52.630 --> 00:00:52.640 align:start position:0% a part of scaling etherium and sort of 00:00:52.640 --> 00:00:56.309 align:start position:0% a part of scaling etherium and sort of making<00:00:52.850> this<00:00:53.179> reach<00:00:54.370> global<00:00:55.370> adoption 00:00:56.309 --> 00:00:56.319 align:start position:0% making this reach global adoption 00:00:56.319 --> 00:01:00.250 align:start position:0% making this reach global adoption capacities<00:00:57.319> so<00:00:58.329> there<00:00:59.329> are<00:00:59.449> six<00:00:59.690> of<00:00:59.809> us<00:00:59.899> today 00:01:00.250 --> 00:01:00.260 align:start position:0% capacities so there are six of us today 00:01:00.260 --> 00:01:02.950 align:start position:0% capacities so there are six of us today most<00:01:01.010> of<00:01:01.159> us<00:01:01.219> are<00:01:01.339> here<00:01:01.670> so<00:01:02.120> come<00:01:02.449> say<00:01:02.629> hi<00:01:02.780> if 00:01:02.950 --> 00:01:02.960 align:start position:0% most of us are here so come say hi if 00:01:02.960 --> 00:01:04.890 align:start position:0% most of us are here so come say hi if you<00:01:03.050> see<00:01:03.260> us<00:01:03.409> if<00:01:03.800> you<00:01:03.890> don't<00:01:04.070> know<00:01:04.159> us<00:01:04.250> already 00:01:04.890 --> 00:01:04.900 align:start position:0% you see us if you don't know us already 00:01:04.900 --> 00:01:08.080 align:start position:0% you see us if you don't know us already we'd<00:01:05.900> love<00:01:06.050> to<00:01:06.080> chat<00:01:06.850> what's<00:01:07.850> interesting 00:01:08.080 --> 00:01:08.090 align:start position:0% we'd love to chat what's interesting 00:01:08.090 --> 00:01:11.700 align:start position:0% we'd love to chat what's interesting about<00:01:08.420> this<00:01:09.080> group<00:01:09.350> is<00:01:09.590> that<00:01:09.950> we<00:01:10.250> all<00:01:10.580> formed 00:01:11.700 --> 00:01:11.710 align:start position:0% about this group is that we all formed 00:01:11.710 --> 00:01:14.290 align:start position:0% about this group is that we all formed really<00:01:12.710> organically<00:01:13.220> online<00:01:13.700> nobody<00:01:14.120> knew 00:01:14.290 --> 00:01:14.300 align:start position:0% really organically online nobody knew 00:01:14.300 --> 00:01:16.840 align:start position:0% really organically online nobody knew anybody<00:01:14.600> before<00:01:15.110> any<00:01:15.290> of<00:01:15.380> this<00:01:15.560> and<00:01:15.890> we<00:01:16.610> were 00:01:16.840 --> 00:01:16.850 align:start position:0% anybody before any of this and we were 00:01:16.850 --> 00:01:20.320 align:start position:0% anybody before any of this and we were just<00:01:17.060> inspired<00:01:17.540> by<00:01:17.900> Vitalik<00:01:18.890> wiki<00:01:19.790> article 00:01:20.320 --> 00:01:20.330 align:start position:0% just inspired by Vitalik wiki article 00:01:20.330 --> 00:01:23.020 align:start position:0% just inspired by Vitalik wiki article about<00:01:20.660> sharding<00:01:21.200> and<00:01:21.440> wanted<00:01:22.430> to<00:01:22.460> start 00:01:23.020 --> 00:01:23.030 align:start position:0% about sharding and wanted to start 00:01:23.030 --> 00:01:24.580 align:start position:0% about sharding and wanted to start making<00:01:23.390> moves<00:01:23.630> on<00:01:23.810> that<00:01:23.960> because<00:01:24.170> we<00:01:24.290> felt 00:01:24.580 --> 00:01:24.590 align:start position:0% making moves on that because we felt 00:01:24.590 --> 00:01:28.000 align:start position:0% making moves on that because we felt that<00:01:24.680> you<00:01:25.640> know<00:01:25.730> at<00:01:25.850> the<00:01:26.060> peak<00:01:26.660> of<00:01:26.900> 2017<00:01:27.740> when 00:01:28.000 --> 00:01:28.010 align:start position:0% that you know at the peak of 2017 when 00:01:28.010 --> 00:01:30.430 align:start position:0% that you know at the peak of 2017 when things<00:01:28.310> were<00:01:28.520> really<00:01:28.700> like<00:01:29.570> maxing<00:01:30.020> out<00:01:30.110> that 00:01:30.430 --> 00:01:30.440 align:start position:0% things were really like maxing out that 00:01:30.440 --> 00:01:32.980 align:start position:0% things were really like maxing out that we<00:01:30.590> needed<00:01:31.190> something<00:01:31.700> soon<00:01:32.510> to<00:01:32.690> keep<00:01:32.840> this 00:01:32.980 --> 00:01:32.990 align:start position:0% we needed something soon to keep this 00:01:32.990 --> 00:01:35.560 align:start position:0% we needed something soon to keep this momentum<00:01:33.320> going<00:01:33.680> so<00:01:34.120> we<00:01:35.120> were<00:01:35.270> one<00:01:35.420> of<00:01:35.450> the 00:01:35.560 --> 00:01:35.570 align:start position:0% momentum going so we were one of the 00:01:35.570 --> 00:01:37.690 align:start position:0% momentum going so we were one of the first<00:01:35.750> teams<00:01:35.900> to<00:01:36.050> come<00:01:36.260> together<00:01:36.380> to<00:01:36.860> start 00:01:37.690 --> 00:01:37.700 align:start position:0% first teams to come together to start 00:01:37.700 --> 00:01:40.330 align:start position:0% first teams to come together to start building<00:01:38.320> sharding<00:01:39.320> and<00:01:39.500> then<00:01:39.830> the<00:01:40.010> proof<00:01:40.160> of 00:01:40.330 --> 00:01:40.340 align:start position:0% building sharding and then the proof of 00:01:40.340 --> 00:01:42.280 align:start position:0% building sharding and then the proof of stake<00:01:40.550> at<00:01:40.820> Casper<00:01:41.210> all<00:01:41.810> these<00:01:42.020> things<00:01:42.200> to 00:01:42.280 --> 00:01:42.290 align:start position:0% stake at Casper all these things to 00:01:42.290 --> 00:01:44.190 align:start position:0% stake at Casper all these things to merge<00:01:42.440> together<00:01:42.650> so<00:01:43.040> it's<00:01:43.190> been<00:01:43.400> pretty 00:01:44.190 --> 00:01:44.200 align:start position:0% merge together so it's been pretty 00:01:44.200 --> 00:01:48.670 align:start position:0% merge together so it's been pretty remarkable<00:01:45.200> journey<00:01:45.470> so<00:01:45.710> far<00:01:47.470> one<00:01:48.470> of<00:01:48.560> our 00:01:48.670 --> 00:01:48.680 align:start position:0% remarkable journey so far one of our 00:01:48.680 --> 00:01:51.940 align:start position:0% remarkable journey so far one of our biggest<00:01:48.890> achievements<00:01:49.550> is<00:01:49.730> to<00:01:50.540> run<00:01:51.530> an<00:01:51.740> actual 00:01:51.940 --> 00:01:51.950 align:start position:0% biggest achievements is to run an actual 00:01:51.950 --> 00:01:56.050 align:start position:0% biggest achievements is to run an actual test<00:01:52.430> net<00:01:52.640> this<00:01:53.480> has<00:01:53.690> been<00:01:54.550> we<00:01:55.550> started<00:01:56.000> doing 00:01:56.050 --> 00:01:56.060 align:start position:0% test net this has been we started doing 00:01:56.060 --> 00:01:58.150 align:start position:0% test net this has been we started doing this<00:01:56.270> back<00:01:56.570> in<00:01:56.810> April<00:01:57.230> of<00:01:57.470> this<00:01:57.560> year<00:01:57.620> so<00:01:58.040> we've 00:01:58.150 --> 00:01:58.160 align:start position:0% this back in April of this year so we've 00:01:58.160 --> 00:01:59.680 align:start position:0% this back in April of this year so we've been<00:01:58.310> running<00:01:58.520> it<00:01:58.610> for<00:01:58.640> I<00:01:59.210> think<00:01:59.270> six<00:01:59.570> months 00:01:59.680 --> 00:01:59.690 align:start position:0% been running it for I think six months 00:01:59.690 --> 00:02:04.210 align:start position:0% been running it for I think six months now<00:02:01.420> what<00:02:02.420> what<00:02:02.810> this<00:02:02.990> comprised<00:02:03.410> what's<00:02:04.100> this 00:02:04.210 --> 00:02:04.220 align:start position:0% now what what this comprised what's this 00:02:04.220 --> 00:02:07.540 align:start position:0% now what what this comprised what's this comprised<00:02:04.670> of<00:02:04.850> is<00:02:05.740> several<00:02:06.740> beacon<00:02:07.370> chain 00:02:07.540 --> 00:02:07.550 align:start position:0% comprised of is several beacon chain 00:02:07.550 --> 00:02:09.490 align:start position:0% comprised of is several beacon chain nodes<00:02:07.820> running<00:02:08.420> in<00:02:08.690> different<00:02:09.349> environments 00:02:09.490 --> 00:02:09.500 align:start position:0% nodes running in different environments 00:02:09.500 --> 00:02:12.670 align:start position:0% nodes running in different environments different<00:02:10.039> areas<00:02:10.519> of<00:02:10.849> the<00:02:10.970> world<00:02:11.379> it's<00:02:12.379> open 00:02:12.670 --> 00:02:12.680 align:start position:0% different areas of the world it's open 00:02:12.680 --> 00:02:15.009 align:start position:0% different areas of the world it's open to<00:02:12.769> anybody<00:02:13.010> so<00:02:13.489> anyone<00:02:13.909> with<00:02:14.209> a<00:02:14.569> computer<00:02:14.989> can 00:02:15.009 --> 00:02:15.019 align:start position:0% to anybody so anyone with a computer can 00:02:15.019 --> 00:02:18.130 align:start position:0% to anybody so anyone with a computer can join<00:02:15.409> it's<00:02:16.120> typically<00:02:17.120> runs<00:02:17.390> for 00:02:18.130 --> 00:02:18.140 align:start position:0% join it's typically runs for 00:02:18.140 --> 00:02:19.690 align:start position:0% join it's typically runs for several<00:02:18.710> weeks<00:02:18.830> or<00:02:19.069> several<00:02:19.130> months 00:02:19.690 --> 00:02:19.700 align:start position:0% several weeks or several months 00:02:19.700 --> 00:02:22.390 align:start position:0% several weeks or several months depending<00:02:20.180> on<00:02:20.450> if<00:02:21.170> we<00:02:21.350> run<00:02:21.560> into<00:02:21.680> some<00:02:22.069> fatal 00:02:22.390 --> 00:02:22.400 align:start position:0% depending on if we run into some fatal 00:02:22.400 --> 00:02:26.740 align:start position:0% depending on if we run into some fatal bug<00:02:22.670> or<00:02:23.000> if<00:02:23.510> there's<00:02:23.990> a<00:02:24.080> new<00:02:25.209> spec<00:02:26.209> update 00:02:26.740 --> 00:02:26.750 align:start position:0% bug or if there's a new spec update 00:02:26.750 --> 00:02:27.850 align:start position:0% bug or if there's a new spec update where<00:02:26.930> we<00:02:27.020> have<00:02:27.140> to<00:02:27.290> rewrite<00:02:27.470> everything 00:02:27.850 --> 00:02:27.860 align:start position:0% where we have to rewrite everything 00:02:27.860 --> 00:02:31.230 align:start position:0% where we have to rewrite everything which<00:02:28.280> has<00:02:28.459> happened<00:02:28.670> many<00:02:28.970> times<00:02:29.950> it's<00:02:30.950> also 00:02:31.230 --> 00:02:31.240 align:start position:0% which has happened many times it's also 00:02:31.240 --> 00:02:34.059 align:start position:0% which has happened many times it's also the<00:02:32.240> full<00:02:32.510> into<00:02:33.260> an<00:02:33.350> experience<00:02:33.709> there's 00:02:34.059 --> 00:02:34.069 align:start position:0% the full into an experience there's 00:02:34.069 --> 00:02:36.220 align:start position:0% the full into an experience there's nothing<00:02:34.430> really<00:02:34.640> mocked<00:02:35.510> out<00:02:35.720> here<00:02:36.050> when 00:02:36.220 --> 00:02:36.230 align:start position:0% nothing really mocked out here when 00:02:36.230 --> 00:02:38.380 align:start position:0% nothing really mocked out here when we're<00:02:36.410> talking<00:02:36.680> about<00:02:36.830> this<00:02:37.280> test<00:02:37.550> net<00:02:37.760> there 00:02:38.380 --> 00:02:38.390 align:start position:0% we're talking about this test net there 00:02:38.390 --> 00:02:41.199 align:start position:0% we're talking about this test net there are<00:02:38.600> real<00:02:39.020> deposits<00:02:39.740> coming<00:02:40.040> from<00:02:40.250> a<00:02:40.670> real 00:02:41.199 --> 00:02:41.209 align:start position:0% are real deposits coming from a real 00:02:41.209 --> 00:02:44.949 align:start position:0% are real deposits coming from a real eath<00:02:41.660> one<00:02:41.990> chain<00:02:42.380> and<00:02:42.790> the<00:02:43.820> POA<00:02:44.480> corley 00:02:44.949 --> 00:02:44.959 align:start position:0% eath one chain and the POA corley 00:02:44.959 --> 00:02:48.220 align:start position:0% eath one chain and the POA corley network<00:02:46.270> we've<00:02:47.270> actually<00:02:47.450> had<00:02:47.720> hundreds<00:02:48.110> of 00:02:48.220 --> 00:02:48.230 align:start position:0% network we've actually had hundreds of 00:02:48.230 --> 00:02:52.600 align:start position:0% network we've actually had hundreds of people<00:02:48.290> join<00:02:49.239> we<00:02:50.500> at<00:02:51.500> one<00:02:51.770> point<00:02:51.800> accidentally 00:02:52.600 --> 00:02:52.610 align:start position:0% people join we at one point accidentally 00:02:52.610 --> 00:02:55.420 align:start position:0% people join we at one point accidentally merged<00:02:52.940> with<00:02:53.239> the<00:02:53.690> ipfs<00:02:54.500> network<00:02:54.860> so<00:02:55.190> we 00:02:55.420 --> 00:02:55.430 align:start position:0% merged with the ipfs network so we 00:02:55.430 --> 00:02:58.090 align:start position:0% merged with the ipfs network so we learned<00:02:56.300> not<00:02:56.600> to<00:02:56.660> use<00:02:56.930> default<00:02:57.380> the<00:02:57.620> p2p 00:02:58.090 --> 00:02:58.100 align:start position:0% learned not to use default the p2p 00:02:58.100 --> 00:03:00.040 align:start position:0% learned not to use default the p2p parameters<00:02:58.820> because<00:02:59.180> that's<00:02:59.420> where<00:02:59.630> the<00:02:59.810> p2p 00:03:00.040 --> 00:03:00.050 align:start position:0% parameters because that's where the p2p 00:03:00.050 --> 00:03:02.140 align:start position:0% parameters because that's where the p2p does<00:03:00.320> and<00:03:00.739> we<00:03:01.370> were<00:03:01.489> wondering<00:03:01.850> why<00:03:01.940> do<00:03:02.000> we 00:03:02.140 --> 00:03:02.150 align:start position:0% does and we were wondering why do we 00:03:02.150 --> 00:03:04.060 align:start position:0% does and we were wondering why do we have<00:03:02.269> 500<00:03:02.750> nodes<00:03:02.930> connected<00:03:03.530> to<00:03:03.620> us<00:03:03.739> all<00:03:03.920> of<00:03:04.010> a 00:03:04.060 --> 00:03:04.070 align:start position:0% have 500 nodes connected to us all of a 00:03:04.070 --> 00:03:07.090 align:start position:0% have 500 nodes connected to us all of a sudden<00:03:04.340> so<00:03:04.550> that<00:03:05.480> was<00:03:05.600> a<00:03:05.630> kind<00:03:05.930> of<00:03:06.080> cool<00:03:06.290> cool 00:03:07.090 --> 00:03:07.100 align:start position:0% sudden so that was a kind of cool cool 00:03:07.100 --> 00:03:09.880 align:start position:0% sudden so that was a kind of cool cool thing<00:03:07.250> to<00:03:07.370> learn<00:03:07.900> but<00:03:08.900> but<00:03:09.290> most<00:03:09.500> of<00:03:09.560> all<00:03:09.650> it's 00:03:09.880 --> 00:03:09.890 align:start position:0% thing to learn but but most of all it's 00:03:09.890 --> 00:03:11.650 align:start position:0% thing to learn but but most of all it's just<00:03:09.950> been<00:03:10.280> really<00:03:10.940> really<00:03:10.970> fun<00:03:11.269> and<00:03:11.540> it's 00:03:11.650 --> 00:03:11.660 align:start position:0% just been really really fun and it's 00:03:11.660 --> 00:03:13.449 align:start position:0% just been really really fun and it's actually<00:03:11.900> really<00:03:12.080> easy<00:03:12.110> to<00:03:12.380> get<00:03:13.040> them<00:03:13.130> all<00:03:13.250> so 00:03:13.449 --> 00:03:13.459 align:start position:0% actually really easy to get them all so 00:03:13.459 --> 00:03:15.370 align:start position:0% actually really easy to get them all so if<00:03:13.550> you<00:03:13.670> wanted<00:03:13.910> to<00:03:14.090> be<00:03:14.420> a<00:03:14.450> validator<00:03:14.840> to<00:03:14.930> see 00:03:15.370 --> 00:03:15.380 align:start position:0% if you wanted to be a validator to see 00:03:15.380 --> 00:03:17.140 align:start position:0% if you wanted to be a validator to see what<00:03:15.500> it's<00:03:15.620> like<00:03:15.680> to<00:03:15.860> stake<00:03:16.070> with<00:03:16.310> prison 00:03:17.140 --> 00:03:17.150 align:start position:0% what it's like to stake with prison 00:03:17.150 --> 00:03:19.320 align:start position:0% what it's like to stake with prison client<00:03:17.540> we<00:03:17.660> have<00:03:17.840> this<00:03:18.050> test<00:03:18.320> net<00:03:18.500> website 00:03:19.320 --> 00:03:19.330 align:start position:0% client we have this test net website 00:03:19.330 --> 00:03:25.380 align:start position:0% client we have this test net website which<00:03:20.330> is<00:03:20.510> super<00:03:21.140> super<00:03:21.170> straightforward<00:03:21.680> so 00:03:25.380 --> 00:03:25.390 align:start position:0% 00:03:25.390 --> 00:03:28.050 align:start position:0% so<00:03:26.390> when<00:03:26.510> we're<00:03:26.630> talking<00:03:26.690> about<00:03:26.840> test<00:03:27.110> nets<00:03:27.320> I 00:03:28.050 --> 00:03:28.060 align:start position:0% so when we're talking about test nets I 00:03:28.060 --> 00:03:30.699 align:start position:0% so when we're talking about test nets I wanted<00:03:29.060> to<00:03:29.120> ask<00:03:29.239> fatale<00:03:29.600> ik<00:03:29.720> what<00:03:30.110> is<00:03:30.230> the<00:03:30.440> best 00:03:30.699 --> 00:03:30.709 align:start position:0% wanted to ask fatale ik what is the best 00:03:30.709 --> 00:03:32.650 align:start position:0% wanted to ask fatale ik what is the best test<00:03:31.010> set<00:03:31.130> of<00:03:31.250> all<00:03:31.340> time<00:03:31.579> and<00:03:31.910> I<00:03:32.299> didn't<00:03:32.510> get<00:03:32.630> a 00:03:32.650 --> 00:03:32.660 align:start position:0% test set of all time and I didn't get a 00:03:32.660 --> 00:03:34.539 align:start position:0% test set of all time and I didn't get a chance<00:03:32.720> to<00:03:32.930> ask<00:03:33.079> him<00:03:33.260> yet<00:03:33.470> so<00:03:33.650> this<00:03:34.220> is<00:03:34.370> what<00:03:34.519> I 00:03:34.539 --> 00:03:34.549 align:start position:0% chance to ask him yet so this is what I 00:03:34.549 --> 00:03:36.300 align:start position:0% chance to ask him yet so this is what I think<00:03:34.640> he's<00:03:34.970> gonna<00:03:35.150> say<00:03:35.420> is<00:03:35.600> he's<00:03:35.750> gonna<00:03:35.870> say 00:03:36.300 --> 00:03:36.310 align:start position:0% think he's gonna say is he's gonna say 00:03:36.310 --> 00:03:39.160 align:start position:0% think he's gonna say is he's gonna say Prismatic<00:03:37.310> labs<00:03:37.610> best<00:03:37.970> test<00:03:38.209> Network<00:03:38.660> and<00:03:39.049> the 00:03:39.160 --> 00:03:39.170 align:start position:0% Prismatic labs best test Network and the 00:03:39.170 --> 00:03:41.140 align:start position:0% Prismatic labs best test Network and the entire<00:03:39.410> planet<00:03:39.890> and<00:03:40.100> the<00:03:40.250> entire<00:03:40.489> universe<00:03:40.850> so 00:03:41.140 --> 00:03:41.150 align:start position:0% entire planet and the entire universe so 00:03:41.150 --> 00:03:44.020 align:start position:0% entire planet and the entire universe so I<00:03:41.720> challenged<00:03:42.380> someone<00:03:42.650> here<00:03:42.950> to<00:03:43.190> find<00:03:43.850> him 00:03:44.020 --> 00:03:44.030 align:start position:0% I challenged someone here to find him 00:03:44.030 --> 00:03:45.490 align:start position:0% I challenged someone here to find him and<00:03:44.209> ask<00:03:44.269> him<00:03:44.540> and<00:03:44.630> see<00:03:44.690> if<00:03:44.900> he<00:03:45.019> he<00:03:45.320> actually 00:03:45.490 --> 00:03:45.500 align:start position:0% and ask him and see if he he actually 00:03:45.500 --> 00:03:47.500 align:start position:0% and ask him and see if he he actually says<00:03:45.920> this<00:03:46.100> but 00:03:47.500 --> 00:03:47.510 align:start position:0% says this but 00:03:47.510 --> 00:03:49.569 align:start position:0% says this but Vitalik<00:03:48.200> has<00:03:48.320> run<00:03:48.530> it<00:03:48.650> as<00:03:48.769> far<00:03:48.950> as<00:03:49.010> we<00:03:49.130> know<00:03:49.280> and 00:03:49.569 --> 00:03:49.579 align:start position:0% Vitalik has run it as far as we know and 00:03:49.579 --> 00:03:51.310 align:start position:0% Vitalik has run it as far as we know and he's<00:03:49.820> really<00:03:50.090> enjoyed<00:03:50.329> it<00:03:50.420> as<00:03:50.570> so<00:03:50.720> many<00:03:51.140> other 00:03:51.310 --> 00:03:51.320 align:start position:0% he's really enjoyed it as so many other 00:03:51.320 --> 00:03:55.479 align:start position:0% he's really enjoyed it as so many other people<00:03:52.269> so<00:03:53.269> the<00:03:53.870> first<00:03:54.350> lesson<00:03:54.950> that<00:03:54.980> we 00:03:55.479 --> 00:03:55.489 align:start position:0% people so the first lesson that we 00:03:55.489 --> 00:03:58.600 align:start position:0% people so the first lesson that we learned<00:03:55.760> is<00:03:56.239> that<00:03:56.620> participation<00:03:57.620> has<00:03:58.220> to<00:03:58.459> be 00:03:58.600 --> 00:03:58.610 align:start position:0% learned is that participation has to be 00:03:58.610 --> 00:04:02.140 align:start position:0% learned is that participation has to be very<00:03:59.090> easy<00:03:59.380> you<00:04:00.380> have<00:04:00.410> users<00:04:01.130> coming<00:04:01.430> from<00:04:01.790> all 00:04:02.140 --> 00:04:02.150 align:start position:0% very easy you have users coming from all 00:04:02.150 --> 00:04:04.449 align:start position:0% very easy you have users coming from all types<00:04:02.450> of<00:04:02.799> technical<00:04:03.799> backgrounds<00:04:04.310> or<00:04:04.430> a 00:04:04.449 --> 00:04:04.459 align:start position:0% types of technical backgrounds or a 00:04:04.459 --> 00:04:06.370 align:start position:0% types of technical backgrounds or a non-technical<00:04:04.760> background<00:04:05.380> people 00:04:06.370 --> 00:04:06.380 align:start position:0% non-technical background people 00:04:06.380 --> 00:04:08.319 align:start position:0% non-technical background people interested<00:04:06.890> and<00:04:07.280> they<00:04:07.730> want<00:04:07.940> to<00:04:08.000> check<00:04:08.150> it<00:04:08.239> out 00:04:08.319 --> 00:04:08.329 align:start position:0% interested and they want to check it out 00:04:08.329 --> 00:04:10.960 align:start position:0% interested and they want to check it out but<00:04:08.510> they're<00:04:08.690> really<00:04:08.900> intimidated<00:04:09.500> by<00:04:09.970> adding 00:04:10.960 --> 00:04:10.970 align:start position:0% but they're really intimidated by adding 00:04:10.970 --> 00:04:13.120 align:start position:0% but they're really intimidated by adding a<00:04:11.090> lot<00:04:11.239> of<00:04:11.269> tooling<00:04:11.959> to<00:04:12.110> the<00:04:12.230> system<00:04:12.590> just<00:04:12.980> to 00:04:13.120 --> 00:04:13.130 align:start position:0% a lot of tooling to the system just to 00:04:13.130 --> 00:04:16.270 align:start position:0% a lot of tooling to the system just to build<00:04:13.340> the<00:04:13.610> project<00:04:14.530> and<00:04:15.530> running<00:04:15.980> into<00:04:16.070> these 00:04:16.270 --> 00:04:16.280 align:start position:0% build the project and running into these 00:04:16.280 --> 00:04:19.060 align:start position:0% build the project and running into these like<00:04:16.519> weird<00:04:17.019> errors<00:04:18.019> where<00:04:18.289> you<00:04:18.410> you<00:04:18.769> you<00:04:18.950> know 00:04:19.060 --> 00:04:19.070 align:start position:0% like weird errors where you you you know 00:04:19.070 --> 00:04:20.560 align:start position:0% like weird errors where you you you know it<00:04:19.519> really<00:04:19.729> depends<00:04:20.030> on<00:04:20.150> your<00:04:20.329> local 00:04:20.560 --> 00:04:20.570 align:start position:0% it really depends on your local 00:04:20.570 --> 00:04:24.400 align:start position:0% it really depends on your local environment<00:04:21.109> so<00:04:21.640> what<00:04:22.640> we've<00:04:22.820> done<00:04:23.060> is<00:04:23.410> built 00:04:24.400 --> 00:04:24.410 align:start position:0% environment so what we've done is built 00:04:24.410 --> 00:04:28.150 align:start position:0% environment so what we've done is built this<00:04:25.000> test<00:04:26.000> net<00:04:26.180> website<00:04:26.630> that<00:04:26.930> within<00:04:27.620> six 00:04:28.150 --> 00:04:28.160 align:start position:0% this test net website that within six 00:04:28.160 --> 00:04:31.690 align:start position:0% this test net website that within six easy<00:04:29.000> steps<00:04:29.390> you<00:04:30.020> can<00:04:30.200> be<00:04:30.440> validating<00:04:31.130> on 00:04:31.690 --> 00:04:31.700 align:start position:0% easy steps you can be validating on 00:04:31.700 --> 00:04:34.690 align:start position:0% easy steps you can be validating on real<00:04:32.180> be<00:04:32.330> contained<00:04:32.780> and<00:04:33.460> actually<00:04:34.460> seeing 00:04:34.690 --> 00:04:34.700 align:start position:0% real be contained and actually seeing 00:04:34.700 --> 00:04:37.660 align:start position:0% real be contained and actually seeing rewards<00:04:35.390> balances<00:04:36.080> change<00:04:36.860> and<00:04:37.220> and<00:04:37.460> even 00:04:37.660 --> 00:04:37.670 align:start position:0% rewards balances change and and even 00:04:37.670 --> 00:04:40.540 align:start position:0% rewards balances change and and even penalties<00:04:38.120> if<00:04:38.330> you<00:04:38.420> go<00:04:38.570> offline<00:04:38.930> so<00:04:39.920> the 00:04:40.540 --> 00:04:40.550 align:start position:0% penalties if you go offline so the 00:04:40.550 --> 00:04:42.790 align:start position:0% penalties if you go offline so the approach<00:04:40.760> we<00:04:40.910> took<00:04:41.060> was<00:04:41.300> to<00:04:41.540> use<00:04:41.840> docker 00:04:42.790 --> 00:04:42.800 align:start position:0% approach we took was to use docker 00:04:42.800 --> 00:04:45.940 align:start position:0% approach we took was to use docker images<00:04:43.220> and<00:04:43.520> this<00:04:44.480> is<00:04:44.690> typically<00:04:45.470> pretty<00:04:45.770> easy 00:04:45.940 --> 00:04:45.950 align:start position:0% images and this is typically pretty easy 00:04:45.950 --> 00:04:48.610 align:start position:0% images and this is typically pretty easy for<00:04:46.280> users<00:04:46.760> to<00:04:46.910> get<00:04:47.240> on<00:04:47.420> on<00:04:47.720> board<00:04:48.350> it<00:04:48.440> still 00:04:48.610 --> 00:04:48.620 align:start position:0% for users to get on on board it still 00:04:48.620 --> 00:04:51.130 align:start position:0% for users to get on on board it still requires<00:04:48.950> some<00:04:49.990> installation<00:04:50.990> on<00:04:51.080> your 00:04:51.130 --> 00:04:51.140 align:start position:0% requires some installation on your 00:04:51.140 --> 00:04:52.750 align:start position:0% requires some installation on your machine<00:04:51.500> and<00:04:51.770> some<00:04:51.920> you<00:04:52.220> know<00:04:52.310> command<00:04:52.670> line 00:04:52.750 --> 00:04:52.760 align:start position:0% machine and some you know command line 00:04:52.760 --> 00:04:55.510 align:start position:0% machine and some you know command line inputs<00:04:53.150> but<00:04:53.650> overall<00:04:54.650> you<00:04:54.950> don't<00:04:55.190> have<00:04:55.370> to 00:04:55.510 --> 00:04:55.520 align:start position:0% inputs but overall you don't have to 00:04:55.520 --> 00:04:57.910 align:start position:0% inputs but overall you don't have to spend<00:04:55.790> 10<00:04:56.690> to<00:04:56.930> 15<00:04:57.200> minutes<00:04:57.530> building<00:04:57.830> the 00:04:57.910 --> 00:04:57.920 align:start position:0% spend 10 to 15 minutes building the 00:04:57.920 --> 00:05:00.370 align:start position:0% spend 10 to 15 minutes building the project<00:04:58.070> as<00:04:58.460> some<00:04:58.640> of<00:04:58.790> you<00:04:58.970> may<00:04:59.930> have<00:04:59.960> seen 00:05:00.370 --> 00:05:00.380 align:start position:0% project as some of you may have seen 00:05:00.380 --> 00:05:02.320 align:start position:0% project as some of you may have seen from<00:05:00.410> the<00:05:00.950> Interop<00:05:01.430> event<00:05:01.760> if<00:05:01.940> you<00:05:02.030> tried<00:05:02.240> to 00:05:02.320 --> 00:05:02.330 align:start position:0% from the Interop event if you tried to 00:05:02.330 --> 00:05:03.640 align:start position:0% from the Interop event if you tried to build<00:05:02.450> prison<00:05:02.810> for<00:05:02.990> the<00:05:03.050> first<00:05:03.080> time<00:05:03.440> it<00:05:03.620> takes 00:05:03.640 --> 00:05:03.650 align:start position:0% build prison for the first time it takes 00:05:03.650 --> 00:05:06.670 align:start position:0% build prison for the first time it takes a<00:05:04.070> really<00:05:04.340> long<00:05:04.430> time<00:05:04.550> so<00:05:05.590> downloading<00:05:06.590> these 00:05:06.670 --> 00:05:06.680 align:start position:0% a really long time so downloading these 00:05:06.680 --> 00:05:10.660 align:start position:0% a really long time so downloading these images<00:05:07.100> which<00:05:07.280> are<00:05:08.260> 15<00:05:09.260> 20<00:05:09.620> megabytes<00:05:10.340> each 00:05:10.660 --> 00:05:10.670 align:start position:0% images which are 15 20 megabytes each 00:05:10.670 --> 00:05:12.970 align:start position:0% images which are 15 20 megabytes each just<00:05:11.030> seems<00:05:11.210> to<00:05:11.390> be<00:05:11.710> pretty<00:05:12.710> straightforward 00:05:12.970 --> 00:05:12.980 align:start position:0% just seems to be pretty straightforward 00:05:12.980 --> 00:05:15.120 align:start position:0% just seems to be pretty straightforward approach<00:05:13.430> haven't<00:05:14.150> had<00:05:14.240> too<00:05:14.390> many<00:05:14.450> problems 00:05:15.120 --> 00:05:15.130 align:start position:0% approach haven't had too many problems 00:05:15.130 --> 00:05:18.280 align:start position:0% approach haven't had too many problems we<00:05:16.130> also<00:05:16.310> made<00:05:17.210> it<00:05:17.240> so<00:05:17.540> that<00:05:17.750> within<00:05:18.020> the 00:05:18.280 --> 00:05:18.290 align:start position:0% we also made it so that within the 00:05:18.290 --> 00:05:19.960 align:start position:0% we also made it so that within the website<00:05:18.440> you<00:05:18.920> could<00:05:19.070> get<00:05:19.190> some<00:05:19.370> test<00:05:19.640> ether 00:05:19.960 --> 00:05:19.970 align:start position:0% website you could get some test ether 00:05:19.970 --> 00:05:21.430 align:start position:0% website you could get some test ether from<00:05:20.270> the<00:05:20.450> Gourley<00:05:20.720> Network<00:05:21.020> we<00:05:21.290> have<00:05:21.320> a 00:05:21.430 --> 00:05:21.440 align:start position:0% from the Gourley Network we have a 00:05:21.440 --> 00:05:25.060 align:start position:0% from the Gourley Network we have a faucet<00:05:21.710> built<00:05:22.010> in<00:05:22.160> here<00:05:23.290> an<00:05:24.290> interface<00:05:24.860> so 00:05:25.060 --> 00:05:25.070 align:start position:0% faucet built in here an interface so 00:05:25.070 --> 00:05:28.480 align:start position:0% faucet built in here an interface so that<00:05:25.100> you<00:05:25.340> can<00:05:25.550> just<00:05:26.120> paste<00:05:26.810> in<00:05:27.080> your<00:05:27.740> deposit 00:05:28.480 --> 00:05:28.490 align:start position:0% that you can just paste in your deposit 00:05:28.490 --> 00:05:30.730 align:start position:0% that you can just paste in your deposit data<00:05:28.640> key<00:05:28.970> and<00:05:29.330> then<00:05:29.600> with<00:05:29.900> meta<00:05:30.200> mask<00:05:30.440> or 00:05:30.730 --> 00:05:30.740 align:start position:0% data key and then with meta mask or 00:05:30.740 --> 00:05:33.010 align:start position:0% data key and then with meta mask or Portus<00:05:31.250> you<00:05:31.850> can<00:05:32.030> send<00:05:32.300> that<00:05:32.390> transaction<00:05:32.990> to 00:05:33.010 --> 00:05:33.020 align:start position:0% Portus you can send that transaction to 00:05:33.020 --> 00:05:37.420 align:start position:0% Portus you can send that transaction to the<00:05:33.320> eath<00:05:33.680> one<00:05:33.920> chain<00:05:34.190> and<00:05:35.140> wait<00:05:36.140> for<00:05:36.560> that<00:05:37.190> to 00:05:37.420 --> 00:05:37.430 align:start position:0% the eath one chain and wait for that to 00:05:37.430 --> 00:05:39.310 align:start position:0% the eath one chain and wait for that to be<00:05:37.610> included<00:05:38.150> into<00:05:38.270> a<00:05:38.420> block<00:05:38.690> and<00:05:39.020> then<00:05:39.140> go 00:05:39.310 --> 00:05:39.320 align:start position:0% be included into a block and then go 00:05:39.320 --> 00:05:40.660 align:start position:0% be included into a block and then go through<00:05:39.530> you<00:05:39.680> know<00:05:39.710> go<00:05:40.220> through<00:05:40.250> the<00:05:40.520> voting 00:05:40.660 --> 00:05:40.670 align:start position:0% through you know go through the voting 00:05:40.670 --> 00:05:42.430 align:start position:0% through you know go through the voting period<00:05:41.180> and<00:05:41.570> and<00:05:41.840> all<00:05:41.960> these<00:05:42.080> are<00:05:42.170> the<00:05:42.230> things 00:05:42.430 --> 00:05:42.440 align:start position:0% period and and all these are the things 00:05:42.440 --> 00:05:45.580 align:start position:0% period and and all these are the things and<00:05:42.740> eventually<00:05:43.220> you<00:05:43.400> do<00:05:44.000> get<00:05:44.210> activated<00:05:44.840> as<00:05:45.050> a 00:05:45.580 --> 00:05:45.590 align:start position:0% and eventually you do get activated as a 00:05:45.590 --> 00:05:46.930 align:start position:0% and eventually you do get activated as a validator<00:05:46.130> and<00:05:46.400> you<00:05:46.460> start<00:05:46.730> receiving 00:05:46.930 --> 00:05:46.940 align:start position:0% validator and you start receiving 00:05:46.940 --> 00:05:49.780 align:start position:0% validator and you start receiving assignments<00:05:47.540> from<00:05:47.600> you<00:05:47.750> beacon<00:05:47.990> chain<00:05:48.790> it 00:05:49.780 --> 00:05:49.790 align:start position:0% assignments from you beacon chain it 00:05:49.790 --> 00:05:51.450 align:start position:0% assignments from you beacon chain it actually<00:05:50.240> has<00:05:50.390> like<00:05:50.540> a<00:05:50.690> really<00:05:50.930> cool<00:05:51.140> like 00:05:51.450 --> 00:05:51.460 align:start position:0% actually has like a really cool like 00:05:51.460 --> 00:05:53.950 align:start position:0% actually has like a really cool like progress<00:05:52.460> bar<00:05:52.730> so<00:05:53.150> you<00:05:53.180> can<00:05:53.330> sort<00:05:53.750> of<00:05:53.780> they<00:05:53.900> see 00:05:53.950 --> 00:05:53.960 align:start position:0% progress bar so you can sort of they see 00:05:53.960 --> 00:05:55.480 align:start position:0% progress bar so you can sort of they see something's<00:05:54.500> happening<00:05:54.980> when<00:05:55.190> nothing's 00:05:55.480 --> 00:05:55.490 align:start position:0% something's happening when nothing's 00:05:55.490 --> 00:05:58.120 align:start position:0% something's happening when nothing's happening<00:05:55.910> so<00:05:56.120> it's<00:05:57.080> it's<00:05:57.620> been<00:05:57.710> really<00:05:58.010> cool 00:05:58.120 --> 00:05:58.130 align:start position:0% happening so it's it's been really cool 00:05:58.130 --> 00:06:01.300 align:start position:0% happening so it's it's been really cool and<00:05:58.370> and<00:05:58.760> this<00:05:59.300> has<00:05:59.480> been<00:05:59.600> our<00:05:59.720> biggest<00:06:00.310> value 00:06:01.300 --> 00:06:01.310 align:start position:0% and and this has been our biggest value 00:06:01.310 --> 00:06:03.010 align:start position:0% and and this has been our biggest value add<00:06:01.460> for<00:06:01.670> users<00:06:01.940> and<00:06:02.180> and<00:06:02.390> one<00:06:02.630> of<00:06:02.720> the<00:06:02.780> biggest 00:06:03.010 --> 00:06:03.020 align:start position:0% add for users and and one of the biggest 00:06:03.020 --> 00:06:04.930 align:start position:0% add for users and and one of the biggest things<00:06:03.140> that<00:06:03.230> we<00:06:03.440> started<00:06:03.770> off<00:06:03.890> with<00:06:04.070> learning 00:06:04.930 --> 00:06:04.940 align:start position:0% things that we started off with learning 00:06:04.940 --> 00:06:08.650 align:start position:0% things that we started off with learning because<00:06:05.650> before<00:06:06.650> we<00:06:06.830> got<00:06:06.980> here<00:06:07.490> it<00:06:08.180> was<00:06:08.420> just 00:06:08.650 --> 00:06:08.660 align:start position:0% because before we got here it was just 00:06:08.660 --> 00:06:10.870 align:start position:0% because before we got here it was just running<00:06:08.930> it<00:06:09.140> on<00:06:09.260> your<00:06:09.500> own<00:06:09.680> computer<00:06:09.950> running 00:06:10.870 --> 00:06:10.880 align:start position:0% running it on your own computer running 00:06:10.880 --> 00:06:12.730 align:start position:0% running it on your own computer running one<00:06:11.060> client<00:06:11.420> and<00:06:11.870> then<00:06:12.080> running<00:06:12.260> two<00:06:12.500> clients 00:06:12.730 --> 00:06:12.740 align:start position:0% one client and then running two clients 00:06:12.740 --> 00:06:14.950 align:start position:0% one client and then running two clients on<00:06:12.980> the<00:06:13.070> same<00:06:13.250> computer<00:06:13.510> then<00:06:14.510> on<00:06:14.630> different 00:06:14.950 --> 00:06:14.960 align:start position:0% on the same computer then on different 00:06:14.960 --> 00:06:18.280 align:start position:0% on the same computer then on different computers<00:06:15.380> and<00:06:15.800> all<00:06:16.550> of<00:06:16.730> this<00:06:16.820> was<00:06:17.360> was<00:06:17.780> you 00:06:18.280 --> 00:06:18.290 align:start position:0% computers and all of this was was you 00:06:18.290 --> 00:06:20.320 align:start position:0% computers and all of this was was you know<00:06:18.320> crawl<00:06:19.190> before<00:06:19.430> you<00:06:19.610> can<00:06:19.670> walk<00:06:19.820> and<00:06:20.210> then 00:06:20.320 --> 00:06:20.330 align:start position:0% know crawl before you can walk and then 00:06:20.330 --> 00:06:22.600 align:start position:0% know crawl before you can walk and then now<00:06:20.870> we're<00:06:21.020> starting<00:06:21.200> to<00:06:21.410> run<00:06:21.560> so<00:06:21.890> this<00:06:22.430> has 00:06:22.600 --> 00:06:22.610 align:start position:0% now we're starting to run so this has 00:06:22.610 --> 00:06:25.000 align:start position:0% now we're starting to run so this has been<00:06:22.760> a<00:06:22.820> my<00:06:23.750> favorite<00:06:24.110> lesson<00:06:24.380> so<00:06:24.530> far<00:06:24.590> just<00:06:24.890> to 00:06:25.000 --> 00:06:25.010 align:start position:0% been a my favorite lesson so far just to 00:06:25.010 --> 00:06:31.300 align:start position:0% been a my favorite lesson so far just to make<00:06:25.330> onboarding<00:06:26.330> easier<00:06:30.250> probably<00:06:31.250> the 00:06:31.300 --> 00:06:31.310 align:start position:0% make onboarding easier probably the 00:06:31.310 --> 00:06:33.340 align:start position:0% make onboarding easier probably the biggest<00:06:31.520> lesson<00:06:31.790> though<00:06:32.240> is<00:06:32.540> that<00:06:32.840> when<00:06:33.170> we're 00:06:33.340 --> 00:06:33.350 align:start position:0% biggest lesson though is that when we're 00:06:33.350 --> 00:06:34.830 align:start position:0% biggest lesson though is that when we're running 00:06:34.830 --> 00:06:34.840 align:start position:0% running 00:06:34.840 --> 00:06:38.020 align:start position:0% running we're<00:06:35.840> running<00:06:36.200> multiple<00:06:37.160> instances<00:06:37.760> of<00:06:37.820> the 00:06:38.020 --> 00:06:38.030 align:start position:0% we're running multiple instances of the 00:06:38.030 --> 00:06:41.380 align:start position:0% we're running multiple instances of the beacon<00:06:38.840> chain<00:06:39.050> in<00:06:39.470> the<00:06:40.250> cloud<00:06:40.460> or<00:06:40.490> whatever 00:06:41.380 --> 00:06:41.390 align:start position:0% beacon chain in the cloud or whatever 00:06:41.390 --> 00:06:44.470 align:start position:0% beacon chain in the cloud or whatever and<00:06:42.370> those<00:06:43.370> are<00:06:43.610> really<00:06:43.760> hard<00:06:44.030> to<00:06:44.300> be 00:06:44.470 --> 00:06:44.480 align:start position:0% and those are really hard to be 00:06:44.480 --> 00:06:45.439 align:start position:0% and those are really hard to be monitoring<00:06:45.020> is<00:06:45.230> he 00:06:45.439 --> 00:06:45.449 align:start position:0% monitoring is he 00:06:45.449 --> 00:06:47.089 align:start position:0% monitoring is he scene<00:06:45.629> from<00:06:45.870> like<00:06:46.199> screenshots<00:06:46.710> from<00:06:46.979> the 00:06:47.089 --> 00:06:47.099 align:start position:0% scene from like screenshots from the 00:06:47.099 --> 00:06:49.249 align:start position:0% scene from like screenshots from the inter<00:06:47.729> prevent<00:06:48.090> when<00:06:48.300> you<00:06:48.479> see<00:06:48.689> seven 00:06:49.249 --> 00:06:49.259 align:start position:0% inter prevent when you see seven 00:06:49.259 --> 00:06:51.559 align:start position:0% inter prevent when you see seven terminals<00:06:49.979> on<00:06:50.189> one<00:06:50.370> screen<00:06:50.729> it<00:06:51.300> could<00:06:51.449> be 00:06:51.559 --> 00:06:51.569 align:start position:0% terminals on one screen it could be 00:06:51.569 --> 00:06:53.629 align:start position:0% terminals on one screen it could be really<00:06:51.749> hard<00:06:52.020> to<00:06:52.110> see<00:06:52.439> that<00:06:53.099> there's<00:06:53.279> an<00:06:53.400> error 00:06:53.629 --> 00:06:53.639 align:start position:0% really hard to see that there's an error 00:06:53.639 --> 00:06:56.540 align:start position:0% really hard to see that there's an error or<00:06:53.909> even<00:06:54.180> to<00:06:54.270> catch<00:06:54.449> that<00:06:54.749> one<00:06:55.289> of<00:06:55.439> those<00:06:55.620> nodes 00:06:56.540 --> 00:06:56.550 align:start position:0% or even to catch that one of those nodes 00:06:56.550 --> 00:06:58.219 align:start position:0% or even to catch that one of those nodes was<00:06:56.849> not<00:06:56.999> even<00:06:57.270> talking<00:06:57.479> to<00:06:57.719> any<00:06:57.960> of<00:06:58.020> the<00:06:58.110> other 00:06:58.219 --> 00:06:58.229 align:start position:0% was not even talking to any of the other 00:06:58.229 --> 00:07:02.239 align:start position:0% was not even talking to any of the other six<00:06:58.669> which<00:06:59.669> did<00:06:59.879> happen<00:07:00.210> so<00:07:00.800> having<00:07:01.800> good 00:07:02.239 --> 00:07:02.249 align:start position:0% six which did happen so having good 00:07:02.249 --> 00:07:03.950 align:start position:0% six which did happen so having good monitoring<00:07:02.639> in<00:07:02.819> places<00:07:03.150> is<00:07:03.389> absolutely 00:07:03.950 --> 00:07:03.960 align:start position:0% monitoring in places is absolutely 00:07:03.960 --> 00:07:07.010 align:start position:0% monitoring in places is absolutely critical<00:07:04.169> to<00:07:04.819> understanding<00:07:05.819> when<00:07:06.629> errors 00:07:07.010 --> 00:07:07.020 align:start position:0% critical to understanding when errors 00:07:07.020 --> 00:07:08.600 align:start position:0% critical to understanding when errors are<00:07:07.229> happening<00:07:07.680> in<00:07:07.830> your<00:07:07.949> system<00:07:08.219> and<00:07:08.400> how<00:07:08.550> to 00:07:08.600 --> 00:07:08.610 align:start position:0% are happening in your system and how to 00:07:08.610 --> 00:07:11.089 align:start position:0% are happening in your system and how to find<00:07:08.999> the<00:07:09.180> root<00:07:09.389> cause<00:07:09.419> of<00:07:09.659> those<00:07:09.960> errors<00:07:10.229> so 00:07:11.089 --> 00:07:11.099 align:start position:0% find the root cause of those errors so 00:07:11.099 --> 00:07:12.559 align:start position:0% find the root cause of those errors so let's<00:07:11.219> take<00:07:11.339> a<00:07:11.370> look<00:07:11.580> at<00:07:11.639> a<00:07:11.699> few<00:07:11.879> a<00:07:11.939> strategy<00:07:12.509> as 00:07:12.559 --> 00:07:12.569 align:start position:0% let's take a look at a few a strategy as 00:07:12.569 --> 00:07:16.700 align:start position:0% let's take a look at a few a strategy as we<00:07:12.719> do<00:07:12.870> for<00:07:13.110> monitoring<00:07:13.559> things<00:07:15.259> from<00:07:16.259> from 00:07:16.700 --> 00:07:16.710 align:start position:0% we do for monitoring things from from 00:07:16.710 --> 00:07:20.050 align:start position:0% we do for monitoring things from from the<00:07:16.860> very<00:07:17.099> beginning<00:07:17.460> we<00:07:17.610> have<00:07:17.930> built<00:07:18.930> out 00:07:20.050 --> 00:07:20.060 align:start position:0% the very beginning we have built out 00:07:20.060 --> 00:07:22.610 align:start position:0% the very beginning we have built out metrics<00:07:21.060> collection<00:07:21.569> and<00:07:21.839> monitoring 00:07:22.610 --> 00:07:22.620 align:start position:0% metrics collection and monitoring 00:07:22.620 --> 00:07:25.189 align:start position:0% metrics collection and monitoring through<00:07:23.159> these<00:07:23.849> metrics<00:07:24.240> so<00:07:24.479> here<00:07:24.960> you<00:07:25.050> can 00:07:25.189 --> 00:07:25.199 align:start position:0% through these metrics so here you can 00:07:25.199 --> 00:07:27.829 align:start position:0% through these metrics so here you can see<00:07:25.460> a<00:07:26.460> test<00:07:26.909> network<00:07:27.240> that's<00:07:27.419> been<00:07:27.629> running 00:07:27.829 --> 00:07:27.839 align:start position:0% see a test network that's been running 00:07:27.839 --> 00:07:30.290 align:start position:0% see a test network that's been running for<00:07:28.169> two<00:07:28.860> hundred<00:07:29.009> and<00:07:29.249> twenty<00:07:29.729> nine<00:07:29.759> thousand 00:07:30.290 --> 00:07:30.300 align:start position:0% for two hundred and twenty nine thousand 00:07:30.300 --> 00:07:32.390 align:start position:0% for two hundred and twenty nine thousand slots<00:07:30.650> it's<00:07:31.650> starting<00:07:31.979> to<00:07:32.039> look<00:07:32.189> a<00:07:32.279> little 00:07:32.390 --> 00:07:32.400 align:start position:0% slots it's starting to look a little 00:07:32.400 --> 00:07:34.309 align:start position:0% slots it's starting to look a little weird<00:07:32.699> because<00:07:32.999> these<00:07:33.389> these<00:07:33.779> graphs<00:07:34.139> should 00:07:34.309 --> 00:07:34.319 align:start position:0% weird because these these graphs should 00:07:34.319 --> 00:07:36.350 align:start position:0% weird because these these graphs should be<00:07:34.529> pretty<00:07:34.800> you<00:07:35.460> know<00:07:35.490> flat<00:07:35.909> and<00:07:36.120> straight 00:07:36.350 --> 00:07:36.360 align:start position:0% be pretty you know flat and straight 00:07:36.360 --> 00:07:41.329 align:start position:0% be pretty you know flat and straight line<00:07:37.339> especially<00:07:38.479> the<00:07:39.479> you<00:07:40.379> know<00:07:40.469> the<00:07:40.770> CPU 00:07:41.329 --> 00:07:41.339 align:start position:0% line especially the you know the CPU 00:07:41.339 --> 00:07:43.179 align:start position:0% line especially the you know the CPU churn<00:07:41.819> and<00:07:42.089> on<00:07:42.149> these<00:07:42.270> kind<00:07:42.479> of<00:07:42.539> things<00:07:42.779> so 00:07:43.179 --> 00:07:43.189 align:start position:0% churn and on these kind of things so 00:07:43.189 --> 00:07:46.639 align:start position:0% churn and on these kind of things so here<00:07:44.189> we<00:07:44.370> might<00:07:44.580> say<00:07:45.449> like<00:07:45.749> oh<00:07:46.020> if<00:07:46.439> there 00:07:46.639 --> 00:07:46.649 align:start position:0% here we might say like oh if there 00:07:46.649 --> 00:07:50.059 align:start position:0% here we might say like oh if there hasn't<00:07:46.830> been<00:07:47.449> a<00:07:48.449> new<00:07:48.689> block<00:07:49.139> in<00:07:49.529> thirty 00:07:50.059 --> 00:07:50.069 align:start position:0% hasn't been a new block in thirty 00:07:50.069 --> 00:07:52.189 align:start position:0% hasn't been a new block in thirty seconds<00:07:50.520> and<00:07:50.669> will<00:07:50.849> fire<00:07:51.300> an<00:07:51.449> alarm<00:07:51.749> and<00:07:52.050> will 00:07:52.189 --> 00:07:52.199 align:start position:0% seconds and will fire an alarm and will 00:07:52.199 --> 00:07:54.559 align:start position:0% seconds and will fire an alarm and will alert<00:07:52.620> to<00:07:52.770> that<00:07:52.800> make<00:07:53.789> your<00:07:53.909> phone<00:07:54.089> ring<00:07:54.120> or 00:07:54.559 --> 00:07:54.569 align:start position:0% alert to that make your phone ring or 00:07:54.569 --> 00:07:58.159 align:start position:0% alert to that make your phone ring or whatever<00:07:54.689> and<00:07:55.639> then<00:07:56.639> when<00:07:56.789> you<00:07:57.029> are<00:07:57.389> woken<00:07:58.080> up 00:07:58.159 --> 00:07:58.169 align:start position:0% whatever and then when you are woken up 00:07:58.169 --> 00:08:00.019 align:start position:0% whatever and then when you are woken up in<00:07:58.289> the<00:07:58.349> middle<00:07:58.499> of<00:07:58.649> the<00:07:58.710> night<00:07:58.860> and<00:07:59.159> you<00:07:59.849> see 00:08:00.019 --> 00:08:00.029 align:start position:0% in the middle of the night and you see 00:08:00.029 --> 00:08:03.199 align:start position:0% in the middle of the night and you see something<00:08:00.300> like<00:08:00.479> this<00:08:00.629> you<00:08:01.620> have<00:08:01.830> some<00:08:02.209> clues 00:08:03.199 --> 00:08:03.209 align:start position:0% something like this you have some clues 00:08:03.209 --> 00:08:04.850 align:start position:0% something like this you have some clues of<00:08:03.689> where<00:08:04.020> to<00:08:04.289> start<00:08:04.319> looking 00:08:04.850 --> 00:08:04.860 align:start position:0% of where to start looking 00:08:04.860 --> 00:08:08.029 align:start position:0% of where to start looking so<00:08:05.129> to<00:08:05.759> digest<00:08:06.719> this<00:08:06.839> for<00:08:07.050> you<00:08:07.110> guys<00:08:07.289> you<00:08:07.860> can 00:08:08.029 --> 00:08:08.039 align:start position:0% so to digest this for you guys you can 00:08:08.039 --> 00:08:09.769 align:start position:0% so to digest this for you guys you can see<00:08:08.219> on<00:08:08.459> the<00:08:08.729> bottom<00:08:08.909> right<00:08:09.240> that<00:08:09.270> there's 00:08:09.769 --> 00:08:09.779 align:start position:0% see on the bottom right that there's 00:08:09.779 --> 00:08:21.079 align:start position:0% see on the bottom right that there's chained<00:08:19.550> reorganization<00:08:20.550> imbalances<00:08:21.060> so 00:08:21.079 --> 00:08:21.089 align:start position:0% chained reorganization imbalances so 00:08:21.089 --> 00:08:23.869 align:start position:0% chained reorganization imbalances so nobody<00:08:21.539> was<00:08:21.629> agreeing<00:08:21.959> on<00:08:22.050> anything<00:08:22.699> we<00:08:23.699> had 00:08:23.869 --> 00:08:23.879 align:start position:0% nobody was agreeing on anything we had 00:08:23.879 --> 00:08:26.149 align:start position:0% nobody was agreeing on anything we had you<00:08:24.719> know<00:08:24.810> hundreds<00:08:25.349> of<00:08:25.439> thousands<00:08:25.860> of<00:08:25.949> go 00:08:26.149 --> 00:08:26.159 align:start position:0% you know hundreds of thousands of go 00:08:26.159 --> 00:08:27.709 align:start position:0% you know hundreds of thousands of go routines<00:08:26.550> happening<00:08:26.999> at<00:08:27.089> the<00:08:27.149> same<00:08:27.300> time<00:08:27.360> so 00:08:27.709 --> 00:08:27.719 align:start position:0% routines happening at the same time so 00:08:27.719 --> 00:08:30.110 align:start position:0% routines happening at the same time so these<00:08:27.810> four<00:08:28.279> concurrent<00:08:29.279> processes<00:08:29.729> and 00:08:30.110 --> 00:08:30.120 align:start position:0% these four concurrent processes and 00:08:30.120 --> 00:08:33.050 align:start position:0% these four concurrent processes and that's<00:08:30.300> leading<00:08:30.689> to<00:08:30.839> the<00:08:31.560> CPU<00:08:32.010> maxing<00:08:32.430> out<00:08:32.579> and 00:08:33.050 --> 00:08:33.060 align:start position:0% that's leading to the CPU maxing out and 00:08:33.060 --> 00:08:35.449 align:start position:0% that's leading to the CPU maxing out and then<00:08:33.810> this<00:08:34.079> nice<00:08:34.349> like<00:08:34.680> up<00:08:34.919> and<00:08:35.219> to<00:08:35.310> the<00:08:35.339> right 00:08:35.449 --> 00:08:35.459 align:start position:0% then this nice like up and to the right 00:08:35.459 --> 00:08:38.540 align:start position:0% then this nice like up and to the right line<00:08:36.120> is<00:08:36.389> starting<00:08:36.750> to<00:08:36.839> fall<00:08:37.019> apart<00:08:37.339> and<00:08:38.339> this 00:08:38.540 --> 00:08:38.550 align:start position:0% line is starting to fall apart and this 00:08:38.550 --> 00:08:41.329 align:start position:0% line is starting to fall apart and this was<00:08:38.729> like<00:08:38.880> a<00:08:38.940> nightmare<00:08:39.240> scenario<00:08:39.990> so<00:08:40.589> having 00:08:41.329 --> 00:08:41.339 align:start position:0% was like a nightmare scenario so having 00:08:41.339 --> 00:08:44.500 align:start position:0% was like a nightmare scenario so having these<00:08:41.639> resources<00:08:42.060> is<00:08:42.360> super<00:08:42.659> super<00:08:42.810> helpful 00:08:44.500 --> 00:08:44.510 align:start position:0% these resources is super super helpful 00:08:44.510 --> 00:08:48.319 align:start position:0% these resources is super super helpful but<00:08:45.510> now<00:08:45.630> you<00:08:45.690> know<00:08:45.870> like<00:08:46.079> what's<00:08:46.670> what's<00:08:47.670> kind 00:08:48.319 --> 00:08:48.329 align:start position:0% but now you know like what's what's kind 00:08:48.329 --> 00:08:49.819 align:start position:0% but now you know like what's what's kind of<00:08:48.389> going<00:08:48.449> wrong<00:08:48.810> like<00:08:49.050> how<00:08:49.230> do<00:08:49.290> you<00:08:49.410> dive<00:08:49.589> into 00:08:49.819 --> 00:08:49.829 align:start position:0% of going wrong like how do you dive into 00:08:49.829 --> 00:08:53.470 align:start position:0% of going wrong like how do you dive into some<00:08:50.519> very<00:08:50.910> specific<00:08:51.449> details<00:08:51.810> like<00:08:52.649> maybe 00:08:53.470 --> 00:08:53.480 align:start position:0% some very specific details like maybe 00:08:53.480 --> 00:08:56.750 align:start position:0% some very specific details like maybe block<00:08:54.529> production<00:08:55.529> is<00:08:55.680> taking<00:08:56.040> longer<00:08:56.640> than 00:08:56.750 --> 00:08:56.760 align:start position:0% block production is taking longer than 00:08:56.760 --> 00:08:58.699 align:start position:0% block production is taking longer than expected<00:08:56.970> and<00:08:57.569> you<00:08:57.630> want<00:08:58.140> to<00:08:58.199> identify<00:08:58.290> the 00:08:58.699 --> 00:08:58.709 align:start position:0% expected and you want to identify the 00:08:58.709 --> 00:08:59.150 align:start position:0% expected and you want to identify the route 00:08:59.150 --> 00:08:59.160 align:start position:0% route 00:08:59.160 --> 00:09:01.670 align:start position:0% route cause<00:08:59.250> of<00:08:59.399> these<00:08:59.519> things<00:08:59.579> so<00:09:00.089> what<00:09:00.899> we<00:09:01.079> do<00:09:01.290> is 00:09:01.670 --> 00:09:01.680 align:start position:0% cause of these things so what we do is 00:09:01.680 --> 00:09:04.340 align:start position:0% cause of these things so what we do is we<00:09:02.069> do<00:09:02.790> a<00:09:02.819> practice<00:09:03.300> called<00:09:03.509> code 00:09:04.340 --> 00:09:04.350 align:start position:0% we do a practice called code 00:09:04.350 --> 00:09:08.389 align:start position:0% we do a practice called code instrumentation<00:09:05.069> so<00:09:06.060> will<00:09:06.389> annotate<00:09:07.399> pieces 00:09:08.389 --> 00:09:08.399 align:start position:0% instrumentation so will annotate pieces 00:09:08.399 --> 00:09:11.030 align:start position:0% instrumentation so will annotate pieces of<00:09:08.579> the<00:09:08.730> code<00:09:08.910> that<00:09:08.940> we<00:09:09.269> think<00:09:09.509> are<00:09:10.040> expensive 00:09:11.030 --> 00:09:11.040 align:start position:0% of the code that we think are expensive 00:09:11.040 --> 00:09:13.850 align:start position:0% of the code that we think are expensive or<00:09:11.310> really<00:09:11.940> interesting<00:09:12.420> to<00:09:12.660> track<00:09:12.930> so<00:09:13.680> here 00:09:13.850 --> 00:09:13.860 align:start position:0% or really interesting to track so here 00:09:13.860 --> 00:09:16.420 align:start position:0% or really interesting to track so here you<00:09:13.980> can<00:09:14.100> see<00:09:14.310> the<00:09:14.550> entire<00:09:14.790> workflow<00:09:15.269> for<00:09:15.870> a 00:09:16.420 --> 00:09:16.430 align:start position:0% you can see the entire workflow for a 00:09:16.430 --> 00:09:19.460 align:start position:0% you can see the entire workflow for a block<00:09:17.430> proposal<00:09:18.389> so<00:09:18.629> starting<00:09:18.930> at<00:09:19.019> the<00:09:19.050> top<00:09:19.110> we 00:09:19.460 --> 00:09:19.470 align:start position:0% block proposal so starting at the top we 00:09:19.470 --> 00:09:20.780 align:start position:0% block proposal so starting at the top we have<00:09:19.500> the<00:09:19.709> validator<00:09:20.069> client<00:09:20.490> which<00:09:20.610> is<00:09:20.759> a 00:09:20.780 --> 00:09:20.790 align:start position:0% have the validator client which is a 00:09:20.790 --> 00:09:22.809 align:start position:0% have the validator client which is a separate<00:09:21.209> process<00:09:21.389> from<00:09:21.779> the<00:09:21.870> beacon<00:09:22.110> chain 00:09:22.809 --> 00:09:22.819 align:start position:0% separate process from the beacon chain 00:09:22.819 --> 00:09:25.869 align:start position:0% separate process from the beacon chain sending<00:09:23.819> the<00:09:24.180> block<00:09:24.389> to<00:09:24.689> the<00:09:24.720> beacon<00:09:25.199> chain 00:09:25.869 --> 00:09:25.879 align:start position:0% sending the block to the beacon chain 00:09:25.879 --> 00:09:30.309 align:start position:0% sending the block to the beacon chain and<00:09:26.879> then<00:09:27.870> something<00:09:28.350> along<00:09:28.680> here<00:09:28.980> is<00:09:29.519> taking 00:09:30.309 --> 00:09:30.319 align:start position:0% and then something along here is taking 00:09:30.319 --> 00:09:33.350 align:start position:0% and then something along here is taking 236<00:09:31.319> milliseconds<00:09:32.009> and<00:09:32.189> you<00:09:32.459> know<00:09:32.639> we<00:09:32.939> kind<00:09:33.269> of 00:09:33.350 --> 00:09:33.360 align:start position:0% 236 milliseconds and you know we kind of 00:09:33.360 --> 00:09:35.540 align:start position:0% 236 milliseconds and you know we kind of expected<00:09:33.810> that<00:09:34.079> to<00:09:34.319> maybe<00:09:34.500> be<00:09:34.740> 50 00:09:35.540 --> 00:09:35.550 align:start position:0% expected that to maybe be 50 00:09:35.550 --> 00:09:38.569 align:start position:0% expected that to maybe be 50 milliseconds<00:09:36.680> typically<00:09:37.680> so<00:09:37.889> we<00:09:38.160> can<00:09:38.370> dive 00:09:38.569 --> 00:09:38.579 align:start position:0% milliseconds typically so we can dive 00:09:38.579 --> 00:09:40.759 align:start position:0% milliseconds typically so we can dive into<00:09:38.759> each<00:09:38.899> part<00:09:39.899> and<00:09:40.110> see<00:09:40.290> why<00:09:40.500> it<00:09:40.560> took<00:09:40.709> so 00:09:40.759 --> 00:09:40.769 align:start position:0% into each part and see why it took so 00:09:40.769 --> 00:09:43.699 align:start position:0% into each part and see why it took so long<00:09:41.540> this<00:09:42.540> one<00:09:42.779> took<00:09:43.019> a<00:09:43.110> long<00:09:43.230> time<00:09:43.470> because 00:09:43.699 --> 00:09:43.709 align:start position:0% long this one took a long time because 00:09:43.709 --> 00:09:46.189 align:start position:0% long this one took a long time because it<00:09:43.949> was<00:09:44.040> processing<00:09:44.490> a<00:09:44.670> lot<00:09:44.790> of<00:09:44.910> slots<00:09:45.149> so<00:09:46.050> we 00:09:46.189 --> 00:09:46.199 align:start position:0% it was processing a lot of slots so we 00:09:46.199 --> 00:09:49.160 align:start position:0% it was processing a lot of slots so we might<00:09:46.439> say<00:09:46.709> was<00:09:47.279> there<00:09:47.610> a<00:09:48.029> long<00:09:48.449> period<00:09:48.959> of 00:09:49.160 --> 00:09:49.170 align:start position:0% might say was there a long period of 00:09:49.170 --> 00:09:51.860 align:start position:0% might say was there a long period of skip<00:09:49.560> slots<00:09:49.860> or<00:09:50.160> can<00:09:50.490> we<00:09:50.670> cache<00:09:51.329> this<00:09:51.629> somehow 00:09:51.860 --> 00:09:51.870 align:start position:0% skip slots or can we cache this somehow 00:09:51.870 --> 00:09:53.540 align:start position:0% skip slots or can we cache this somehow so<00:09:52.350> that<00:09:52.439> we<00:09:52.500> don't<00:09:52.620> have<00:09:52.800> to<00:09:52.949> do<00:09:53.100> this<00:09:53.279> work 00:09:53.540 --> 00:09:53.550 align:start position:0% so that we don't have to do this work 00:09:53.550 --> 00:09:56.840 align:start position:0% so that we don't have to do this work redundantly<00:09:54.620> where<00:09:55.620> can<00:09:55.829> we<00:09:55.949> find<00:09:55.980> ways<00:09:56.579> to 00:09:56.840 --> 00:09:56.850 align:start position:0% redundantly where can we find ways to 00:09:56.850 --> 00:09:58.999 align:start position:0% redundantly where can we find ways to improve<00:09:57.120> and<00:09:57.269> make<00:09:57.420> things<00:09:57.600> faster<00:09:58.050> and<00:09:58.290> some 00:09:58.999 --> 00:09:59.009 align:start position:0% improve and make things faster and some 00:09:59.009 --> 00:10:01.309 align:start position:0% improve and make things faster and some of<00:09:59.100> these<00:09:59.189> things<00:09:59.399> like<00:09:59.759> you<00:10:00.420> know<00:10:00.449> from<00:10:01.110> your 00:10:01.309 --> 00:10:01.319 align:start position:0% of these things like you know from your 00:10:01.319 --> 00:10:02.660 align:start position:0% of these things like you know from your intuition<00:10:01.740> and<00:10:01.920> you<00:10:02.009> can<00:10:02.160> write<00:10:02.339> tests<00:10:02.579> for 00:10:02.660 --> 00:10:02.670 align:start position:0% intuition and you can write tests for 00:10:02.670 --> 00:10:05.120 align:start position:0% intuition and you can write tests for and<00:10:03.089> kind<00:10:03.240> of<00:10:03.420> write<00:10:03.990> you<00:10:04.439> know<00:10:04.500> benchmarks<00:10:05.040> to 00:10:05.120 --> 00:10:05.130 align:start position:0% and kind of write you know benchmarks to 00:10:05.130 --> 00:10:06.829 align:start position:0% and kind of write you know benchmarks to see<00:10:05.430> like<00:10:05.610> okay<00:10:05.910> I<00:10:05.939> know<00:10:06.029> this<00:10:06.240> is<00:10:06.389> a<00:10:06.600> place 00:10:06.829 --> 00:10:06.839 align:start position:0% see like okay I know this is a place 00:10:06.839 --> 00:10:07.999 align:start position:0% see like okay I know this is a place that<00:10:07.050> could<00:10:07.170> be<00:10:07.290> bad<00:10:07.410> and<00:10:07.589> I<00:10:07.649> can<00:10:07.800> make<00:10:07.920> it 00:10:07.999 --> 00:10:08.009 align:start position:0% that could be bad and I can make it 00:10:08.009 --> 00:10:09.769 align:start position:0% that could be bad and I can make it better<00:10:08.160> but<00:10:08.399> other<00:10:08.880> times<00:10:09.120> you<00:10:09.240> won't<00:10:09.449> know<00:10:09.750> it 00:10:09.769 --> 00:10:09.779 align:start position:0% better but other times you won't know it 00:10:09.779 --> 00:10:11.629 align:start position:0% better but other times you won't know it until<00:10:10.079> it<00:10:10.560> happens<00:10:10.860> in<00:10:10.980> the<00:10:11.040> real<00:10:11.160> world<00:10:11.399> so 00:10:11.629 --> 00:10:11.639 align:start position:0% until it happens in the real world so 00:10:11.639 --> 00:10:15.490 align:start position:0% until it happens in the real world so this<00:10:12.060> has<00:10:12.240> been<00:10:12.300> super<00:10:12.449> helpful<00:10:12.779> for<00:10:13.079> us 00:10:15.490 --> 00:10:15.500 align:start position:0% 00:10:15.500 --> 00:10:20.179 align:start position:0% speaking<00:10:16.500> of<00:10:16.589> benchmarking<00:10:17.189> we<00:10:17.790> do<00:10:19.189> profiling 00:10:20.179 --> 00:10:20.189 align:start position:0% speaking of benchmarking we do profiling 00:10:20.189 --> 00:10:22.999 align:start position:0% speaking of benchmarking we do profiling so<00:10:20.430> we'll<00:10:20.639> take<00:10:20.910> when<00:10:21.089> we<00:10:21.240> see<00:10:21.540> that<00:10:21.899> there<00:10:22.709> are 00:10:22.999 --> 00:10:23.009 align:start position:0% so we'll take when we see that there are 00:10:23.009 --> 00:10:24.710 align:start position:0% so we'll take when we see that there are a<00:10:23.040> hundred<00:10:23.519> thousand<00:10:23.819> go<00:10:23.939> routines<00:10:24.329> or 00:10:24.710 --> 00:10:24.720 align:start position:0% a hundred thousand go routines or 00:10:24.720 --> 00:10:26.269 align:start position:0% a hundred thousand go routines or something's<00:10:25.079> taking<00:10:25.410> a<00:10:25.560> long<00:10:25.680> time<00:10:25.920> using<00:10:26.130> a 00:10:26.269 --> 00:10:26.279 align:start position:0% something's taking a long time using a 00:10:26.279 --> 00:10:29.809 align:start position:0% something's taking a long time using a lot<00:10:26.430> of<00:10:26.459> CPU<00:10:27.180> will<00:10:28.019> take<00:10:28.339> maybe<00:10:29.339> like<00:10:29.519> a<00:10:29.550> 30 00:10:29.809 --> 00:10:29.819 align:start position:0% lot of CPU will take maybe like a 30 00:10:29.819 --> 00:10:31.970 align:start position:0% lot of CPU will take maybe like a 30 second<00:10:30.209> snapshot<00:10:30.449> of<00:10:30.660> the<00:10:30.870> CPU<00:10:31.230> profile<00:10:31.709> and 00:10:31.970 --> 00:10:31.980 align:start position:0% second snapshot of the CPU profile and 00:10:31.980 --> 00:10:34.970 align:start position:0% second snapshot of the CPU profile and then<00:10:32.550> look<00:10:32.819> at<00:10:33.029> this<00:10:33.389> this<00:10:34.350> is<00:10:34.410> called<00:10:34.680> a<00:10:34.740> flame 00:10:34.970 --> 00:10:34.980 align:start position:0% then look at this this is called a flame 00:10:34.980 --> 00:10:37.069 align:start position:0% then look at this this is called a flame graph<00:10:35.279> because<00:10:35.790> it<00:10:36.000> looks<00:10:36.240> kind<00:10:36.509> of<00:10:36.600> like<00:10:36.870> a 00:10:37.069 --> 00:10:37.079 align:start position:0% graph because it looks kind of like a 00:10:37.079 --> 00:10:41.030 align:start position:0% graph because it looks kind of like a flame<00:10:37.439> and<00:10:37.850> you<00:10:38.850> can<00:10:39.029> dive<00:10:39.509> into<00:10:39.750> the<00:10:40.170> most 00:10:41.030 --> 00:10:41.040 align:start position:0% flame and you can dive into the most 00:10:41.040 --> 00:10:42.439 align:start position:0% flame and you can dive into the most expensive<00:10:41.579> parts<00:10:41.819> the<00:10:41.910> things<00:10:42.089> that<00:10:42.209> took<00:10:42.300> the 00:10:42.439 --> 00:10:42.449 align:start position:0% expensive parts the things that took the 00:10:42.449 --> 00:10:46.910 align:start position:0% expensive parts the things that took the longest<00:10:42.600> and<00:10:43.519> go<00:10:44.519> and<00:10:44.819> then<00:10:45.740> reproduce<00:10:46.740> this 00:10:46.910 --> 00:10:46.920 align:start position:0% longest and go and then reproduce this 00:10:46.920 --> 00:10:48.650 align:start position:0% longest and go and then reproduce this in<00:10:47.100> a<00:10:47.189> local<00:10:47.490> test<00:10:47.670> and<00:10:47.939> be<00:10:48.059> able<00:10:48.180> to<00:10:48.449> iterate 00:10:48.650 --> 00:10:48.660 align:start position:0% in a local test and be able to iterate 00:10:48.660 --> 00:10:50.230 align:start position:0% in a local test and be able to iterate on<00:10:48.779> that<00:10:48.899> and<00:10:49.259> make<00:10:49.410> it<00:10:49.439> a<00:10:49.589> little<00:10:49.709> bit<00:10:49.889> better 00:10:50.230 --> 00:10:50.240 align:start position:0% on that and make it a little bit better 00:10:50.240 --> 00:10:54.369 align:start position:0% on that and make it a little bit better this<00:10:51.240> has<00:10:51.389> been<00:10:51.569> super<00:10:51.689> helpful<00:10:51.899> for<00:10:52.199> us 00:10:54.369 --> 00:10:54.379 align:start position:0% 00:10:54.379 --> 00:10:57.499 align:start position:0% another<00:10:55.379> key<00:10:55.649> thing<00:10:55.860> that<00:10:56.069> we've<00:10:56.220> done<00:10:56.509> is 00:10:57.499 --> 00:10:57.509 align:start position:0% another key thing that we've done is 00:10:57.509 --> 00:10:59.990 align:start position:0% another key thing that we've done is this<00:10:58.439> is<00:10:58.500> kind<00:10:58.800> of<00:10:58.889> recent<00:10:59.220> is<00:10:59.339> to<00:10:59.519> do<00:10:59.639> log 00:10:59.990 --> 00:11:00.000 align:start position:0% this is kind of recent is to do log 00:11:00.000 --> 00:11:02.179 align:start position:0% this is kind of recent is to do log ingestion<00:11:00.389> so<00:11:00.930> when<00:11:01.199> we're<00:11:01.410> running<00:11:01.500> multiple 00:11:02.179 --> 00:11:02.189 align:start position:0% ingestion so when we're running multiple 00:11:02.189 --> 00:11:04.429 align:start position:0% ingestion so when we're running multiple processes<00:11:02.730> and<00:11:03.089> trying<00:11:03.839> to<00:11:03.930> understand 00:11:04.429 --> 00:11:04.439 align:start position:0% processes and trying to understand 00:11:04.439 --> 00:11:07.910 align:start position:0% processes and trying to understand errors<00:11:05.009> and<00:11:05.309> we<00:11:05.430> have<00:11:05.870> evidence<00:11:06.870> we've<00:11:07.110> users 00:11:07.910 --> 00:11:07.920 align:start position:0% errors and we have evidence we've users 00:11:07.920 --> 00:11:09.949 align:start position:0% errors and we have evidence we've users are<00:11:08.069> filing<00:11:08.430> bugs<00:11:08.610> that<00:11:08.850> the<00:11:09.300> signatures<00:11:09.839> are 00:11:09.949 --> 00:11:09.959 align:start position:0% are filing bugs that the signatures are 00:11:09.959 --> 00:11:13.260 align:start position:0% are filing bugs that the signatures are not<00:11:10.079> verifying<00:11:10.559> we<00:11:11.250> can<00:11:11.490> query<00:11:12.360> the 00:11:13.260 --> 00:11:13.270 align:start position:0% not verifying we can query the 00:11:13.270 --> 00:11:15.810 align:start position:0% not verifying we can query the the<00:11:13.600> central<00:11:14.050> repository<00:11:14.200> of<00:11:14.740> logs<00:11:15.459> and<00:11:15.670> see 00:11:15.810 --> 00:11:15.820 align:start position:0% the central repository of logs and see 00:11:15.820 --> 00:11:18.170 align:start position:0% the central repository of logs and see on<00:11:16.029> what<00:11:16.600> frequencies<00:11:17.140> are<00:11:17.320> happening<00:11:17.770> and 00:11:18.170 --> 00:11:18.180 align:start position:0% on what frequencies are happening and 00:11:18.180 --> 00:11:20.670 align:start position:0% on what frequencies are happening and did<00:11:19.180> we<00:11:19.300> really<00:11:19.510> solve<00:11:19.810> the<00:11:19.959> problem<00:11:20.320> so<00:11:20.350> here 00:11:20.670 --> 00:11:20.680 align:start position:0% did we really solve the problem so here 00:11:20.680 --> 00:11:23.040 align:start position:0% did we really solve the problem so here we<00:11:20.800> can<00:11:20.920> see<00:11:21.130> that<00:11:21.430> these<00:11:21.730> should<00:11:22.240> be<00:11:22.300> zero 00:11:23.040 --> 00:11:23.050 align:start position:0% we can see that these should be zero 00:11:23.050 --> 00:11:25.199 align:start position:0% we can see that these should be zero because<00:11:23.470> this<00:11:23.680> is<00:11:23.890> an<00:11:24.250> error<00:11:24.730> that<00:11:24.940> we<00:11:25.029> don't 00:11:25.199 --> 00:11:25.209 align:start position:0% because this is an error that we don't 00:11:25.209 --> 00:11:26.699 align:start position:0% because this is an error that we don't want<00:11:25.390> to<00:11:25.420> have<00:11:25.630> but<00:11:25.930> we<00:11:25.959> can<00:11:26.230> see<00:11:26.410> it<00:11:26.500> at<00:11:26.620> like 00:11:26.699 --> 00:11:26.709 align:start position:0% want to have but we can see it at like 00:11:26.709 --> 00:11:29.070 align:start position:0% want to have but we can see it at like specific<00:11:27.370> times<00:11:27.670> and<00:11:28.000> maybe<00:11:28.570> correlate<00:11:29.050> that 00:11:29.070 --> 00:11:29.080 align:start position:0% specific times and maybe correlate that 00:11:29.080 --> 00:11:31.530 align:start position:0% specific times and maybe correlate that with<00:11:29.290> some<00:11:29.740> other<00:11:30.310> interesting<00:11:31.089> data<00:11:31.300> that 00:11:31.530 --> 00:11:31.540 align:start position:0% with some other interesting data that 00:11:31.540 --> 00:11:33.840 align:start position:0% with some other interesting data that will<00:11:31.690> help<00:11:31.720> us<00:11:31.959> identify<00:11:32.529> the<00:11:32.920> root<00:11:33.550> cause<00:11:33.820> of 00:11:33.840 --> 00:11:33.850 align:start position:0% will help us identify the root cause of 00:11:33.850 --> 00:11:36.630 align:start position:0% will help us identify the root cause of this<00:11:34.560> because<00:11:35.560> you<00:11:35.980> know<00:11:36.070> typically<00:11:36.430> it's 00:11:36.630 --> 00:11:36.640 align:start position:0% this because you know typically it's 00:11:36.640 --> 00:11:38.970 align:start position:0% this because you know typically it's easy<00:11:36.790> to<00:11:37.060> make<00:11:37.660> a<00:11:37.690> problem<00:11:38.050> go<00:11:38.140> away<00:11:38.230> but<00:11:38.560> to 00:11:38.970 --> 00:11:38.980 align:start position:0% easy to make a problem go away but to 00:11:38.980 --> 00:11:40.769 align:start position:0% easy to make a problem go away but to really<00:11:39.160> understand<00:11:39.580> what<00:11:40.060> was<00:11:40.240> the<00:11:40.420> actual 00:11:40.769 --> 00:11:40.779 align:start position:0% really understand what was the actual 00:11:40.779 --> 00:11:42.540 align:start position:0% really understand what was the actual problem<00:11:41.709> is<00:11:41.860> how<00:11:41.980> you're<00:11:42.130> going<00:11:42.220> to<00:11:42.279> get<00:11:42.399> the 00:11:42.540 --> 00:11:42.550 align:start position:0% problem is how you're going to get the 00:11:42.550 --> 00:11:48.420 align:start position:0% problem is how you're going to get the best<00:11:42.700> solution<00:11:45.810> lastly<00:11:46.810> on<00:11:46.990> monitoring<00:11:47.500> we<00:11:48.190> do 00:11:48.420 --> 00:11:48.430 align:start position:0% best solution lastly on monitoring we do 00:11:48.430 --> 00:11:51.750 align:start position:0% best solution lastly on monitoring we do something<00:11:48.910> called<00:11:49.089> canary<00:11:49.570> analysis<00:11:50.160> so<00:11:51.160> we 00:11:51.750 --> 00:11:51.760 align:start position:0% something called canary analysis so we 00:11:51.760 --> 00:11:53.639 align:start position:0% something called canary analysis so we are<00:11:52.000> always<00:11:52.240> so<00:11:52.600> when<00:11:53.080> the<00:11:53.200> test<00:11:53.380> net<00:11:53.560> is 00:11:53.639 --> 00:11:53.649 align:start position:0% are always so when the test net is 00:11:53.649 --> 00:11:55.230 align:start position:0% are always so when the test net is running<00:11:53.950> which<00:11:54.100> is<00:11:54.130> almost<00:11:54.850> all<00:11:54.970> the<00:11:55.029> time 00:11:55.230 --> 00:11:55.240 align:start position:0% running which is almost all the time 00:11:55.240 --> 00:11:57.420 align:start position:0% running which is almost all the time we're<00:11:55.690> pushing<00:11:55.990> code<00:11:56.500> out<00:11:56.680> as<00:11:56.860> frequently<00:11:57.279> as 00:11:57.420 --> 00:11:57.430 align:start position:0% we're pushing code out as frequently as 00:11:57.430 --> 00:12:01.410 align:start position:0% we're pushing code out as frequently as possible<00:11:57.540> with<00:11:58.540> and<00:11:59.200> without<00:12:00.420> catastrophic 00:12:01.410 --> 00:12:01.420 align:start position:0% possible with and without catastrophic 00:12:01.420 --> 00:12:03.389 align:start position:0% possible with and without catastrophic ly<00:12:01.570> breaking<00:12:01.930> it<00:12:02.050> so<00:12:02.260> what<00:12:02.770> we'll<00:12:02.920> do<00:12:03.070> is<00:12:03.250> we'll 00:12:03.389 --> 00:12:03.399 align:start position:0% ly breaking it so what we'll do is we'll 00:12:03.399 --> 00:12:06.480 align:start position:0% ly breaking it so what we'll do is we'll run<00:12:04.080> the<00:12:05.080> new<00:12:05.200> the<00:12:05.589> new<00:12:05.620> binary<00:12:06.130> that<00:12:06.339> we've 00:12:06.480 --> 00:12:06.490 align:start position:0% run the new the new binary that we've 00:12:06.490 --> 00:12:08.010 align:start position:0% run the new the new binary that we've just<00:12:06.850> got<00:12:07.029> the<00:12:07.209> new<00:12:07.330> image<00:12:07.570> that<00:12:07.750> we've<00:12:07.870> just 00:12:08.010 --> 00:12:08.020 align:start position:0% just got the new image that we've just 00:12:08.020 --> 00:12:10.220 align:start position:0% just got the new image that we've just created<00:12:08.110> we'll<00:12:08.920> run<00:12:09.070> it<00:12:09.220> alongside<00:12:09.580> the 00:12:10.220 --> 00:12:10.230 align:start position:0% created we'll run it alongside the 00:12:10.230 --> 00:12:14.760 align:start position:0% created we'll run it alongside the existing<00:12:11.670> the<00:12:12.670> existing<00:12:13.120> binary<00:12:13.330> and<00:12:13.770> start 00:12:14.760 --> 00:12:14.770 align:start position:0% existing the existing binary and start 00:12:14.770 --> 00:12:16.380 align:start position:0% existing the existing binary and start them<00:12:14.920> at<00:12:15.010> the<00:12:15.100> same<00:12:15.220> time<00:12:15.279> and<00:12:15.820> then<00:12:16.029> see<00:12:16.270> if 00:12:16.380 --> 00:12:16.390 align:start position:0% them at the same time and then see if 00:12:16.390 --> 00:12:18.420 align:start position:0% them at the same time and then see if there<00:12:16.570> are<00:12:16.660> any<00:12:16.870> major<00:12:17.110> regressions<00:12:17.770> so<00:12:18.100> in 00:12:18.420 --> 00:12:18.430 align:start position:0% there are any major regressions so in 00:12:18.430 --> 00:12:20.819 align:start position:0% there are any major regressions so in this<00:12:18.970> case<00:12:19.149> we<00:12:19.390> see<00:12:19.600> that<00:12:19.810> memory<00:12:20.110> usage<00:12:20.470> is<00:12:20.649> up 00:12:20.819 --> 00:12:20.829 align:start position:0% this case we see that memory usage is up 00:12:20.829 --> 00:12:24.660 align:start position:0% this case we see that memory usage is up by<00:12:21.010> 56%<00:12:21.550> and<00:12:22.029> that<00:12:22.510> is<00:12:22.630> kind<00:12:22.899> of<00:12:22.959> concerning<00:12:23.670> so 00:12:24.660 --> 00:12:24.670 align:start position:0% by 56% and that is kind of concerning so 00:12:24.670 --> 00:12:27.480 align:start position:0% by 56% and that is kind of concerning so this<00:12:24.850> canary<00:12:25.450> report<00:12:25.930> got<00:12:26.110> a<00:12:26.170> score<00:12:26.529> of<00:12:26.560> 77 00:12:27.480 --> 00:12:27.490 align:start position:0% this canary report got a score of 77 00:12:27.490 --> 00:12:30.329 align:start position:0% this canary report got a score of 77 percent<00:12:27.910> which<00:12:28.839> based<00:12:29.589> on<00:12:29.800> your<00:12:29.890> tolerance 00:12:30.329 --> 00:12:30.339 align:start position:0% percent which based on your tolerance 00:12:30.339 --> 00:12:33.030 align:start position:0% percent which based on your tolerance you<00:12:30.399> may<00:12:30.730> want<00:12:30.970> to<00:12:31.060> fail<00:12:32.020> immediately<00:12:32.260> or<00:12:32.829> not 00:12:33.030 --> 00:12:33.040 align:start position:0% you may want to fail immediately or not 00:12:33.040 --> 00:12:34.590 align:start position:0% you may want to fail immediately or not let<00:12:33.190> this<00:12:33.310> go<00:12:33.520> through<00:12:33.790> maybe<00:12:34.120> you<00:12:34.270> want<00:12:34.450> to 00:12:34.590 --> 00:12:34.600 align:start position:0% let this go through maybe you want to 00:12:34.600 --> 00:12:36.900 align:start position:0% let this go through maybe you want to have<00:12:34.959> some<00:12:35.170> manual<00:12:35.470> approval<00:12:35.890> before<00:12:36.760> pushing 00:12:36.900 --> 00:12:36.910 align:start position:0% have some manual approval before pushing 00:12:36.910 --> 00:12:40.800 align:start position:0% have some manual approval before pushing it<00:12:37.120> out<00:12:37.270> our<00:12:38.040> you<00:12:39.040> know<00:12:39.070> success<00:12:39.670> or<00:12:39.970> auto 00:12:40.800 --> 00:12:40.810 align:start position:0% it out our you know success or auto 00:12:40.810 --> 00:12:43.350 align:start position:0% it out our you know success or auto approved<00:12:41.520> criteria<00:12:42.520> is<00:12:42.610> 90<00:12:42.850> percent<00:12:43.000> so 00:12:43.350 --> 00:12:43.360 align:start position:0% approved criteria is 90 percent so 00:12:43.360 --> 00:12:44.940 align:start position:0% approved criteria is 90 percent so anything<00:12:43.630> that's<00:12:43.720> almost<00:12:44.170> almost<00:12:44.500> perfect 00:12:44.940 --> 00:12:44.950 align:start position:0% anything that's almost almost perfect 00:12:44.950 --> 00:12:47.250 align:start position:0% anything that's almost almost perfect will<00:12:45.070> go<00:12:45.190> through<00:12:45.510> anything<00:12:46.510> about<00:12:46.990> 50 00:12:47.250 --> 00:12:47.260 align:start position:0% will go through anything about 50 00:12:47.260 --> 00:12:49.380 align:start position:0% will go through anything about 50 percent<00:12:47.380> we<00:12:47.680> can<00:12:47.829> look<00:12:47.980> at<00:12:48.160> and<00:12:48.399> anything 00:12:49.380 --> 00:12:49.390 align:start position:0% percent we can look at and anything 00:12:49.390 --> 00:12:51.840 align:start position:0% percent we can look at and anything that's<00:12:49.570> below<00:12:49.779> 50<00:12:50.320> percent<00:12:50.350> will<00:12:51.040> just<00:12:51.399> stop 00:12:51.840 --> 00:12:51.850 align:start position:0% that's below 50 percent will just stop 00:12:51.850 --> 00:12:53.940 align:start position:0% that's below 50 percent will just stop immediately<00:12:52.390> and<00:12:52.600> and<00:12:52.839> we'll<00:12:53.470> take<00:12:53.589> a<00:12:53.649> look<00:12:53.860> at 00:12:53.940 --> 00:12:53.950 align:start position:0% immediately and and we'll take a look at 00:12:53.950 --> 00:12:56.490 align:start position:0% immediately and and we'll take a look at it<00:12:54.040> later<00:12:54.899> but<00:12:55.899> that's<00:12:56.020> been<00:12:56.200> really<00:12:56.320> great 00:12:56.490 --> 00:12:56.500 align:start position:0% it later but that's been really great 00:12:56.500 --> 00:12:58.829 align:start position:0% it later but that's been really great from<00:12:56.829> keeping<00:12:57.160> us<00:12:57.250> from<00:12:57.600> tearing<00:12:58.600> down<00:12:58.690> our 00:12:58.829 --> 00:12:58.839 align:start position:0% from keeping us from tearing down our 00:12:58.839 --> 00:13:04.019 align:start position:0% from keeping us from tearing down our tests<00:12:59.050> out<00:12:59.170> on<00:12:59.290> accident<00:13:02.490> okay<00:13:03.490> and<00:13:03.700> this<00:13:03.970> is 00:13:04.019 --> 00:13:04.029 align:start position:0% tests out on accident okay and this is 00:13:04.029 --> 00:13:07.290 align:start position:0% tests out on accident okay and this is something<00:13:04.690> that<00:13:04.990> that<00:13:05.610> we<00:13:06.610> that<00:13:06.940> we<00:13:07.060> really 00:13:07.290 --> 00:13:07.300 align:start position:0% something that that we that we really 00:13:07.300 --> 00:13:10.079 align:start position:0% something that that we that we really learned<00:13:07.480> pretty<00:13:08.170> recently<00:13:08.620> so<00:13:08.920> our<00:13:09.790> team<00:13:10.060> is 00:13:10.079 --> 00:13:10.089 align:start position:0% learned pretty recently so our team is 00:13:10.089 --> 00:13:12.180 align:start position:0% learned pretty recently so our team is is<00:13:10.690> very<00:13:11.050> distributed<00:13:11.410> across<00:13:11.800> the<00:13:11.890> world 00:13:12.180 --> 00:13:12.190 align:start position:0% is very distributed across the world 00:13:12.190 --> 00:13:14.430 align:start position:0% is very distributed across the world when<00:13:12.370> nobody<00:13:12.670> lives<00:13:12.880> in<00:13:13.000> the<00:13:13.089> same<00:13:13.270> city<00:13:13.510> it's 00:13:14.430 --> 00:13:14.440 align:start position:0% when nobody lives in the same city it's 00:13:14.440 --> 00:13:16.260 align:start position:0% when nobody lives in the same city it's it's<00:13:14.740> uncommon<00:13:15.520> that<00:13:15.790> we're<00:13:15.910> all<00:13:16.029> in<00:13:16.149> the<00:13:16.240> same 00:13:16.260 --> 00:13:16.270 align:start position:0% it's uncommon that we're all in the same 00:13:16.270 --> 00:13:20.300 align:start position:0% it's uncommon that we're all in the same time<00:13:16.630> zone<00:13:16.899> so<00:13:18.029> actually<00:13:19.029> being<00:13:19.209> all<00:13:19.540> here<00:13:20.020> in 00:13:20.300 --> 00:13:20.310 align:start position:0% time zone so actually being all here in 00:13:20.310 --> 00:13:23.400 align:start position:0% time zone so actually being all here in Def<00:13:21.310> Con<00:13:21.550> we're<00:13:22.240> on<00:13:22.360> the<00:13:22.449> same<00:13:22.600> time<00:13:22.839> zone<00:13:23.110> so 00:13:23.400 --> 00:13:23.410 align:start position:0% Def Con we're on the same time zone so 00:13:23.410 --> 00:13:26.069 align:start position:0% Def Con we're on the same time zone so this<00:13:23.740> this<00:13:24.449> monster<00:13:25.449> is<00:13:25.510> coming<00:13:25.600> out<00:13:25.839> a<00:13:25.870> night 00:13:26.069 --> 00:13:26.079 align:start position:0% this this monster is coming out a night 00:13:26.079 --> 00:13:27.090 align:start position:0% this this monster is coming out a night is<00:13:26.199> even<00:13:26.470> more<00:13:26.620> scarier 00:13:27.090 --> 00:13:27.100 align:start position:0% is even more scarier 00:13:27.100 --> 00:13:28.980 align:start position:0% is even more scarier because<00:13:27.370> what's<00:13:28.120> actually<00:13:28.330> happening<00:13:28.840> is 00:13:28.980 --> 00:13:28.990 align:start position:0% because what's actually happening is 00:13:28.990 --> 00:13:31.920 align:start position:0% because what's actually happening is like<00:13:29.020> when<00:13:29.500> all<00:13:30.130> of<00:13:30.400> us<00:13:30.550> in<00:13:30.790> North<00:13:31.210> America<00:13:31.690> go 00:13:31.920 --> 00:13:31.930 align:start position:0% like when all of us in North America go 00:13:31.930 --> 00:13:34.620 align:start position:0% like when all of us in North America go to<00:13:31.990> sleep<00:13:32.320> the<00:13:33.280> test<00:13:33.790> net<00:13:33.970> starts<00:13:34.270> to<00:13:34.420> fall 00:13:34.620 --> 00:13:34.630 align:start position:0% to sleep the test net starts to fall 00:13:34.630 --> 00:13:36.660 align:start position:0% to sleep the test net starts to fall apart<00:13:34.660> and<00:13:35.230> then<00:13:35.470> the<00:13:35.500> people<00:13:36.010> over<00:13:36.130> here<00:13:36.430> and 00:13:36.660 --> 00:13:36.670 align:start position:0% apart and then the people over here and 00:13:36.670 --> 00:13:39.330 align:start position:0% apart and then the people over here and in<00:13:36.880> Asia<00:13:37.120> are<00:13:37.480> stuck<00:13:38.380> with<00:13:38.530> it<00:13:38.680> and<00:13:38.710> like<00:13:39.100> kind 00:13:39.330 --> 00:13:39.340 align:start position:0% in Asia are stuck with it and like kind 00:13:39.340 --> 00:13:41.010 align:start position:0% in Asia are stuck with it and like kind of<00:13:39.370> on<00:13:39.550> their<00:13:39.700> own<00:13:39.730> by<00:13:40.120> themselves<00:13:40.660> or<00:13:40.840> maybe 00:13:41.010 --> 00:13:41.020 align:start position:0% of on their own by themselves or maybe 00:13:41.020 --> 00:13:43.230 align:start position:0% of on their own by themselves or maybe they<00:13:41.170> didn't<00:13:41.350> have<00:13:41.470> the<00:13:41.620> same<00:13:41.890> knowledge<00:13:42.580> that 00:13:43.230 --> 00:13:43.240 align:start position:0% they didn't have the same knowledge that 00:13:43.240 --> 00:13:45.060 align:start position:0% they didn't have the same knowledge that we<00:13:43.600> had<00:13:43.870> learned<00:13:44.110> that<00:13:44.320> day<00:13:44.590> and<00:13:44.800> we<00:13:44.890> go<00:13:45.010> to 00:13:45.060 --> 00:13:45.070 align:start position:0% we had learned that day and we go to 00:13:45.070 --> 00:13:47.430 align:start position:0% we had learned that day and we go to sleep<00:13:45.220> we<00:13:45.370> try<00:13:45.610> to<00:13:45.670> pass<00:13:45.850> off<00:13:46.090> things 00:13:47.430 --> 00:13:47.440 align:start position:0% sleep we try to pass off things 00:13:47.440 --> 00:13:49.380 align:start position:0% sleep we try to pass off things but<00:13:47.710> it's<00:13:47.830> always<00:13:48.040> like<00:13:48.400> when<00:13:48.850> you<00:13:49.000> when 00:13:49.380 --> 00:13:49.390 align:start position:0% but it's always like when you when 00:13:49.390 --> 00:13:51.420 align:start position:0% but it's always like when you when everything<00:13:49.870> is<00:13:50.080> sounds<00:13:50.830> really<00:13:51.040> good<00:13:51.220> and 00:13:51.420 --> 00:13:51.430 align:start position:0% everything is sounds really good and 00:13:51.430 --> 00:13:53.310 align:start position:0% everything is sounds really good and really<00:13:51.520> nice<00:13:51.730> and<00:13:52.120> when<00:13:52.480> you<00:13:52.600> least<00:13:52.780> expect<00:13:52.900> it 00:13:53.310 --> 00:13:53.320 align:start position:0% really nice and when you least expect it 00:13:53.320 --> 00:13:55.590 align:start position:0% really nice and when you least expect it things<00:13:53.710> start<00:13:53.980> to<00:13:54.010> fall<00:13:54.130> apart<00:13:54.520> so<00:13:54.730> that's 00:13:55.590 --> 00:13:55.600 align:start position:0% things start to fall apart so that's 00:13:55.600 --> 00:13:57.960 align:start position:0% things start to fall apart so that's been<00:13:55.780> super<00:13:55.930> interesting<00:13:56.530> when<00:13:57.280> we<00:13:57.430> landed 00:13:57.960 --> 00:13:57.970 align:start position:0% been super interesting when we landed 00:13:57.970 --> 00:14:01.020 align:start position:0% been super interesting when we landed here<00:13:58.030> our<00:13:58.540> for<00:13:59.290> Def<00:13:59.740> Con<00:13:59.950> day<00:14:00.130> one<00:14:00.370> we<00:14:00.850> were 00:14:01.020 --> 00:14:01.030 align:start position:0% here our for Def Con day one we were 00:14:01.030 --> 00:14:03.690 align:start position:0% here our for Def Con day one we were right<00:14:01.630> all<00:14:01.840> in<00:14:01.930> the<00:14:02.020> same<00:14:02.170> time<00:14:02.440> zone<00:14:02.470> and<00:14:02.950> when 00:14:03.690 --> 00:14:03.700 align:start position:0% right all in the same time zone and when 00:14:03.700 --> 00:14:05.370 align:start position:0% right all in the same time zone and when we<00:14:03.820> woke<00:14:03.970> up<00:14:04.150> in<00:14:04.330> the<00:14:04.420> morning<00:14:04.600> the<00:14:04.960> test<00:14:05.230> and 00:14:05.370 --> 00:14:05.380 align:start position:0% we woke up in the morning the test and 00:14:05.380 --> 00:14:07.380 align:start position:0% we woke up in the morning the test and it<00:14:05.440> had<00:14:05.590> been<00:14:05.770> stuck<00:14:06.100> for<00:14:06.340> two<00:14:06.490> hours<00:14:06.850> and<00:14:07.180> no 00:14:07.380 --> 00:14:07.390 align:start position:0% it had been stuck for two hours and no 00:14:07.390 --> 00:14:10.290 align:start position:0% it had been stuck for two hours and no one<00:14:07.510> was<00:14:07.630> around<00:14:07.900> to<00:14:08.170> see<00:14:08.500> it<00:14:08.620> and<00:14:08.860> we<00:14:09.300> couldn't 00:14:10.290 --> 00:14:10.300 align:start position:0% one was around to see it and we couldn't 00:14:10.300 --> 00:14:12.440 align:start position:0% one was around to see it and we couldn't fix<00:14:10.540> it<00:14:10.720> in<00:14:10.930> time<00:14:11.170> before<00:14:11.410> it<00:14:12.070> was<00:14:12.190> just 00:14:12.440 --> 00:14:12.450 align:start position:0% fix it in time before it was just 00:14:12.450 --> 00:14:14.820 align:start position:0% fix it in time before it was just unrecoverable<00:14:13.450> amount<00:14:14.380> of<00:14:14.470> time<00:14:14.650> since 00:14:14.820 --> 00:14:14.830 align:start position:0% unrecoverable amount of time since 00:14:14.830 --> 00:14:17.010 align:start position:0% unrecoverable amount of time since finality<00:14:15.400> so<00:14:15.640> only<00:14:16.540> ended<00:14:16.780> up<00:14:16.870> just 00:14:17.010 --> 00:14:17.020 align:start position:0% finality so only ended up just 00:14:17.020 --> 00:14:18.180 align:start position:0% finality so only ended up just restarting<00:14:17.470> it<00:14:17.560> but<00:14:17.680> that<00:14:17.800> was<00:14:17.890> something 00:14:18.180 --> 00:14:18.190 align:start position:0% restarting it but that was something 00:14:18.190 --> 00:14:21.150 align:start position:0% restarting it but that was something something<00:14:18.670> that<00:14:18.940> we've<00:14:19.060> learned<00:14:19.330> is<00:14:19.630> that<00:14:20.160> if 00:14:21.150 --> 00:14:21.160 align:start position:0% something that we've learned is that if 00:14:21.160 --> 00:14:23.940 align:start position:0% something that we've learned is that if you<00:14:21.400> can<00:14:21.700> put<00:14:22.510> somebody<00:14:22.750> so<00:14:23.530> you're<00:14:23.710> always 00:14:23.940 --> 00:14:23.950 align:start position:0% you can put somebody so you're always 00:14:23.950 --> 00:14:27.330 align:start position:0% you can put somebody so you're always awake<00:14:24.420> 24/7<00:14:25.420> or<00:14:25.630> the<00:14:26.350> the<00:14:26.560> actual<00:14:26.830> solution<00:14:27.100> we 00:14:27.330 --> 00:14:27.340 align:start position:0% awake 24/7 or the the actual solution we 00:14:27.340 --> 00:14:30.420 align:start position:0% awake 24/7 or the the actual solution we did<00:14:27.490> for<00:14:27.640> this<00:14:27.760> was<00:14:28.060> to<00:14:28.900> have<00:14:29.110> the<00:14:29.440> alerting 00:14:30.420 --> 00:14:30.430 align:start position:0% did for this was to have the alerting 00:14:30.430 --> 00:14:33.390 align:start position:0% did for this was to have the alerting that<00:14:30.670> goes<00:14:30.880> to<00:14:31.270> your<00:14:31.780> phone<00:14:31.990> you<00:14:32.290> know<00:14:32.890> pass 00:14:33.390 --> 00:14:33.400 align:start position:0% that goes to your phone you know pass 00:14:33.400 --> 00:14:36.300 align:start position:0% that goes to your phone you know pass through<00:14:33.760> your<00:14:34.000> do<00:14:34.540> not<00:14:34.690> disturb<00:14:35.100> wake<00:14:36.100> you<00:14:36.250> up 00:14:36.300 --> 00:14:36.310 align:start position:0% through your do not disturb wake you up 00:14:36.310 --> 00:14:37.890 align:start position:0% through your do not disturb wake you up in<00:14:36.460> the<00:14:36.490> middle<00:14:36.640> of<00:14:36.820> night<00:14:36.970> and<00:14:37.330> make<00:14:37.720> sure 00:14:37.890 --> 00:14:37.900 align:start position:0% in the middle of night and make sure 00:14:37.900 --> 00:14:40.080 align:start position:0% in the middle of night and make sure that<00:14:38.080> like<00:14:38.320> when<00:14:38.560> when<00:14:39.310> no<00:14:39.580> finalities 00:14:40.080 --> 00:14:40.090 align:start position:0% that like when when no finalities 00:14:40.090 --> 00:14:42.270 align:start position:0% that like when when no finalities happened<00:14:40.540> in<00:14:40.660> 100<00:14:41.140> ebox<00:14:41.500> that<00:14:41.740> someone<00:14:42.130> is 00:14:42.270 --> 00:14:42.280 align:start position:0% happened in 100 ebox that someone is 00:14:42.280 --> 00:14:44.250 align:start position:0% happened in 100 ebox that someone is there<00:14:42.550> looking<00:14:42.940> at<00:14:43.090> that<00:14:43.270> because<00:14:43.510> that's<00:14:44.050> a 00:14:44.250 --> 00:14:44.260 align:start position:0% there looking at that because that's a 00:14:44.260 --> 00:14:47.930 align:start position:0% there looking at that because that's a really<00:14:44.500> really<00:14:44.710> bad<00:14:44.860> thing<00:14:45.850> to<00:14:46.000> happen<00:14:46.150> so 00:14:47.930 --> 00:14:47.940 align:start position:0% really really bad thing to happen so 00:14:47.940 --> 00:14:51.270 align:start position:0% really really bad thing to happen so lastly<00:14:48.940> like<00:14:49.150> why<00:14:49.600> did<00:14:50.320> we<00:14:50.350> do<00:14:50.620> a<00:14:50.650> test<00:14:50.890> net<00:14:51.130> so 00:14:51.270 --> 00:14:51.280 align:start position:0% lastly like why did we do a test net so 00:14:51.280 --> 00:14:53.340 align:start position:0% lastly like why did we do a test net so early<00:14:51.640> like<00:14:52.000> I<00:14:52.120> said<00:14:52.330> we've<00:14:52.510> been<00:14:52.540> doing<00:14:53.140> one 00:14:53.340 --> 00:14:53.350 align:start position:0% early like I said we've been doing one 00:14:53.350 --> 00:14:56.070 align:start position:0% early like I said we've been doing one since<00:14:53.620> April<00:14:54.070> and<00:14:54.370> this<00:14:54.580> was<00:14:54.760> before<00:14:55.000> we<00:14:55.750> even 00:14:56.070 --> 00:14:56.080 align:start position:0% since April and this was before we even 00:14:56.080 --> 00:14:58.050 align:start position:0% since April and this was before we even had<00:14:56.410> a<00:14:56.440> networking<00:14:56.770> spec<00:14:57.190> so<00:14:57.370> we<00:14:57.490> kind<00:14:57.730> of<00:14:57.760> made 00:14:58.050 --> 00:14:58.060 align:start position:0% had a networking spec so we kind of made 00:14:58.060 --> 00:15:00.150 align:start position:0% had a networking spec so we kind of made one<00:14:58.270> up<00:14:58.450> to<00:14:59.170> get<00:14:59.320> our<00:14:59.440> clients<00:14:59.770> to<00:14:59.860> talk<00:15:00.010> to 00:15:00.150 --> 00:15:00.160 align:start position:0% one up to get our clients to talk to 00:15:00.160 --> 00:15:01.320 align:start position:0% one up to get our clients to talk to each<00:15:00.250> other<00:15:00.430> and<00:15:00.640> we<00:15:00.730> knew<00:15:00.880> it<00:15:00.970> wasn't<00:15:01.120> gonna 00:15:01.320 --> 00:15:01.330 align:start position:0% each other and we knew it wasn't gonna 00:15:01.330 --> 00:15:02.580 align:start position:0% each other and we knew it wasn't gonna be<00:15:01.420> final<00:15:01.810> we<00:15:01.900> knew<00:15:02.020> we're<00:15:02.170> gonna<00:15:02.260> have<00:15:02.470> to 00:15:02.580 --> 00:15:02.590 align:start position:0% be final we knew we're gonna have to 00:15:02.590 --> 00:15:05.700 align:start position:0% be final we knew we're gonna have to rewrite<00:15:02.890> everything<00:15:03.660> in<00:15:04.660> that<00:15:05.080> regards<00:15:05.440> but 00:15:05.700 --> 00:15:05.710 align:start position:0% rewrite everything in that regards but 00:15:05.710 --> 00:15:08.700 align:start position:0% rewrite everything in that regards but why<00:15:06.490> did<00:15:06.670> we<00:15:06.790> do<00:15:06.910> this<00:15:07.200> we<00:15:08.200> actually<00:15:08.320> learned<00:15:08.620> a 00:15:08.700 --> 00:15:08.710 align:start position:0% why did we do this we actually learned a 00:15:08.710 --> 00:15:10.650 align:start position:0% why did we do this we actually learned a lot<00:15:08.800> of<00:15:08.860> things<00:15:09.130> besides<00:15:09.580> those<00:15:09.760> those<00:15:10.480> few 00:15:10.650 --> 00:15:10.660 align:start position:0% lot of things besides those those few 00:15:10.660 --> 00:15:12.750 align:start position:0% lot of things besides those those few lessons<00:15:10.840> so<00:15:11.170> we're<00:15:11.680> able<00:15:11.740> to<00:15:11.890> find<00:15:12.100> race 00:15:12.750 --> 00:15:12.760 align:start position:0% lessons so we're able to find race 00:15:12.760 --> 00:15:15.600 align:start position:0% lessons so we're able to find race conditions<00:15:13.210> which<00:15:13.630> you<00:15:13.930> can<00:15:14.530> check<00:15:15.010> for<00:15:15.250> with 00:15:15.600 --> 00:15:15.610 align:start position:0% conditions which you can check for with 00:15:15.610 --> 00:15:17.490 align:start position:0% conditions which you can check for with race<00:15:16.180> detection<00:15:16.840> but<00:15:16.990> it's<00:15:17.170> not<00:15:17.350> always 00:15:17.490 --> 00:15:17.500 align:start position:0% race detection but it's not always 00:15:17.500 --> 00:15:20.460 align:start position:0% race detection but it's not always perfect<00:15:18.720> one<00:15:19.720> of<00:15:19.780> the<00:15:19.840> biggest<00:15:20.080> things<00:15:20.260> is 00:15:20.460 --> 00:15:20.470 align:start position:0% perfect one of the biggest things is 00:15:20.470 --> 00:15:24.060 align:start position:0% perfect one of the biggest things is it's<00:15:21.010> long<00:15:21.370> long<00:15:22.090> lasting<00:15:22.720> memory<00:15:23.680> leaks<00:15:23.860> so 00:15:24.060 --> 00:15:24.070 align:start position:0% it's long long lasting memory leaks so 00:15:24.070 --> 00:15:27.390 align:start position:0% it's long long lasting memory leaks so things<00:15:24.340> that<00:15:24.460> grow<00:15:24.970> over<00:15:25.330> many<00:15:26.200> days<00:15:26.410> will 00:15:27.390 --> 00:15:27.400 align:start position:0% things that grow over many days will 00:15:27.400 --> 00:15:29.910 align:start position:0% things that grow over many days will have<00:15:27.550> binary<00:15:28.240> that<00:15:28.630> runs<00:15:28.810> for<00:15:29.110> maybe<00:15:29.650> a<00:15:29.710> week 00:15:29.910 --> 00:15:29.920 align:start position:0% have binary that runs for maybe a week 00:15:29.920 --> 00:15:31.260 align:start position:0% have binary that runs for maybe a week and<00:15:30.040> we'll<00:15:30.220> see<00:15:30.400> that<00:15:30.430> the<00:15:30.730> memory<00:15:30.970> use<00:15:31.090> is 00:15:31.260 --> 00:15:31.270 align:start position:0% and we'll see that the memory use is 00:15:31.270 --> 00:15:33.870 align:start position:0% and we'll see that the memory use is just<00:15:31.510> slowly<00:15:31.750> slowly<00:15:32.080> slowly<00:15:32.200> going<00:15:32.530> up<00:15:32.880> when 00:15:33.870 --> 00:15:33.880 align:start position:0% just slowly slowly slowly going up when 00:15:33.880 --> 00:15:35.160 align:start position:0% just slowly slowly slowly going up when you're<00:15:33.970> running<00:15:34.150> this<00:15:34.210> on<00:15:34.360> your<00:15:34.480> laptop<00:15:34.840> you 00:15:35.160 --> 00:15:35.170 align:start position:0% you're running this on your laptop you 00:15:35.170 --> 00:15:37.290 align:start position:0% you're running this on your laptop you don't<00:15:35.800> ever<00:15:35.980> see<00:15:36.220> it<00:15:36.310> we<00:15:36.490> don't<00:15:36.700> never<00:15:36.940> see<00:15:37.210> it 00:15:37.290 --> 00:15:37.300 align:start position:0% don't ever see it we don't never see it 00:15:37.300 --> 00:15:39.300 align:start position:0% don't ever see it we don't never see it in<00:15:37.420> a<00:15:37.510> unit<00:15:37.750> test<00:15:37.900> so<00:15:38.290> these<00:15:38.770> kind<00:15:39.040> of<00:15:39.100> things 00:15:39.300 --> 00:15:39.310 align:start position:0% in a unit test so these kind of things 00:15:39.310 --> 00:15:40.939 align:start position:0% in a unit test so these kind of things are<00:15:39.490> super<00:15:39.760> helpful 00:15:40.939 --> 00:15:40.949 align:start position:0% are super helpful 00:15:40.949 --> 00:15:43.879 align:start position:0% are super helpful UX<00:15:41.699> problems<00:15:41.970> users<00:15:42.929> come<00:15:43.079> to<00:15:43.259> us<00:15:43.350> and<00:15:43.529> say<00:15:43.709> hey 00:15:43.879 --> 00:15:43.889 align:start position:0% UX problems users come to us and say hey 00:15:43.889 --> 00:15:46.609 align:start position:0% UX problems users come to us and say hey I<00:15:43.920> don't<00:15:44.160> understand<00:15:44.999> why<00:15:45.899> you<00:15:45.959> won't<00:15:46.350> let<00:15:46.559> me 00:15:46.609 --> 00:15:46.619 align:start position:0% I don't understand why you won't let me 00:15:46.619 --> 00:15:49.280 align:start position:0% I don't understand why you won't let me validate<00:15:47.249> like<00:15:47.429> what's<00:15:47.639> going<00:15:47.819> on<00:15:47.939> and<00:15:48.290> you 00:15:49.280 --> 00:15:49.290 align:start position:0% validate like what's going on and you 00:15:49.290 --> 00:15:50.840 align:start position:0% validate like what's going on and you know<00:15:49.379> that's<00:15:49.619> been<00:15:49.799> great<00:15:49.980> to<00:15:50.189> just<00:15:50.459> make<00:15:50.730> the 00:15:50.840 --> 00:15:50.850 align:start position:0% know that's been great to just make the 00:15:50.850 --> 00:15:52.489 align:start position:0% know that's been great to just make the client<00:15:51.179> better<00:15:51.269> for<00:15:51.689> everyone<00:15:51.989> for<00:15:52.230> usability 00:15:52.489 --> 00:15:52.499 align:start position:0% client better for everyone for usability 00:15:52.499 --> 00:15:55.189 align:start position:0% client better for everyone for usability so<00:15:52.980> we<00:15:53.160> can<00:15:53.369> help<00:15:54.119> understand<00:15:54.569> what<00:15:54.839> users<00:15:55.170> are 00:15:55.189 --> 00:15:55.199 align:start position:0% so we can help understand what users are 00:15:55.199 --> 00:15:57.799 align:start position:0% so we can help understand what users are frustrated<00:15:55.739> with<00:15:55.859> or<00:15:56.009> what<00:15:56.129> they<00:15:56.249> want<00:15:56.809> and 00:15:57.799 --> 00:15:57.809 align:start position:0% frustrated with or what they want and 00:15:57.809 --> 00:15:59.470 align:start position:0% frustrated with or what they want and then<00:15:57.959> we<00:15:58.049> even<00:15:58.199> actually<00:15:58.499> run<00:15:58.649> into 00:15:59.470 --> 00:15:59.480 align:start position:0% then we even actually run into 00:15:59.480 --> 00:16:02.179 align:start position:0% then we even actually run into aggregation<00:16:00.480> issues<00:16:00.749> so<00:16:00.989> some<00:16:01.259> recent<00:16:02.040> thing 00:16:02.179 --> 00:16:02.189 align:start position:0% aggregation issues so some recent thing 00:16:02.189 --> 00:16:04.759 align:start position:0% aggregation issues so some recent thing that<00:16:02.369> we<00:16:02.519> learned<00:16:02.850> is<00:16:03.089> that<00:16:03.449> you<00:16:04.230> can't 00:16:04.759 --> 00:16:04.769 align:start position:0% that we learned is that you can't 00:16:04.769 --> 00:16:08.900 align:start position:0% that we learned is that you can't overlap<00:16:07.100> aggregated<00:16:08.100> at<00:16:08.189> two<00:16:08.339> stations<00:16:08.699> that 00:16:08.900 --> 00:16:08.910 align:start position:0% overlap aggregated at two stations that 00:16:08.910 --> 00:16:11.509 align:start position:0% overlap aggregated at two stations that are<00:16:08.970> already<00:16:09.269> aggregated<00:16:09.989> so<00:16:10.290> we<00:16:11.040> were<00:16:11.220> seeing 00:16:11.509 --> 00:16:11.519 align:start position:0% are already aggregated so we were seeing 00:16:11.519 --> 00:16:14.509 align:start position:0% are already aggregated so we were seeing we<00:16:12.089> were<00:16:12.179> trying<00:16:12.480> to<00:16:12.980> aggregate<00:16:13.980> those<00:16:14.249> and 00:16:14.509 --> 00:16:14.519 align:start position:0% we were trying to aggregate those and 00:16:14.519 --> 00:16:16.699 align:start position:0% we were trying to aggregate those and they<00:16:14.669> were<00:16:14.759> failing<00:16:14.970> but<00:16:15.749> silently<00:16:16.290> failing 00:16:16.699 --> 00:16:16.709 align:start position:0% they were failing but silently failing 00:16:16.709 --> 00:16:18.489 align:start position:0% they were failing but silently failing for<00:16:16.980> some<00:16:17.129> reason<00:16:17.429> and<00:16:17.639> then<00:16:17.819> we<00:16:17.999> were<00:16:18.119> getting 00:16:18.489 --> 00:16:18.499 align:start position:0% for some reason and then we were getting 00:16:18.499 --> 00:16:21.889 align:start position:0% for some reason and then we were getting this<00:16:19.499> is<00:16:19.679> not<00:16:19.889> very<00:16:20.129> consistent<00:16:20.669> graph<00:16:21.600> of 00:16:21.889 --> 00:16:21.899 align:start position:0% this is not very consistent graph of 00:16:21.899 --> 00:16:24.049 align:start position:0% this is not very consistent graph of finality<00:16:22.259> so<00:16:22.739> that<00:16:23.249> was<00:16:23.399> something<00:16:23.759> that<00:16:23.790> we 00:16:24.049 --> 00:16:24.059 align:start position:0% finality so that was something that we 00:16:24.059 --> 00:16:25.819 align:start position:0% finality so that was something that we only<00:16:24.239> realized<00:16:24.660> by<00:16:24.899> running<00:16:25.169> the<00:16:25.559> test 00:16:25.819 --> 00:16:25.829 align:start position:0% only realized by running the test 00:16:25.829 --> 00:16:28.659 align:start position:0% only realized by running the test Network<00:16:26.069> for<00:16:26.279> a<00:16:26.339> few<00:16:26.489> weeks<00:16:26.899> and<00:16:27.899> then<00:16:28.319> lastly 00:16:28.659 --> 00:16:28.669 align:start position:0% Network for a few weeks and then lastly 00:16:28.669 --> 00:16:31.129 align:start position:0% Network for a few weeks and then lastly testing<00:16:29.669> gaps<00:16:29.879> things<00:16:30.329> we<00:16:30.540> can<00:16:30.689> do<00:16:30.809> to<00:16:30.959> make<00:16:31.079> it 00:16:31.129 --> 00:16:31.139 align:start position:0% testing gaps things we can do to make it 00:16:31.139 --> 00:16:33.439 align:start position:0% testing gaps things we can do to make it better<00:16:31.259> so<00:16:32.009> like<00:16:32.160> the<00:16:32.279> canary<00:16:32.549> analysis<00:16:33.059> and 00:16:33.439 --> 00:16:33.449 align:start position:0% better so like the canary analysis and 00:16:33.449 --> 00:16:35.479 align:start position:0% better so like the canary analysis and and<00:16:33.720> other<00:16:33.989> other<00:16:34.410> like<00:16:34.829> long-lasting 00:16:35.479 --> 00:16:35.489 align:start position:0% and other other like long-lasting 00:16:35.489 --> 00:16:39.169 align:start position:0% and other other like long-lasting tooling<00:16:35.850> to<00:16:36.139> make<00:16:37.139> the<00:16:37.919> code<00:16:38.160> safer<00:16:38.579> and<00:16:38.850> to<00:16:38.939> be 00:16:39.169 --> 00:16:39.179 align:start position:0% tooling to make the code safer and to be 00:16:39.179 --> 00:16:40.909 align:start position:0% tooling to make the code safer and to be able<00:16:39.269> to<00:16:39.419> have<00:16:39.869> releases<00:16:40.350> more<00:16:40.499> frequently 00:16:40.909 --> 00:16:40.919 align:start position:0% able to have releases more frequently 00:16:40.919 --> 00:16:44.449 align:start position:0% able to have releases more frequently and<00:16:41.249> just<00:16:41.549> to<00:16:41.639> make<00:16:41.790> it<00:16:41.819> a<00:16:42.149> lot<00:16:42.660> more<00:16:42.839> secure<00:16:43.459> so 00:16:44.449 --> 00:16:44.459 align:start position:0% and just to make it a lot more secure so 00:16:44.459 --> 00:16:47.140 align:start position:0% and just to make it a lot more secure so that's<00:16:44.759> it<00:16:44.879> for<00:16:45.029> me<00:16:45.149> thanks<00:16:45.720> everybody 00:16:47.140 --> 00:16:47.150 align:start position:0% that's it for me thanks everybody 00:16:47.150 --> 00:16:57.350 align:start position:0% that's it for me thanks everybody [Applause] 00:16:57.350 --> 00:16:57.360 align:start position:0% 00:16:57.360 --> 00:16:59.420 align:start position:0% you