WEBVTT Kind: captions Language: en 00:00:00.000 --> 00:00:04.350 align:start position:0% foreign 00:00:04.350 --> 00:00:04.360 align:start position:0% 00:00:04.360 --> 00:00:12.129 align:start position:0% [Music] 00:00:12.129 --> 00:00:12.139 align:start position:0% 00:00:12.139 --> 00:00:14.749 align:start position:0% really<00:00:13.139> just<00:00:13.320> our<00:00:13.559> story<00:00:13.799> and<00:00:14.219> how<00:00:14.519> we've 00:00:14.749 --> 00:00:14.759 align:start position:0% really just our story and how we've 00:00:14.759 --> 00:00:16.070 align:start position:0% really just our story and how we've developed<00:00:15.000> lodestar<00:00:15.420> over<00:00:15.599> the<00:00:15.719> past<00:00:15.839> three 00:00:16.070 --> 00:00:16.080 align:start position:0% developed lodestar over the past three 00:00:16.080 --> 00:00:19.070 align:start position:0% developed lodestar over the past three years<00:00:16.320> so<00:00:16.980> about<00:00:17.220> us<00:00:17.580> my<00:00:18.300> name<00:00:18.420> is<00:00:18.480> Cayman<00:00:18.779> I 00:00:19.070 --> 00:00:19.080 align:start position:0% years so about us my name is Cayman I 00:00:19.080 --> 00:00:21.849 align:start position:0% years so about us my name is Cayman I work<00:00:19.140> at<00:00:19.320> chainsafe<00:00:19.980> it<00:00:20.580> is<00:00:20.760> a<00:00:20.939> great<00:00:21.119> company 00:00:21.849 --> 00:00:21.859 align:start position:0% work at chainsafe it is a great company 00:00:21.859 --> 00:00:24.109 align:start position:0% work at chainsafe it is a great company we're<00:00:22.859> hiring<00:00:23.220> we<00:00:23.640> basically<00:00:23.939> build 00:00:24.109 --> 00:00:24.119 align:start position:0% we're hiring we basically build 00:00:24.119 --> 00:00:26.570 align:start position:0% we're hiring we basically build blockchains<00:00:24.900> we<00:00:25.800> have<00:00:26.039> a<00:00:26.220> whole<00:00:26.340> bunch<00:00:26.460> of 00:00:26.570 --> 00:00:26.580 align:start position:0% blockchains we have a whole bunch of 00:00:26.580 --> 00:00:28.849 align:start position:0% blockchains we have a whole bunch of layer<00:00:26.880> one<00:00:27.000> projects<00:00:27.599> lodestar<00:00:28.439> is<00:00:28.560> one<00:00:28.740> of 00:00:28.849 --> 00:00:28.859 align:start position:0% layer one projects lodestar is one of 00:00:28.859 --> 00:00:30.529 align:start position:0% layer one projects lodestar is one of them<00:00:28.980> but<00:00:29.340> if<00:00:29.760> you<00:00:29.880> are<00:00:30.060> interested<00:00:30.420> in 00:00:30.529 --> 00:00:30.539 align:start position:0% them but if you are interested in 00:00:30.539 --> 00:00:32.030 align:start position:0% them but if you are interested in getting<00:00:30.660> into<00:00:30.840> core<00:00:31.080> development<00:00:31.439> we're<00:00:31.800> a 00:00:32.030 --> 00:00:32.040 align:start position:0% getting into core development we're a 00:00:32.040 --> 00:00:34.130 align:start position:0% getting into core development we're a really<00:00:32.099> great<00:00:32.399> place<00:00:32.820> great<00:00:33.180> culture<00:00:33.600> so 00:00:34.130 --> 00:00:34.140 align:start position:0% really great place great culture so 00:00:34.140 --> 00:00:36.709 align:start position:0% really great place great culture so lodestar<00:00:34.739> is<00:00:35.160> it's<00:00:35.700> one<00:00:36.000> of<00:00:36.120> these<00:00:36.239> projects 00:00:36.709 --> 00:00:36.719 align:start position:0% lodestar is it's one of these projects 00:00:36.719 --> 00:00:39.770 align:start position:0% lodestar is it's one of these projects at<00:00:37.020> chain<00:00:37.380> safe<00:00:37.680> we<00:00:38.219> we're<00:00:39.360> um<00:00:39.360> we're 00:00:39.770 --> 00:00:39.780 align:start position:0% at chain safe we we're um we're 00:00:39.780 --> 00:00:42.530 align:start position:0% at chain safe we we're um we're basically<00:00:40.020> an<00:00:40.260> eth2<00:00:40.739> or<00:00:41.160> ethereum<00:00:41.760> consensus 00:00:42.530 --> 00:00:42.540 align:start position:0% basically an eth2 or ethereum consensus 00:00:42.540 --> 00:00:45.650 align:start position:0% basically an eth2 or ethereum consensus ecosystem<00:00:43.379> written<00:00:43.800> entirely<00:00:44.160> in<00:00:44.340> typescript 00:00:45.650 --> 00:00:45.660 align:start position:0% ecosystem written entirely in typescript 00:00:45.660 --> 00:00:48.049 align:start position:0% ecosystem written entirely in typescript um<00:00:45.719> we<00:00:46.200> really<00:00:46.260> take<00:00:46.440> the<00:00:46.620> open<00:00:46.739> source<00:00:47.340> ethos 00:00:48.049 --> 00:00:48.059 align:start position:0% um we really take the open source ethos 00:00:48.059 --> 00:00:50.690 align:start position:0% um we really take the open source ethos to<00:00:48.300> heart<00:00:48.480> we<00:00:49.320> have<00:00:49.379> all<00:00:49.620> of<00:00:49.739> our<00:00:49.980> meetings 00:00:50.690 --> 00:00:50.700 align:start position:0% to heart we have all of our meetings 00:00:50.700 --> 00:00:52.610 align:start position:0% to heart we have all of our meetings they're<00:00:51.120> public<00:00:51.420> everything<00:00:52.079> we<00:00:52.320> do<00:00:52.500> is 00:00:52.610 --> 00:00:52.620 align:start position:0% they're public everything we do is 00:00:52.620 --> 00:00:54.650 align:start position:0% they're public everything we do is public<00:00:52.860> and<00:00:53.579> we<00:00:53.760> really<00:00:53.879> encourage<00:00:54.239> anyone 00:00:54.650 --> 00:00:54.660 align:start position:0% public and we really encourage anyone 00:00:54.660 --> 00:00:56.930 align:start position:0% public and we really encourage anyone who<00:00:55.199> wants<00:00:55.440> to<00:00:55.500> contribute<00:00:55.920> we<00:00:56.219> we<00:00:56.399> try<00:00:56.820> to 00:00:56.930 --> 00:00:56.940 align:start position:0% who wants to contribute we we try to 00:00:56.940 --> 00:00:58.130 align:start position:0% who wants to contribute we we try to help<00:00:57.059> people<00:00:57.239> out<00:00:57.420> and<00:00:57.600> get<00:00:57.780> get<00:00:57.899> people 00:00:58.130 --> 00:00:58.140 align:start position:0% help people out and get get people 00:00:58.140 --> 00:01:00.590 align:start position:0% help people out and get get people involved<00:00:58.739> a<00:00:59.460> lot<00:00:59.640> of<00:00:59.760> our<00:00:59.940> team<00:01:00.239> members 00:01:00.590 --> 00:01:00.600 align:start position:0% involved a lot of our team members 00:01:00.600 --> 00:01:04.189 align:start position:0% involved a lot of our team members actually<00:01:00.899> started<00:01:01.760> started<00:01:02.760> their<00:01:03.719> work<00:01:03.960> on 00:01:04.189 --> 00:01:04.199 align:start position:0% actually started started their work on 00:01:04.199 --> 00:01:05.750 align:start position:0% actually started started their work on the<00:01:04.320> project<00:01:04.500> through<00:01:04.979> their<00:01:05.220> open<00:01:05.339> source 00:01:05.750 --> 00:01:05.760 align:start position:0% the project through their open source 00:01:05.760 --> 00:01:07.670 align:start position:0% the project through their open source contributions<00:01:06.299> and<00:01:06.659> it<00:01:06.840> was<00:01:06.960> kind<00:01:07.080> of<00:01:07.200> like<00:01:07.320> a 00:01:07.670 --> 00:01:07.680 align:start position:0% contributions and it was kind of like a 00:01:07.680 --> 00:01:09.890 align:start position:0% contributions and it was kind of like a open<00:01:07.920> source<00:01:08.460> to<00:01:08.760> contract<00:01:09.119> to<00:01:09.299> hire<00:01:09.540> kind<00:01:09.780> of 00:01:09.890 --> 00:01:09.900 align:start position:0% open source to contract to hire kind of 00:01:09.900 --> 00:01:13.190 align:start position:0% open source to contract to hire kind of a<00:01:10.020> situation<00:01:10.380> so<00:01:10.799> we<00:01:11.400> really<00:01:11.640> we<00:01:12.240> really<00:01:12.479> like 00:01:13.190 --> 00:01:13.200 align:start position:0% a situation so we really we really like 00:01:13.200 --> 00:01:16.070 align:start position:0% a situation so we really we really like that<00:01:13.619> that<00:01:13.920> kind<00:01:14.220> of<00:01:14.340> ethos<00:01:15.180> but<00:01:15.900> we're 00:01:16.070 --> 00:01:16.080 align:start position:0% that that kind of ethos but we're 00:01:16.080 --> 00:01:19.070 align:start position:0% that that kind of ethos but we're talking<00:01:16.439> about<00:01:16.560> today<00:01:16.799> is<00:01:17.580> specifically<00:01:18.180> our 00:01:19.070 --> 00:01:19.080 align:start position:0% talking about today is specifically our 00:01:19.080 --> 00:01:20.929 align:start position:0% talking about today is specifically our consensus<00:01:19.560> client<00:01:19.920> and<00:01:20.220> how<00:01:20.400> we<00:01:20.520> got<00:01:20.700> from 00:01:20.929 --> 00:01:20.939 align:start position:0% consensus client and how we got from 00:01:20.939 --> 00:01:24.530 align:start position:0% consensus client and how we got from like<00:01:21.500> prototyping<00:01:22.700> typical<00:01:23.700> kind<00:01:23.939> of<00:01:24.060> code<00:01:24.240> to 00:01:24.530 --> 00:01:24.540 align:start position:0% like prototyping typical kind of code to 00:01:24.540 --> 00:01:26.210 align:start position:0% like prototyping typical kind of code to something<00:01:24.900> that's<00:01:25.080> like<00:01:25.439> production<00:01:25.979> ready 00:01:26.210 --> 00:01:26.220 align:start position:0% something that's like production ready 00:01:26.220 --> 00:01:27.830 align:start position:0% something that's like production ready something<00:01:26.640> that<00:01:26.939> you<00:01:27.119> can<00:01:27.180> really<00:01:27.360> rely<00:01:27.720> on 00:01:27.830 --> 00:01:27.840 align:start position:0% something that you can really rely on 00:01:27.840 --> 00:01:30.230 align:start position:0% something that you can really rely on something<00:01:28.619> that<00:01:28.920> doesn't<00:01:29.100> just<00:01:29.400> blow<00:01:29.759> up<00:01:29.820> all 00:01:30.230 --> 00:01:30.240 align:start position:0% something that doesn't just blow up all 00:01:30.240 --> 00:01:32.210 align:start position:0% something that doesn't just blow up all the<00:01:30.360> time<00:01:30.540> so<00:01:31.380> before<00:01:31.619> I<00:01:31.799> kind<00:01:31.920> of<00:01:32.040> step<00:01:32.100> into 00:01:32.210 --> 00:01:32.220 align:start position:0% the time so before I kind of step into 00:01:32.220 --> 00:01:33.649 align:start position:0% the time so before I kind of step into our<00:01:32.400> story<00:01:32.520> I<00:01:32.700> just<00:01:32.759> want<00:01:32.880> to<00:01:33.000> kind<00:01:33.119> of<00:01:33.180> do<00:01:33.479> a 00:01:33.649 --> 00:01:33.659 align:start position:0% our story I just want to kind of do a 00:01:33.659 --> 00:01:36.770 align:start position:0% our story I just want to kind of do a very<00:01:33.780> high<00:01:34.020> level<00:01:34.200> of<00:01:34.740> what<00:01:34.979> metrics<00:01:35.520> are<00:01:35.780> uh 00:01:36.770 --> 00:01:36.780 align:start position:0% very high level of what metrics are uh 00:01:36.780 --> 00:01:39.050 align:start position:0% very high level of what metrics are uh this<00:01:36.900> is<00:01:37.079> I'm<00:01:37.259> not<00:01:37.439> going<00:01:37.560> to<00:01:37.740> go<00:01:38.400> too<00:01:38.579> deep<00:01:38.759> if 00:01:39.050 --> 00:01:39.060 align:start position:0% this is I'm not going to go too deep if 00:01:39.060 --> 00:01:41.149 align:start position:0% this is I'm not going to go too deep if if<00:01:39.240> you<00:01:39.420> really<00:01:39.540> want<00:01:39.659> to<00:01:39.780> go<00:01:39.900> no<00:01:40.200> more<00:01:40.380> you<00:01:41.100> can 00:01:41.149 --> 00:01:41.159 align:start position:0% if you really want to go no more you can 00:01:41.159 --> 00:01:44.569 align:start position:0% if you really want to go no more you can like<00:01:41.280> read<00:01:41.460> the<00:01:41.640> docs<00:01:42.119> about<00:01:42.299> these<00:01:42.780> things<00:01:42.960> so 00:01:44.569 --> 00:01:44.579 align:start position:0% like read the docs about these things so 00:01:44.579 --> 00:01:46.429 align:start position:0% like read the docs about these things so um metrics<00:01:45.420> in<00:01:45.780> our<00:01:45.900> case<00:01:46.020> we're<00:01:46.200> using 00:01:46.429 --> 00:01:46.439 align:start position:0% um metrics in our case we're using 00:01:46.439 --> 00:01:47.590 align:start position:0% um metrics in our case we're using Prometheus 00:01:47.590 --> 00:01:47.600 align:start position:0% Prometheus 00:01:47.600 --> 00:01:50.210 align:start position:0% Prometheus Prometheus<00:01:48.600> basically<00:01:48.960> gives<00:01:49.200> you<00:01:49.380> the<00:01:49.740> the 00:01:50.210 --> 00:01:50.220 align:start position:0% Prometheus basically gives you the the 00:01:50.220 --> 00:01:53.749 align:start position:0% Prometheus basically gives you the the tools<00:01:50.640> to<00:01:51.439> build<00:01:52.439> time<00:01:52.740> series<00:01:52.920> data<00:01:53.460> for 00:01:53.749 --> 00:01:53.759 align:start position:0% tools to build time series data for 00:01:53.759 --> 00:01:55.609 align:start position:0% tools to build time series data for specific<00:01:54.479> elements<00:01:55.079> that<00:01:55.200> you<00:01:55.320> want<00:01:55.380> to<00:01:55.439> track 00:01:55.609 --> 00:01:55.619 align:start position:0% specific elements that you want to track 00:01:55.619 --> 00:01:57.770 align:start position:0% specific elements that you want to track in<00:01:55.740> your<00:01:55.920> code<00:01:56.159> the<00:01:57.119> kind<00:01:57.299> of<00:01:57.420> three<00:01:57.540> main 00:01:57.770 --> 00:01:57.780 align:start position:0% in your code the kind of three main 00:01:57.780 --> 00:01:59.270 align:start position:0% in your code the kind of three main things<00:01:57.960> that<00:01:58.079> you're<00:01:58.200> tracking<00:01:58.680> is<00:01:59.100> either 00:01:59.270 --> 00:01:59.280 align:start position:0% things that you're tracking is either 00:01:59.280 --> 00:02:01.670 align:start position:0% things that you're tracking is either counter<00:01:59.759> a<00:01:59.939> gauge<00:02:00.180> or<00:02:00.360> a<00:02:00.479> histogram<00:02:00.840> uh<00:02:01.439> we<00:02:01.560> can 00:02:01.670 --> 00:02:01.680 align:start position:0% counter a gauge or a histogram uh we can 00:02:01.680 --> 00:02:03.889 align:start position:0% counter a gauge or a histogram uh we can go<00:02:01.799> we'll<00:02:01.979> go<00:02:02.100> into<00:02:02.220> kind<00:02:02.520> of<00:02:03.119> some<00:02:03.600> examples 00:02:03.889 --> 00:02:03.899 align:start position:0% go we'll go into kind of some examples 00:02:03.899 --> 00:02:05.330 align:start position:0% go we'll go into kind of some examples of<00:02:04.079> all<00:02:04.200> of<00:02:04.320> these<00:02:04.500> things<00:02:04.619> but<00:02:04.979> basically 00:02:05.330 --> 00:02:05.340 align:start position:0% of all of these things but basically 00:02:05.340 --> 00:02:06.709 align:start position:0% of all of these things but basically what<00:02:05.520> you<00:02:05.640> want<00:02:05.759> to<00:02:05.880> know<00:02:05.939> about<00:02:06.060> metrics<00:02:06.600> are 00:02:06.709 --> 00:02:06.719 align:start position:0% what you want to know about metrics are 00:02:06.719 --> 00:02:08.449 align:start position:0% what you want to know about metrics are you<00:02:07.079> have<00:02:07.259> some<00:02:07.439> value<00:02:07.619> something<00:02:08.039> in<00:02:08.340> your 00:02:08.449 --> 00:02:08.459 align:start position:0% you have some value something in your 00:02:08.459 --> 00:02:10.490 align:start position:0% you have some value something in your code<00:02:08.640> that<00:02:08.819> you're<00:02:08.940> wanting<00:02:09.179> to<00:02:09.300> track<00:02:09.500> and 00:02:10.490 --> 00:02:10.500 align:start position:0% code that you're wanting to track and 00:02:10.500 --> 00:02:12.890 align:start position:0% code that you're wanting to track and you<00:02:10.619> want<00:02:10.800> and<00:02:11.400> Prometheus<00:02:12.060> will<00:02:12.239> kind<00:02:12.420> of 00:02:12.890 --> 00:02:12.900 align:start position:0% you want and Prometheus will kind of 00:02:12.900 --> 00:02:15.470 align:start position:0% you want and Prometheus will kind of query<00:02:13.560> the<00:02:13.920> query<00:02:14.340> your<00:02:14.580> software<00:02:15.000> over<00:02:15.239> time 00:02:15.470 --> 00:02:15.480 align:start position:0% query the query your software over time 00:02:15.480 --> 00:02:18.170 align:start position:0% query the query your software over time and<00:02:15.720> you<00:02:15.840> can<00:02:15.959> get<00:02:16.080> a<00:02:16.260> sense<00:02:16.440> over<00:02:17.280> time<00:02:17.580> of<00:02:17.940> of 00:02:18.170 --> 00:02:18.180 align:start position:0% and you can get a sense over time of of 00:02:18.180 --> 00:02:20.089 align:start position:0% and you can get a sense over time of of how<00:02:18.599> things<00:02:18.840> are<00:02:19.020> changing<00:02:19.379> how<00:02:19.680> things<00:02:19.800> are 00:02:20.089 --> 00:02:20.099 align:start position:0% how things are changing how things are 00:02:20.099 --> 00:02:21.710 align:start position:0% how things are changing how things are are<00:02:20.459> moving 00:02:21.710 --> 00:02:21.720 align:start position:0% are moving 00:02:21.720 --> 00:02:24.589 align:start position:0% are moving um<00:02:21.780> then<00:02:22.500> you<00:02:22.680> use<00:02:22.860> grafana<00:02:23.340> to<00:02:23.819> visualize 00:02:24.589 --> 00:02:24.599 align:start position:0% um then you use grafana to visualize 00:02:24.599 --> 00:02:27.710 align:start position:0% um then you use grafana to visualize these<00:02:25.260> metrics<00:02:25.680> so 00:02:27.710 --> 00:02:27.720 align:start position:0% these metrics so 00:02:27.720 --> 00:02:30.710 align:start position:0% these metrics so um<00:02:27.780> a<00:02:28.260> picture<00:02:28.379> is<00:02:28.700> like<00:02:29.700> a<00:02:29.760> thousand<00:02:30.000> words<00:02:30.180> so 00:02:30.710 --> 00:02:30.720 align:start position:0% um a picture is like a thousand words so 00:02:30.720 --> 00:02:32.930 align:start position:0% um a picture is like a thousand words so it's<00:02:31.280> basically<00:02:32.280> the<00:02:32.459> difference<00:02:32.580> between 00:02:32.930 --> 00:02:32.940 align:start position:0% it's basically the difference between 00:02:32.940 --> 00:02:34.970 align:start position:0% it's basically the difference between like<00:02:33.239> looking<00:02:33.840> at<00:02:34.020> a<00:02:34.140> bunch<00:02:34.260> of<00:02:34.379> logs<00:02:34.739> and<00:02:34.860> then 00:02:34.970 --> 00:02:34.980 align:start position:0% like looking at a bunch of logs and then 00:02:34.980 --> 00:02:36.710 align:start position:0% like looking at a bunch of logs and then looking<00:02:35.099> at<00:02:35.340> like<00:02:35.580> pretty<00:02:35.940> graphs<00:02:36.360> that<00:02:36.540> can 00:02:36.710 --> 00:02:36.720 align:start position:0% looking at like pretty graphs that can 00:02:36.720 --> 00:02:38.089 align:start position:0% looking at like pretty graphs that can show<00:02:36.840> you<00:02:37.020> give<00:02:37.500> you<00:02:37.620> a<00:02:37.800> little<00:02:37.860> bit<00:02:37.980> more 00:02:38.089 --> 00:02:38.099 align:start position:0% show you give you a little bit more 00:02:38.099 --> 00:02:40.369 align:start position:0% show you give you a little bit more insight<00:02:38.340> give<00:02:38.580> you<00:02:38.700> something<00:02:38.819> at<00:02:39.360> a<00:02:39.660> higher 00:02:40.369 --> 00:02:40.379 align:start position:0% insight give you something at a higher 00:02:40.379 --> 00:02:43.550 align:start position:0% insight give you something at a higher level<00:02:40.819> than<00:02:41.819> just<00:02:42.120> squinting 00:02:43.550 --> 00:02:43.560 align:start position:0% level than just squinting 00:02:43.560 --> 00:02:44.990 align:start position:0% level than just squinting so 00:02:44.990 --> 00:02:45.000 align:start position:0% so 00:02:45.000 --> 00:02:46.550 align:start position:0% so our<00:02:45.599> story 00:02:46.550 --> 00:02:46.560 align:start position:0% our story 00:02:46.560 --> 00:02:49.550 align:start position:0% our story um basically<00:02:47.340> we<00:02:47.700> we<00:02:47.760> started 00:02:49.550 --> 00:02:49.560 align:start position:0% um basically we we started 00:02:49.560 --> 00:02:54.290 align:start position:0% um basically we we started um<00:02:49.620> with<00:02:50.340> really<00:02:50.700> bad<00:02:51.440> uh<00:02:52.819> with<00:02:53.819> not<00:02:54.000> very<00:02:54.120> much 00:02:54.290 --> 00:02:54.300 align:start position:0% um with really bad uh with not very much 00:02:54.300 --> 00:02:56.330 align:start position:0% um with really bad uh with not very much infrastructure<00:02:54.959> not<00:02:55.680> very<00:02:55.860> much<00:02:56.040> a 00:02:56.330 --> 00:02:56.340 align:start position:0% infrastructure not very much a 00:02:56.340 --> 00:02:58.610 align:start position:0% infrastructure not very much a discipline<00:02:57.000> and<00:02:57.239> how<00:02:57.660> we 00:02:58.610 --> 00:02:58.620 align:start position:0% discipline and how we 00:02:58.620 --> 00:02:59.930 align:start position:0% discipline and how we tackled 00:02:59.930 --> 00:02:59.940 align:start position:0% tackled 00:02:59.940 --> 00:03:03.050 align:start position:0% tackled um<00:03:00.379> metrics<00:03:01.379> or<00:03:01.440> tackled<00:03:01.800> like<00:03:02.160> the<00:03:02.640> data<00:03:02.940> that 00:03:03.050 --> 00:03:03.060 align:start position:0% um metrics or tackled like the data that 00:03:03.060 --> 00:03:06.410 align:start position:0% um metrics or tackled like the data that we<00:03:03.239> we<00:03:03.420> track<00:03:03.800> and<00:03:04.800> eventually<00:03:05.340> we<00:03:05.700> kind<00:03:05.819> of 00:03:06.410 --> 00:03:06.420 align:start position:0% we we track and eventually we kind of 00:03:06.420 --> 00:03:08.690 align:start position:0% we we track and eventually we kind of like<00:03:06.959> over<00:03:07.560> time<00:03:07.860> and<00:03:08.040> like<00:03:08.220> through<00:03:08.340> a<00:03:08.519> lot<00:03:08.580> of 00:03:08.690 --> 00:03:08.700 align:start position:0% like over time and like through a lot of 00:03:08.700 --> 00:03:10.970 align:start position:0% like over time and like through a lot of pain<00:03:08.819> and<00:03:09.120> and<00:03:09.480> suffering<00:03:10.379> we<00:03:10.560> kind<00:03:10.920> of 00:03:10.970 --> 00:03:10.980 align:start position:0% pain and and suffering we kind of 00:03:10.980 --> 00:03:12.470 align:start position:0% pain and and suffering we kind of realized<00:03:11.340> that<00:03:11.580> metrics<00:03:11.940> are<00:03:12.060> the<00:03:12.120> way<00:03:12.300> to<00:03:12.360> go 00:03:12.470 --> 00:03:12.480 align:start position:0% realized that metrics are the way to go 00:03:12.480 --> 00:03:14.449 align:start position:0% realized that metrics are the way to go metrics<00:03:12.840> give<00:03:13.019> you<00:03:13.140> uh<00:03:13.379> insights<00:03:14.159> that<00:03:14.280> you'll 00:03:14.449 --> 00:03:14.459 align:start position:0% metrics give you uh insights that you'll 00:03:14.459 --> 00:03:16.970 align:start position:0% metrics give you uh insights that you'll never<00:03:14.940> get<00:03:15.180> just<00:03:15.540> by<00:03:15.720> looking<00:03:15.900> at<00:03:16.140> logs<00:03:16.500> so 00:03:16.970 --> 00:03:16.980 align:start position:0% never get just by looking at logs so 00:03:16.980 --> 00:03:18.830 align:start position:0% never get just by looking at logs so really<00:03:17.220> metrics<00:03:17.819> aggregate<00:03:18.480> a<00:03:18.599> bunch<00:03:18.720> of 00:03:18.830 --> 00:03:18.840 align:start position:0% really metrics aggregate a bunch of 00:03:18.840 --> 00:03:20.809 align:start position:0% really metrics aggregate a bunch of information<00:03:19.019> and<00:03:19.620> give<00:03:19.860> you 00:03:20.809 --> 00:03:20.819 align:start position:0% information and give you 00:03:20.819 --> 00:03:22.190 align:start position:0% information and give you um the<00:03:21.120> tools<00:03:21.360> you<00:03:21.480> need<00:03:21.599> to<00:03:21.720> make<00:03:21.840> better 00:03:22.190 --> 00:03:22.200 align:start position:0% um the tools you need to make better 00:03:22.200 --> 00:03:24.410 align:start position:0% um the tools you need to make better decisions<00:03:22.620> about<00:03:22.920> how<00:03:23.760> you<00:03:23.879> release<00:03:24.060> your 00:03:24.410 --> 00:03:24.420 align:start position:0% decisions about how you release your 00:03:24.420 --> 00:03:27.589 align:start position:0% decisions about how you release your product<00:03:24.739> what<00:03:25.739> things<00:03:25.920> to<00:03:26.159> focus<00:03:26.400> on<00:03:26.640> like 00:03:27.589 --> 00:03:27.599 align:start position:0% product what things to focus on like 00:03:27.599 --> 00:03:30.290 align:start position:0% product what things to focus on like what<00:03:27.780> are<00:03:28.019> priorities<00:03:28.640> and<00:03:29.640> they're<00:03:30.060> also 00:03:30.290 --> 00:03:30.300 align:start position:0% what are priorities and they're also 00:03:30.300 --> 00:03:32.809 align:start position:0% what are priorities and they're also just<00:03:30.420> really<00:03:30.540> pretty<00:03:30.780> so<00:03:31.200> they're<00:03:31.620> cool<00:03:31.860> so 00:03:32.809 --> 00:03:32.819 align:start position:0% just really pretty so they're cool so 00:03:32.819 --> 00:03:34.970 align:start position:0% just really pretty so they're cool so this<00:03:33.659> is<00:03:33.780> a<00:03:33.900> picture<00:03:34.019> from<00:03:34.379> the<00:03:34.440> early<00:03:34.560> days<00:03:34.800> of 00:03:34.970 --> 00:03:34.980 align:start position:0% this is a picture from the early days of 00:03:34.980 --> 00:03:37.670 align:start position:0% this is a picture from the early days of lodestar<00:03:35.519> from<00:03:36.480> the<00:03:36.659> interop<00:03:37.140> walk-in<00:03:37.500> in 00:03:37.670 --> 00:03:37.680 align:start position:0% lodestar from the interop walk-in in 00:03:37.680 --> 00:03:40.550 align:start position:0% lodestar from the interop walk-in in Muskoka<00:03:38.220> in<00:03:38.400> 2019<00:03:39.019> here's<00:03:40.019> our<00:03:40.080> message<00:03:40.260> yeah 00:03:40.550 --> 00:03:40.560 align:start position:0% Muskoka in 2019 here's our message yeah 00:03:40.560 --> 00:03:43.850 align:start position:0% Muskoka in 2019 here's our message yeah we're<00:03:40.799> stuck<00:03:41.099> syncing<00:03:41.459> and<00:03:41.640> some<00:03:41.760> P2P<00:03:42.180> issues 00:03:43.850 --> 00:03:43.860 align:start position:0% we're stuck syncing and some P2P issues 00:03:43.860 --> 00:03:45.890 align:start position:0% we're stuck syncing and some P2P issues uh<00:03:44.459> we<00:03:44.580> didn't<00:03:44.760> really<00:03:44.879> know<00:03:45.060> what<00:03:45.299> what<00:03:45.360> was 00:03:45.890 --> 00:03:45.900 align:start position:0% uh we didn't really know what what was 00:03:45.900 --> 00:03:47.869 align:start position:0% uh we didn't really know what what was going<00:03:46.080> on<00:03:46.260> like<00:03:46.680> why<00:03:47.280> things<00:03:47.459> were<00:03:47.580> breaking 00:03:47.869 --> 00:03:47.879 align:start position:0% going on like why things were breaking 00:03:47.879 --> 00:03:49.430 align:start position:0% going on like why things were breaking we<00:03:48.000> were<00:03:48.120> looking<00:03:48.239> at<00:03:48.420> logs<00:03:48.780> actually<00:03:49.080> on<00:03:49.319> the 00:03:49.430 --> 00:03:49.440 align:start position:0% we were looking at logs actually on the 00:03:49.440 --> 00:03:50.750 align:start position:0% we were looking at logs actually on the other<00:03:49.500> side<00:03:49.620> of<00:03:49.739> the<00:03:49.860> screen<00:03:49.980> here<00:03:50.220> you<00:03:50.400> you 00:03:50.750 --> 00:03:50.760 align:start position:0% other side of the screen here you you 00:03:50.760 --> 00:03:52.309 align:start position:0% other side of the screen here you you know<00:03:50.940> we<00:03:51.060> look<00:03:51.180> happy<00:03:51.360> but<00:03:51.599> it's<00:03:51.720> like<00:03:52.019> uh<00:03:52.140> oh 00:03:52.309 --> 00:03:52.319 align:start position:0% know we look happy but it's like uh oh 00:03:52.319 --> 00:03:54.589 align:start position:0% know we look happy but it's like uh oh things<00:03:52.620> are<00:03:52.799> breaking 00:03:54.589 --> 00:03:54.599 align:start position:0% things are breaking 00:03:54.599 --> 00:03:56.509 align:start position:0% things are breaking here's<00:03:55.140> some<00:03:55.260> uh 00:03:56.509 --> 00:03:56.519 align:start position:0% here's some uh 00:03:56.519 --> 00:03:59.390 align:start position:0% here's some uh um<00:03:56.640> some<00:03:57.299> funny<00:03:57.540> issues<00:03:58.019> that<00:03:58.260> we<00:03:58.440> uh<00:03:58.680> had<00:03:58.920> you 00:03:59.390 --> 00:03:59.400 align:start position:0% um some funny issues that we uh had you 00:03:59.400 --> 00:04:01.910 align:start position:0% um some funny issues that we uh had you can<00:03:59.519> see<00:03:59.640> priority<00:04:00.180> low<00:04:00.540> adding<00:04:01.319> Prometheus 00:04:01.910 --> 00:04:01.920 align:start position:0% can see priority low adding Prometheus 00:04:01.920 --> 00:04:03.649 align:start position:0% can see priority low adding Prometheus monitoring<00:04:02.400> we<00:04:02.819> didn't<00:04:02.940> even<00:04:03.120> know<00:04:03.299> we<00:04:03.480> were 00:04:03.649 --> 00:04:03.659 align:start position:0% monitoring we didn't even know we were 00:04:03.659 --> 00:04:05.089 align:start position:0% monitoring we didn't even know we were doing<00:04:03.840> uh 00:04:05.089 --> 00:04:05.099 align:start position:0% doing uh 00:04:05.099 --> 00:04:07.490 align:start position:0% doing uh um<00:04:05.220> yeah<00:04:05.340> out<00:04:05.580> of<00:04:05.700> memory<00:04:05.940> om<00:04:06.299> out<00:04:07.140> of<00:04:07.260> memory 00:04:07.490 --> 00:04:07.500 align:start position:0% um yeah out of memory om out of memory 00:04:07.500 --> 00:04:09.229 align:start position:0% um yeah out of memory om out of memory it<00:04:07.860> was<00:04:08.040> just<00:04:08.099> like<00:04:08.340> plaguing<00:04:08.760> a<00:04:08.940> lot<00:04:09.000> of<00:04:09.120> our 00:04:09.229 --> 00:04:09.239 align:start position:0% it was just like plaguing a lot of our 00:04:09.239 --> 00:04:13.429 align:start position:0% it was just like plaguing a lot of our development<00:04:09.720> throughout<00:04:10.500> 2019<00:04:11.060> into<00:04:12.060> 2020<00:04:12.659> uh 00:04:13.429 --> 00:04:13.439 align:start position:0% development throughout 2019 into 2020 uh 00:04:13.439 --> 00:04:15.470 align:start position:0% development throughout 2019 into 2020 uh lodestar<00:04:13.920> was<00:04:14.099> like<00:04:14.280> very<00:04:14.459> Proto<00:04:14.819> very<00:04:15.360> much 00:04:15.470 --> 00:04:15.480 align:start position:0% lodestar was like very Proto very much 00:04:15.480 --> 00:04:18.170 align:start position:0% lodestar was like very Proto very much like<00:04:15.659> a<00:04:15.840> prototype<00:04:16.440> and<00:04:16.799> we<00:04:17.340> ran<00:04:17.820> into<00:04:17.940> the 00:04:18.170 --> 00:04:18.180 align:start position:0% like a prototype and we ran into the 00:04:18.180 --> 00:04:20.090 align:start position:0% like a prototype and we ran into the issues<00:04:18.540> time<00:04:18.720> and<00:04:18.900> time<00:04:19.079> again<00:04:19.199> like<00:04:19.620> we<00:04:19.919> would 00:04:20.090 --> 00:04:20.100 align:start position:0% issues time and time again like we would 00:04:20.100 --> 00:04:21.590 align:start position:0% issues time and time again like we would have<00:04:20.280> the<00:04:20.459> same<00:04:20.639> thing<00:04:20.820> happen<00:04:21.120> multiple 00:04:21.590 --> 00:04:21.600 align:start position:0% have the same thing happen multiple 00:04:21.600 --> 00:04:23.090 align:start position:0% have the same thing happen multiple times<00:04:21.780> and<00:04:22.019> we<00:04:22.199> weren't<00:04:22.380> like<00:04:22.680> like 00:04:23.090 --> 00:04:23.100 align:start position:0% times and we weren't like like 00:04:23.100 --> 00:04:24.770 align:start position:0% times and we weren't like like regressions<00:04:23.820> that<00:04:24.060> were<00:04:24.180> happening<00:04:24.479> and<00:04:24.660> we 00:04:24.770 --> 00:04:24.780 align:start position:0% regressions that were happening and we 00:04:24.780 --> 00:04:25.969 align:start position:0% regressions that were happening and we weren't<00:04:24.900> really<00:04:25.020> like<00:04:25.199> learning<00:04:25.500> our<00:04:25.620> lesson 00:04:25.969 --> 00:04:25.979 align:start position:0% weren't really like learning our lesson 00:04:25.979 --> 00:04:29.689 align:start position:0% weren't really like learning our lesson kind<00:04:26.820> of<00:04:26.940> the<00:04:27.320> turning<00:04:28.320> point<00:04:28.560> was<00:04:28.919> you<00:04:29.639> know 00:04:29.689 --> 00:04:29.699 align:start position:0% kind of the turning point was you know 00:04:29.699 --> 00:04:31.790 align:start position:0% kind of the turning point was you know starting<00:04:29.940> to<00:04:30.300> take<00:04:30.540> some<00:04:30.780> time<00:04:30.960> to<00:04:31.259> to<00:04:31.440> Really 00:04:31.790 --> 00:04:31.800 align:start position:0% starting to take some time to to Really 00:04:31.800 --> 00:04:35.090 align:start position:0% starting to take some time to to Really uh<00:04:33.180> which<00:04:33.900> takes<00:04:34.320> the<00:04:34.500> time<00:04:34.620> to<00:04:34.800> like<00:04:34.919> build 00:04:35.090 --> 00:04:35.100 align:start position:0% uh which takes the time to like build 00:04:35.100 --> 00:04:36.469 align:start position:0% uh which takes the time to like build out<00:04:35.280> these<00:04:35.460> metrics<00:04:35.820> and<00:04:35.940> build<00:04:36.120> out<00:04:36.240> these 00:04:36.469 --> 00:04:36.479 align:start position:0% out these metrics and build out these 00:04:36.479 --> 00:04:39.409 align:start position:0% out these metrics and build out these dashboards<00:04:37.020> so<00:04:37.620> we<00:04:38.340> had<00:04:38.460> a<00:04:38.580> contributor<00:04:39.060> I 00:04:39.409 --> 00:04:39.419 align:start position:0% dashboards so we had a contributor I 00:04:39.419 --> 00:04:42.110 align:start position:0% dashboards so we had a contributor I think<00:04:39.540> I<00:04:39.840> can<00:04:40.020> save<00:04:40.800> a<00:04:41.040> new<00:04:41.160> world<00:04:41.340> maybe<00:04:41.759> with 00:04:42.110 --> 00:04:42.120 align:start position:0% think I can save a new world maybe with 00:04:42.120 --> 00:04:43.310 align:start position:0% think I can save a new world maybe with peace 00:04:43.310 --> 00:04:43.320 align:start position:0% peace 00:04:43.320 --> 00:04:46.189 align:start position:0% peace um<00:04:43.380> and<00:04:43.560> so<00:04:43.680> throughout<00:04:44.040> 2021<00:04:44.639> we<00:04:45.600> started 00:04:46.189 --> 00:04:46.199 align:start position:0% um and so throughout 2021 we started 00:04:46.199 --> 00:04:48.890 align:start position:0% um and so throughout 2021 we started going<00:04:47.040> really<00:04:47.400> hardcore<00:04:47.880> into<00:04:48.240> adding<00:04:48.720> a 00:04:48.890 --> 00:04:48.900 align:start position:0% going really hardcore into adding a 00:04:48.900 --> 00:04:51.530 align:start position:0% going really hardcore into adding a bunch<00:04:49.020> of<00:04:49.199> just<00:04:49.800> out<00:04:50.220> of<00:04:50.340> a<00:04:50.400> bunch<00:04:50.580> of<00:04:50.639> metrics 00:04:51.530 --> 00:04:51.540 align:start position:0% bunch of just out of a bunch of metrics 00:04:51.540 --> 00:04:53.689 align:start position:0% bunch of just out of a bunch of metrics um<00:04:52.020> and<00:04:52.500> taking<00:04:52.680> time<00:04:52.979> to<00:04:53.160> think<00:04:53.280> through<00:04:53.460> what 00:04:53.689 --> 00:04:53.699 align:start position:0% um and taking time to think through what 00:04:53.699 --> 00:04:56.270 align:start position:0% um and taking time to think through what questions<00:04:53.940> that<00:04:54.300> we<00:04:54.419> want<00:04:54.540> to<00:04:54.720> answer<00:04:55.280> what 00:04:56.270 --> 00:04:56.280 align:start position:0% questions that we want to answer what 00:04:56.280 --> 00:04:57.909 align:start position:0% questions that we want to answer what pieces<00:04:56.520> of<00:04:56.699> code<00:04:56.880> are<00:04:57.120> dark 00:04:57.909 --> 00:04:57.919 align:start position:0% pieces of code are dark 00:04:57.919 --> 00:05:00.230 align:start position:0% pieces of code are dark where<00:04:58.919> is 00:05:00.230 --> 00:05:00.240 align:start position:0% where is 00:05:00.240 --> 00:05:01.850 align:start position:0% where is are<00:05:00.780> you<00:05:01.020> know<00:05:01.080> where<00:05:01.199> are<00:05:01.320> we<00:05:01.440> spending<00:05:01.680> time 00:05:01.850 --> 00:05:01.860 align:start position:0% are you know where are we spending time 00:05:01.860 --> 00:05:04.550 align:start position:0% are you know where are we spending time in<00:05:02.160> the<00:05:02.280> in<00:05:02.520> in<00:05:02.699> our<00:05:02.940> software<00:05:03.479> where<00:05:04.080> are<00:05:04.380> we 00:05:04.550 --> 00:05:04.560 align:start position:0% in the in in our software where are we 00:05:04.560 --> 00:05:06.469 align:start position:0% in the in in our software where are we where<00:05:05.280> is<00:05:05.400> a<00:05:05.460> bunch<00:05:05.580> of<00:05:05.699> memory<00:05:06.000> being<00:05:06.240> spent 00:05:06.469 --> 00:05:06.479 align:start position:0% where is a bunch of memory being spent 00:05:06.479 --> 00:05:09.710 align:start position:0% where is a bunch of memory being spent we<00:05:06.840> really<00:05:07.380> kind<00:05:07.860> of<00:05:07.979> uncovering<00:05:08.639> all<00:05:09.360> of<00:05:09.479> the 00:05:09.710 --> 00:05:09.720 align:start position:0% we really kind of uncovering all of the 00:05:09.720 --> 00:05:12.409 align:start position:0% we really kind of uncovering all of the complexity<00:05:10.199> of<00:05:10.620> this<00:05:10.860> running<00:05:11.220> software<00:05:11.759> I 00:05:12.409 --> 00:05:12.419 align:start position:0% complexity of this running software I 00:05:12.419 --> 00:05:13.850 align:start position:0% complexity of this running software I mean<00:05:12.600> blockchains<00:05:13.199> are<00:05:13.320> really<00:05:13.500> complicated 00:05:13.850 --> 00:05:13.860 align:start position:0% mean blockchains are really complicated 00:05:13.860 --> 00:05:15.350 align:start position:0% mean blockchains are really complicated software<00:05:14.280> there's<00:05:14.520> a<00:05:14.699> whole<00:05:14.759> bunch<00:05:14.940> going<00:05:15.120> on 00:05:15.350 --> 00:05:15.360 align:start position:0% software there's a whole bunch going on 00:05:15.360 --> 00:05:16.850 align:start position:0% software there's a whole bunch going on and 00:05:16.850 --> 00:05:16.860 align:start position:0% and 00:05:16.860 --> 00:05:18.050 align:start position:0% and um unless<00:05:17.280> you're<00:05:17.400> really<00:05:17.639> tracking 00:05:18.050 --> 00:05:18.060 align:start position:0% um unless you're really tracking 00:05:18.060 --> 00:05:21.230 align:start position:0% um unless you're really tracking tracking<00:05:18.780> key<00:05:19.080> pieces<00:05:19.500> you're<00:05:20.280> you're<00:05:20.940> kind 00:05:21.230 --> 00:05:21.240 align:start position:0% tracking key pieces you're you're kind 00:05:21.240 --> 00:05:23.810 align:start position:0% tracking key pieces you're you're kind of<00:05:21.360> going<00:05:21.660> blind<00:05:22.100> and<00:05:23.100> now<00:05:23.340> we're<00:05:23.520> finally 00:05:23.810 --> 00:05:23.820 align:start position:0% of going blind and now we're finally 00:05:23.820 --> 00:05:25.670 align:start position:0% of going blind and now we're finally kind<00:05:23.940> of<00:05:24.060> at<00:05:24.180> a<00:05:24.360> point<00:05:24.479> where 00:05:25.670 --> 00:05:25.680 align:start position:0% kind of at a point where 00:05:25.680 --> 00:05:26.350 align:start position:0% kind of at a point where um 00:05:26.350 --> 00:05:26.360 align:start position:0% um 00:05:26.360 --> 00:05:29.510 align:start position:0% um we<00:05:27.360> for<00:05:28.139> any<00:05:28.440> new<00:05:28.620> feature<00:05:28.919> that<00:05:29.100> we're<00:05:29.220> adding 00:05:29.510 --> 00:05:29.520 align:start position:0% we for any new feature that we're adding 00:05:29.520 --> 00:05:32.870 align:start position:0% we for any new feature that we're adding we<00:05:30.180> ask<00:05:30.419> for<00:05:30.720> metrics<00:05:31.560> so 00:05:32.870 --> 00:05:32.880 align:start position:0% we ask for metrics so 00:05:32.880 --> 00:05:34.430 align:start position:0% we ask for metrics so it'd<00:05:33.240> be<00:05:33.360> very<00:05:33.479> important<00:05:33.780> to<00:05:33.960> track<00:05:34.139> the 00:05:34.430 --> 00:05:34.440 align:start position:0% it'd be very important to track the 00:05:34.440 --> 00:05:37.550 align:start position:0% it'd be very important to track the retry<00:05:34.740> Behavior<00:05:35.160> meaning<00:05:35.460> add<00:05:35.880> metrics<00:05:36.419> for 00:05:37.550 --> 00:05:37.560 align:start position:0% retry Behavior meaning add metrics for 00:05:37.560 --> 00:05:39.050 align:start position:0% retry Behavior meaning add metrics for so<00:05:37.800> that<00:05:37.979> brings<00:05:38.160> me<00:05:38.280> to<00:05:38.400> metrics<00:05:38.759> driven 00:05:39.050 --> 00:05:39.060 align:start position:0% so that brings me to metrics driven 00:05:39.060 --> 00:05:40.550 align:start position:0% so that brings me to metrics driven development 00:05:40.550 --> 00:05:40.560 align:start position:0% development 00:05:40.560 --> 00:05:42.409 align:start position:0% development um<00:05:40.620> really<00:05:40.800> the<00:05:41.039> the<00:05:41.160> kind<00:05:41.580> of<00:05:41.699> the<00:05:41.820> key<00:05:42.000> the 00:05:42.409 --> 00:05:42.419 align:start position:0% um really the the kind of the key the 00:05:42.419 --> 00:05:43.850 align:start position:0% um really the the kind of the key the key<00:05:42.660> thing<00:05:42.840> that<00:05:43.020> we've<00:05:43.199> learned<00:05:43.440> is<00:05:43.680> that 00:05:43.850 --> 00:05:43.860 align:start position:0% key thing that we've learned is that 00:05:43.860 --> 00:05:46.010 align:start position:0% key thing that we've learned is that every<00:05:44.100> large<00:05:44.340> feature<00:05:44.759> should<00:05:45.360> be<00:05:45.479> documented 00:05:46.010 --> 00:05:46.020 align:start position:0% every large feature should be documented 00:05:46.020 --> 00:05:49.249 align:start position:0% every large feature should be documented with<00:05:46.259> metrics<00:05:46.740> so<00:05:47.280> if<00:05:47.639> you<00:05:47.880> are<00:05:48.120> adding<00:05:48.539> a<00:05:49.020> case 00:05:49.249 --> 00:05:49.259 align:start position:0% with metrics so if you are adding a case 00:05:49.259 --> 00:05:51.110 align:start position:0% with metrics so if you are adding a case of<00:05:49.380> a<00:05:49.440> retry<00:05:49.800> mechanism<00:05:50.220> you're<00:05:50.520> adding<00:05:50.820> some 00:05:51.110 --> 00:05:51.120 align:start position:0% of a retry mechanism you're adding some 00:05:51.120 --> 00:05:52.909 align:start position:0% of a retry mechanism you're adding some new<00:05:51.720> feature<00:05:52.080> you<00:05:52.320> need<00:05:52.440> to<00:05:52.500> make<00:05:52.620> sure<00:05:52.740> that 00:05:52.909 --> 00:05:52.919 align:start position:0% new feature you need to make sure that 00:05:52.919 --> 00:05:54.770 align:start position:0% new feature you need to make sure that it's<00:05:53.039> actually<00:05:53.340> working<00:05:53.639> and<00:05:54.360> the<00:05:54.479> only<00:05:54.600> way 00:05:54.770 --> 00:05:54.780 align:start position:0% it's actually working and the only way 00:05:54.780 --> 00:05:55.909 align:start position:0% it's actually working and the only way that<00:05:54.960> you<00:05:55.139> can 00:05:55.909 --> 00:05:55.919 align:start position:0% that you can 00:05:55.919 --> 00:05:58.129 align:start position:0% that you can or<00:05:56.759> maybe<00:05:57.000> not<00:05:57.240> the<00:05:57.360> only<00:05:57.479> way<00:05:57.600> but<00:05:57.840> a<00:05:57.960> really 00:05:58.129 --> 00:05:58.139 align:start position:0% or maybe not the only way but a really 00:05:58.139 --> 00:05:59.629 align:start position:0% or maybe not the only way but a really the<00:05:58.380> really<00:05:58.560> great<00:05:58.740> way<00:05:58.919> of<00:05:59.039> doing<00:05:59.220> that<00:05:59.340> is 00:05:59.629 --> 00:05:59.639 align:start position:0% the really great way of doing that is 00:05:59.639 --> 00:06:01.790 align:start position:0% the really great way of doing that is being<00:06:00.000> able<00:06:00.180> to<00:06:00.300> show<00:06:00.419> that<00:06:00.600> visually<00:06:00.900> being 00:06:01.790 --> 00:06:01.800 align:start position:0% being able to show that visually being 00:06:01.800 --> 00:06:04.430 align:start position:0% being able to show that visually being able<00:06:02.039> to<00:06:02.160> show<00:06:02.340> okay<00:06:02.900> our<00:06:03.900> cache<00:06:04.199> is<00:06:04.259> now 00:06:04.430 --> 00:06:04.440 align:start position:0% able to show okay our cache is now 00:06:04.440 --> 00:06:05.689 align:start position:0% able to show okay our cache is now bounded<00:06:04.800> let's<00:06:04.979> see<00:06:05.100> let's<00:06:05.280> see<00:06:05.400> that<00:06:05.520> in<00:06:05.580> the 00:06:05.689 --> 00:06:05.699 align:start position:0% bounded let's see let's see that in the 00:06:05.699 --> 00:06:07.310 align:start position:0% bounded let's see let's see that in the metrics<00:06:06.000> let's<00:06:06.120> see<00:06:06.300> how<00:06:06.419> it's<00:06:06.600> How<00:06:07.139> that 00:06:07.310 --> 00:06:07.320 align:start position:0% metrics let's see how it's How that 00:06:07.320 --> 00:06:09.770 align:start position:0% metrics let's see how it's How that cache<00:06:07.919> is<00:06:07.979> growing<00:06:08.280> over<00:06:08.400> time<00:06:08.639> or<00:06:09.300> let's<00:06:09.539> see 00:06:09.770 --> 00:06:09.780 align:start position:0% cache is growing over time or let's see 00:06:09.780 --> 00:06:11.629 align:start position:0% cache is growing over time or let's see how<00:06:09.960> many<00:06:10.080> times<00:06:10.259> the<00:06:10.740> retry<00:06:11.039> mechanism<00:06:11.460> is 00:06:11.629 --> 00:06:11.639 align:start position:0% how many times the retry mechanism is 00:06:11.639 --> 00:06:13.550 align:start position:0% how many times the retry mechanism is actually<00:06:11.880> being<00:06:12.120> activated 00:06:13.550 --> 00:06:13.560 align:start position:0% actually being activated 00:06:13.560 --> 00:06:14.270 align:start position:0% actually being activated um 00:06:14.270 --> 00:06:14.280 align:start position:0% um 00:06:14.280 --> 00:06:15.650 align:start position:0% um those<00:06:14.820> are<00:06:14.940> the<00:06:15.060> sorts<00:06:15.240> of<00:06:15.300> things<00:06:15.419> that<00:06:15.539> you 00:06:15.650 --> 00:06:15.660 align:start position:0% those are the sorts of things that you 00:06:15.660 --> 00:06:17.510 align:start position:0% those are the sorts of things that you really<00:06:15.780> only<00:06:15.960> look<00:06:16.259> confined<00:06:16.979> by<00:06:17.100> through 00:06:17.510 --> 00:06:17.520 align:start position:0% really only look confined by through 00:06:17.520 --> 00:06:19.670 align:start position:0% really only look confined by through through<00:06:17.880> these<00:06:18.300> graphical<00:06:19.020> methods<00:06:19.500> through 00:06:19.670 --> 00:06:19.680 align:start position:0% through these graphical methods through 00:06:19.680 --> 00:06:21.230 align:start position:0% through these graphical methods through these<00:06:20.039> metrics 00:06:21.230 --> 00:06:21.240 align:start position:0% these metrics 00:06:21.240 --> 00:06:22.249 align:start position:0% these metrics um you're<00:06:21.539> going<00:06:21.720> to<00:06:21.780> have<00:06:21.900> a<00:06:22.020> really<00:06:22.080> hard 00:06:22.249 --> 00:06:22.259 align:start position:0% um you're going to have a really hard 00:06:22.259 --> 00:06:24.590 align:start position:0% um you're going to have a really hard time<00:06:22.560> looking<00:06:22.979> at<00:06:23.160> the<00:06:23.280> logs<00:06:23.639> for<00:06:23.759> that<00:06:23.880> the 00:06:24.590 --> 00:06:24.600 align:start position:0% time looking at the logs for that the 00:06:24.600 --> 00:06:25.730 align:start position:0% time looking at the logs for that the other<00:06:24.780> great<00:06:25.020> the<00:06:25.259> other<00:06:25.380> thing<00:06:25.560> about 00:06:25.730 --> 00:06:25.740 align:start position:0% other great the other thing about 00:06:25.740 --> 00:06:27.350 align:start position:0% other great the other thing about metrics<00:06:26.280> driven<00:06:26.460> development<00:06:26.759> is<00:06:27.000> that<00:06:27.120> any 00:06:27.350 --> 00:06:27.360 align:start position:0% metrics driven development is that any 00:06:27.360 --> 00:06:29.629 align:start position:0% metrics driven development is that any deployed<00:06:27.900> software<00:06:28.259> that<00:06:28.440> you<00:06:28.560> use<00:06:28.740> you<00:06:29.520> need 00:06:29.629 --> 00:06:29.639 align:start position:0% deployed software that you use you need 00:06:29.639 --> 00:06:31.790 align:start position:0% deployed software that you use you need to<00:06:29.759> be<00:06:29.819> monitoring<00:06:30.360> it<00:06:30.539> very<00:06:30.780> carefully<00:06:31.199> for 00:06:31.790 --> 00:06:31.800 align:start position:0% to be monitoring it very carefully for 00:06:31.800 --> 00:06:35.510 align:start position:0% to be monitoring it very carefully for regressions<00:06:32.400> so<00:06:33.199> look<00:06:34.199> here<00:06:34.380> this<00:06:34.800> is 00:06:35.510 --> 00:06:35.520 align:start position:0% regressions so look here this is 00:06:35.520 --> 00:06:39.890 align:start position:0% regressions so look here this is the<00:06:36.300> process<00:06:36.780> Heap<00:06:37.400> increasing<00:06:38.400> over<00:06:38.639> time 00:06:39.890 --> 00:06:39.900 align:start position:0% the process Heap increasing over time 00:06:39.900 --> 00:06:41.870 align:start position:0% the process Heap increasing over time this<00:06:40.440> is<00:06:40.560> a<00:06:40.680> comment<00:06:40.919> from<00:06:41.039> Lion<00:06:41.220> for<00:06:41.639> the<00:06:41.759> last 00:06:41.870 --> 00:06:41.880 align:start position:0% this is a comment from Lion for the last 00:06:41.880 --> 00:06:44.390 align:start position:0% this is a comment from Lion for the last two<00:06:42.060> days<00:06:42.240> the<00:06:42.840> leak<00:06:43.139> is<00:06:43.199> clearly<00:06:43.500> visible<00:06:43.860> an 00:06:44.390 --> 00:06:44.400 align:start position:0% two days the leak is clearly visible an 00:06:44.400 --> 00:06:46.490 align:start position:0% two days the leak is clearly visible an impact<00:06:44.639> of<00:06:44.819> 75<00:06:45.180> megabytes<00:06:45.660> a<00:06:45.840> day<00:06:45.960> yeah<00:06:46.319> so 00:06:46.490 --> 00:06:46.500 align:start position:0% impact of 75 megabytes a day yeah so 00:06:46.500 --> 00:06:47.570 align:start position:0% impact of 75 megabytes a day yeah so that's<00:06:46.620> going<00:06:46.740> to<00:06:46.800> be<00:06:46.919> a<00:06:47.039> problem<00:06:47.160> if<00:06:47.460> your 00:06:47.570 --> 00:06:47.580 align:start position:0% that's going to be a problem if your 00:06:47.580 --> 00:06:49.969 align:start position:0% that's going to be a problem if your blockchain<00:06:47.940> is<00:06:48.660> if<00:06:49.139> it's<00:06:49.259> just<00:06:49.440> increasing<00:06:49.800> at 00:06:49.969 --> 00:06:49.979 align:start position:0% blockchain is if it's just increasing at 00:06:49.979 --> 00:06:52.010 align:start position:0% blockchain is if it's just increasing at 75<00:06:50.280> megabytes<00:06:50.759> a<00:06:50.880> day<00:06:51.000> continuously<00:06:51.900> you're 00:06:52.010 --> 00:06:52.020 align:start position:0% 75 megabytes a day continuously you're 00:06:52.020 --> 00:06:53.450 align:start position:0% 75 megabytes a day continuously you're going<00:06:52.199> to<00:06:52.199> run<00:06:52.319> out<00:06:52.440> of<00:06:52.560> memory 00:06:53.450 --> 00:06:53.460 align:start position:0% going to run out of memory 00:06:53.460 --> 00:06:55.249 align:start position:0% going to run out of memory and<00:06:53.880> these<00:06:54.000> are<00:06:54.060> this<00:06:54.240> we<00:06:54.600> were<00:06:54.780> able<00:06:54.900> to<00:06:55.020> catch 00:06:55.249 --> 00:06:55.259 align:start position:0% and these are this we were able to catch 00:06:55.259 --> 00:06:57.469 align:start position:0% and these are this we were able to catch this<00:06:55.440> this<00:06:55.620> issue<00:06:55.979> before<00:06:56.160> we<00:06:56.460> actually 00:06:57.469 --> 00:06:57.479 align:start position:0% this this issue before we actually 00:06:57.479 --> 00:07:00.290 align:start position:0% this this issue before we actually before<00:06:58.080> we<00:06:58.319> actually<00:06:58.500> cut<00:06:58.680> a<00:06:58.860> release<00:06:58.979> so 00:07:00.290 --> 00:07:00.300 align:start position:0% before we actually cut a release so 00:07:00.300 --> 00:07:01.189 align:start position:0% before we actually cut a release so um 00:07:01.189 --> 00:07:01.199 align:start position:0% um 00:07:01.199 --> 00:07:03.409 align:start position:0% um it<00:07:01.560> makes<00:07:01.680> your<00:07:01.860> release<00:07:02.039> process<00:07:02.460> a<00:07:02.759> lot<00:07:02.940> at 00:07:03.409 --> 00:07:03.419 align:start position:0% it makes your release process a lot at 00:07:03.419 --> 00:07:05.629 align:start position:0% it makes your release process a lot at least<00:07:03.479> release<00:07:03.780> process<00:07:04.319> is<00:07:04.500> a<00:07:04.680> lot<00:07:04.800> better 00:07:05.629 --> 00:07:05.639 align:start position:0% least release process is a lot better 00:07:05.639 --> 00:07:07.670 align:start position:0% least release process is a lot better here's<00:07:06.060> another<00:07:06.120> here's<00:07:06.660> another<00:07:06.780> example<00:07:07.500> of 00:07:07.670 --> 00:07:07.680 align:start position:0% here's another here's another example of 00:07:07.680 --> 00:07:10.850 align:start position:0% here's another here's another example of a<00:07:07.740> regression<00:07:08.220> uh<00:07:09.180> we<00:07:09.479> deployed<00:07:10.139> deployed<00:07:10.740> a 00:07:10.850 --> 00:07:10.860 align:start position:0% a regression uh we deployed deployed a 00:07:10.860 --> 00:07:12.170 align:start position:0% a regression uh we deployed deployed a version<00:07:10.979> of<00:07:11.160> our<00:07:11.280> software<00:07:11.580> and<00:07:11.759> we<00:07:11.940> saw<00:07:12.000> the 00:07:12.170 --> 00:07:12.180 align:start position:0% version of our software and we saw the 00:07:12.180 --> 00:07:14.570 align:start position:0% version of our software and we saw the cache<00:07:12.539> size<00:07:12.660> grow<00:07:13.080> it's<00:07:13.500> like<00:07:13.620> oh<00:07:13.800> is<00:07:14.280> that<00:07:14.400> is 00:07:14.570 --> 00:07:14.580 align:start position:0% cache size grow it's like oh is that is 00:07:14.580 --> 00:07:16.370 align:start position:0% cache size grow it's like oh is that is that<00:07:14.699> is<00:07:14.880> that<00:07:15.000> a<00:07:15.120> bug<00:07:15.180> is<00:07:15.419> that<00:07:15.539> a 00:07:16.370 --> 00:07:16.380 align:start position:0% that is that a bug is that a 00:07:16.380 --> 00:07:18.950 align:start position:0% that is that a bug is that a is<00:07:16.919> is<00:07:17.039> it<00:07:17.580> okay<00:07:17.759> well<00:07:18.360> we<00:07:18.539> only<00:07:18.720> actually 00:07:18.950 --> 00:07:18.960 align:start position:0% is is it okay well we only actually 00:07:18.960 --> 00:07:20.570 align:start position:0% is is it okay well we only actually caught<00:07:19.199> it<00:07:19.380> because<00:07:19.560> we're<00:07:19.860> tracking<00:07:20.400> these 00:07:20.570 --> 00:07:20.580 align:start position:0% caught it because we're tracking these 00:07:20.580 --> 00:07:21.409 align:start position:0% caught it because we're tracking these things 00:07:21.409 --> 00:07:21.419 align:start position:0% things 00:07:21.419 --> 00:07:23.270 align:start position:0% things uh<00:07:22.020> you<00:07:22.259> probably<00:07:22.800> aren't<00:07:22.919> even<00:07:23.099> going<00:07:23.220> to 00:07:23.270 --> 00:07:23.280 align:start position:0% uh you probably aren't even going to 00:07:23.280 --> 00:07:24.650 align:start position:0% uh you probably aren't even going to have<00:07:23.400> a<00:07:23.520> log<00:07:23.639> that's<00:07:23.940> going<00:07:24.060> to<00:07:24.120> show<00:07:24.300> you<00:07:24.419> your 00:07:24.650 --> 00:07:24.660 align:start position:0% have a log that's going to show you your 00:07:24.660 --> 00:07:26.990 align:start position:0% have a log that's going to show you your cash<00:07:24.960> sizes<00:07:25.440> so<00:07:25.680> unless<00:07:26.280> you<00:07:26.460> you<00:07:26.759> actually 00:07:26.990 --> 00:07:27.000 align:start position:0% cash sizes so unless you you actually 00:07:27.000 --> 00:07:29.210 align:start position:0% cash sizes so unless you you actually measure<00:07:27.419> this<00:07:28.139> you're<00:07:28.620> going<00:07:28.740> to<00:07:28.800> be<00:07:28.919> blind 00:07:29.210 --> 00:07:29.220 align:start position:0% measure this you're going to be blind 00:07:29.220 --> 00:07:30.790 align:start position:0% measure this you're going to be blind the<00:07:29.819> other<00:07:29.880> great<00:07:30.060> thing<00:07:30.180> about<00:07:30.300> the<00:07:30.539> about 00:07:30.790 --> 00:07:30.800 align:start position:0% the other great thing about the about 00:07:30.800 --> 00:07:33.350 align:start position:0% the other great thing about the about dashboards<00:07:31.800> and<00:07:31.919> about<00:07:32.039> metrics<00:07:32.580> is<00:07:32.699> that<00:07:32.940> it 00:07:33.350 --> 00:07:33.360 align:start position:0% dashboards and about metrics is that it 00:07:33.360 --> 00:07:34.790 align:start position:0% dashboards and about metrics is that it lets<00:07:33.599> you<00:07:33.660> correlate<00:07:34.139> different<00:07:34.440> different 00:07:34.790 --> 00:07:34.800 align:start position:0% lets you correlate different different 00:07:34.800 --> 00:07:36.950 align:start position:0% lets you correlate different different pieces<00:07:35.280> of<00:07:35.460> the<00:07:35.639> code<00:07:35.819> to<00:07:36.120> find<00:07:36.240> out<00:07:36.419> where<00:07:36.780> the 00:07:36.950 --> 00:07:36.960 align:start position:0% pieces of the code to find out where the 00:07:36.960 --> 00:07:38.930 align:start position:0% pieces of the code to find out where the problem<00:07:37.139> is<00:07:37.500> so<00:07:38.039> here<00:07:38.220> we<00:07:38.340> have<00:07:38.460> different<00:07:38.639> you 00:07:38.930 --> 00:07:38.940 align:start position:0% problem is so here we have different you 00:07:38.940 --> 00:07:40.010 align:start position:0% problem is so here we have different you can<00:07:39.060> you<00:07:39.240> can<00:07:39.419> see<00:07:39.539> a<00:07:39.660> bunch<00:07:39.780> of<00:07:39.900> different 00:07:40.010 --> 00:07:40.020 align:start position:0% can you can see a bunch of different 00:07:40.020 --> 00:07:41.990 align:start position:0% can you can see a bunch of different graphs<00:07:40.500> kind<00:07:40.860> of<00:07:40.919> all<00:07:41.039> at<00:07:41.220> the<00:07:41.340> same<00:07:41.460> time<00:07:41.699> and 00:07:41.990 --> 00:07:42.000 align:start position:0% graphs kind of all at the same time and 00:07:42.000 --> 00:07:43.129 align:start position:0% graphs kind of all at the same time and see 00:07:43.129 --> 00:07:43.139 align:start position:0% see 00:07:43.139 --> 00:07:45.830 align:start position:0% see um<00:07:43.620> okay<00:07:44.099> something<00:07:44.400> is<00:07:44.639> affecting<00:07:45.060> the<00:07:45.360> event 00:07:45.830 --> 00:07:45.840 align:start position:0% um okay something is affecting the event 00:07:45.840 --> 00:07:47.570 align:start position:0% um okay something is affecting the event the<00:07:46.139> event<00:07:46.319> Loop<00:07:46.620> lag 00:07:47.570 --> 00:07:47.580 align:start position:0% the event Loop lag 00:07:47.580 --> 00:07:49.670 align:start position:0% the event Loop lag I<00:07:48.060> can<00:07:48.180> see<00:07:48.300> at<00:07:48.539> the<00:07:48.720> same<00:07:48.840> time<00:07:49.020> the<00:07:49.380> active 00:07:49.670 --> 00:07:49.680 align:start position:0% I can see at the same time the active 00:07:49.680 --> 00:07:52.010 align:start position:0% I can see at the same time the active number<00:07:49.860> of<00:07:50.039> handles<00:07:50.340> is<00:07:50.580> growing<00:07:50.940> the<00:07:51.660> number 00:07:52.010 --> 00:07:52.020 align:start position:0% number of handles is growing the number 00:07:52.020 --> 00:07:53.870 align:start position:0% number of handles is growing the number of<00:07:52.199> requests<00:07:52.620> is<00:07:52.740> growing<00:07:53.039> so<00:07:53.280> you<00:07:53.520> can<00:07:53.639> kind 00:07:53.870 --> 00:07:53.880 align:start position:0% of requests is growing so you can kind 00:07:53.880 --> 00:07:54.650 align:start position:0% of requests is growing so you can kind of 00:07:54.650 --> 00:07:54.660 align:start position:0% of 00:07:54.660 --> 00:07:55.189 align:start position:0% of um 00:07:55.189 --> 00:07:55.199 align:start position:0% um 00:07:55.199 --> 00:07:57.830 align:start position:0% um you<00:07:55.680> know<00:07:55.800> kind<00:07:55.979> of<00:07:56.099> solve<00:07:56.340> the<00:07:56.400> mystery<00:07:56.819> in<00:07:57.479> a 00:07:57.830 --> 00:07:57.840 align:start position:0% you know kind of solve the mystery in a 00:07:57.840 --> 00:07:59.629 align:start position:0% you know kind of solve the mystery in a in<00:07:58.139> a<00:07:58.380> in<00:07:58.800> a<00:07:58.979> way<00:07:59.099> that<00:07:59.220> you're<00:07:59.280> not<00:07:59.400> going<00:07:59.520> to 00:07:59.629 --> 00:07:59.639 align:start position:0% in a in a way that you're not going to 00:07:59.639 --> 00:08:01.730 align:start position:0% in a in a way that you're not going to be<00:07:59.699> able<00:07:59.759> to<00:07:59.940> do<00:08:00.060> in<00:08:00.180> the<00:08:00.300> logs<00:08:00.900> another<00:08:01.440> great 00:08:01.730 --> 00:08:01.740 align:start position:0% be able to do in the logs another great 00:08:01.740 --> 00:08:02.870 align:start position:0% be able to do in the logs another great strategy 00:08:02.870 --> 00:08:02.880 align:start position:0% strategy 00:08:02.880 --> 00:08:04.909 align:start position:0% strategy uh<00:08:03.720> is 00:08:04.909 --> 00:08:04.919 align:start position:0% uh is 00:08:04.919 --> 00:08:05.749 align:start position:0% uh is um 00:08:05.749 --> 00:08:05.759 align:start position:0% um 00:08:05.759 --> 00:08:07.309 align:start position:0% um is<00:08:06.180> using<00:08:06.479> different<00:08:06.660> versions<00:08:06.960> if<00:08:07.139> you're 00:08:07.309 --> 00:08:07.319 align:start position:0% is using different versions if you're 00:08:07.319 --> 00:08:08.990 align:start position:0% is using different versions if you're running<00:08:07.500> software<00:08:07.860> so<00:08:08.099> running 00:08:08.990 --> 00:08:09.000 align:start position:0% running software so running 00:08:09.000 --> 00:08:10.430 align:start position:0% running software so running um<00:08:09.060> like<00:08:09.419> a<00:08:09.599> staging<00:08:09.900> version<00:08:10.139> of<00:08:10.319> your 00:08:10.430 --> 00:08:10.440 align:start position:0% um like a staging version of your 00:08:10.440 --> 00:08:12.290 align:start position:0% um like a staging version of your software<00:08:10.800> against<00:08:11.220> a<00:08:11.460> production<00:08:11.759> version<00:08:12.000> or 00:08:12.290 --> 00:08:12.300 align:start position:0% software against a production version or 00:08:12.300 --> 00:08:14.210 align:start position:0% software against a production version or against<00:08:12.479> several<00:08:13.139> recent<00:08:13.560> versions<00:08:13.919> and 00:08:14.210 --> 00:08:14.220 align:start position:0% against several recent versions and 00:08:14.220 --> 00:08:16.370 align:start position:0% against several recent versions and being<00:08:14.520> able<00:08:14.699> to<00:08:14.759> overlay<00:08:15.120> data<00:08:15.539> on<00:08:15.720> top<00:08:15.840> of<00:08:16.080> on 00:08:16.370 --> 00:08:16.380 align:start position:0% being able to overlay data on top of on 00:08:16.380 --> 00:08:18.589 align:start position:0% being able to overlay data on top of on top<00:08:16.800> of<00:08:16.919> one<00:08:17.099> another<00:08:17.220> and<00:08:17.880> being<00:08:18.060> able<00:08:18.300> to<00:08:18.419> see 00:08:18.589 --> 00:08:18.599 align:start position:0% top of one another and being able to see 00:08:18.599 --> 00:08:21.110 align:start position:0% top of one another and being able to see okay<00:08:19.259> well<00:08:19.680> it<00:08:20.220> looks<00:08:20.400> like<00:08:20.460> our<00:08:20.639> beta<00:08:21.000> version 00:08:21.110 --> 00:08:21.120 align:start position:0% okay well it looks like our beta version 00:08:21.120 --> 00:08:22.430 align:start position:0% okay well it looks like our beta version is 00:08:22.430 --> 00:08:22.440 align:start position:0% is 00:08:22.440 --> 00:08:24.650 align:start position:0% is is<00:08:23.160> using<00:08:23.520> a<00:08:23.639> whole<00:08:23.759> bunch<00:08:23.879> more<00:08:24.060> memory<00:08:24.419> than 00:08:24.650 --> 00:08:24.660 align:start position:0% is using a whole bunch more memory than 00:08:24.660 --> 00:08:26.990 align:start position:0% is using a whole bunch more memory than our<00:08:24.900> unstable<00:08:25.440> version<00:08:25.800> so<00:08:26.580> you<00:08:26.819> know<00:08:26.879> being 00:08:26.990 --> 00:08:27.000 align:start position:0% our unstable version so you know being 00:08:27.000 --> 00:08:29.510 align:start position:0% our unstable version so you know being able<00:08:27.240> to<00:08:27.419> kind<00:08:27.539> of<00:08:27.599> compare<00:08:27.960> and<00:08:28.139> contrast 00:08:29.510 --> 00:08:29.520 align:start position:0% able to kind of compare and contrast 00:08:29.520 --> 00:08:32.630 align:start position:0% able to kind of compare and contrast um<00:08:30.000> it<00:08:30.479> gives<00:08:30.660> you<00:08:30.780> the<00:08:30.900> tools<00:08:31.379> to<00:08:31.500> do<00:08:31.620> this<00:08:31.800> and 00:08:32.630 --> 00:08:32.640 align:start position:0% um it gives you the tools to do this and 00:08:32.640 --> 00:08:34.490 align:start position:0% um it gives you the tools to do this and so<00:08:32.760> just<00:08:32.940> just<00:08:33.120> kind<00:08:33.360> of<00:08:33.479> like<00:08:33.659> drilling<00:08:34.380> down 00:08:34.490 --> 00:08:34.500 align:start position:0% so just just kind of like drilling down 00:08:34.500 --> 00:08:36.769 align:start position:0% so just just kind of like drilling down to<00:08:34.620> a<00:08:34.800> few<00:08:34.919> a<00:08:35.099> few<00:08:35.279> few<00:08:35.279> tips 00:08:36.769 --> 00:08:36.779 align:start position:0% to a few a few few tips 00:08:36.779 --> 00:08:38.870 align:start position:0% to a few a few few tips um<00:08:36.839> these<00:08:37.560> are<00:08:37.680> very<00:08:37.919> very<00:08:38.159> practical<00:08:38.580> kind<00:08:38.760> of 00:08:38.870 --> 00:08:38.880 align:start position:0% um these are very very practical kind of 00:08:38.880 --> 00:08:40.490 align:start position:0% um these are very very practical kind of tips<00:08:39.000> don't<00:08:39.240> abuse<00:08:39.479> the<00:08:39.659> histograms<00:08:40.140> so<00:08:40.320> I 00:08:40.490 --> 00:08:40.500 align:start position:0% tips don't abuse the histograms so I 00:08:40.500 --> 00:08:41.510 align:start position:0% tips don't abuse the histograms so I said<00:08:40.620> there's<00:08:40.740> three<00:08:40.919> different<00:08:41.099> types<00:08:41.399> of 00:08:41.510 --> 00:08:41.520 align:start position:0% said there's three different types of 00:08:41.520 --> 00:08:43.190 align:start position:0% said there's three different types of metrics 00:08:43.190 --> 00:08:43.200 align:start position:0% metrics 00:08:43.200 --> 00:08:44.990 align:start position:0% metrics um you<00:08:43.740> just<00:08:43.919> wouldn't<00:08:44.099> doubt<00:08:44.399> you<00:08:44.459> use<00:08:44.820> the 00:08:44.990 --> 00:08:45.000 align:start position:0% um you just wouldn't doubt you use the 00:08:45.000 --> 00:08:46.910 align:start position:0% um you just wouldn't doubt you use the simplest<00:08:45.300> tool<00:08:45.600> possible<00:08:45.959> histograms<00:08:46.800> are 00:08:46.910 --> 00:08:46.920 align:start position:0% simplest tool possible histograms are 00:08:46.920 --> 00:08:48.290 align:start position:0% simplest tool possible histograms are really<00:08:47.100> only<00:08:47.279> used<00:08:47.459> if<00:08:47.700> you're<00:08:47.820> wanting<00:08:48.120> to 00:08:48.290 --> 00:08:48.300 align:start position:0% really only used if you're wanting to 00:08:48.300 --> 00:08:50.449 align:start position:0% really only used if you're wanting to see<00:08:48.660> the<00:08:48.839> kind<00:08:48.959> of<00:08:49.080> the<00:08:49.140> distribution<00:08:49.560> of<00:08:49.980> of 00:08:50.449 --> 00:08:50.459 align:start position:0% see the kind of the distribution of of 00:08:50.459 --> 00:08:53.509 align:start position:0% see the kind of the distribution of of uh<00:08:51.060> of<00:08:51.660> something<00:08:51.899> so<00:08:52.260> like<00:08:52.500> maybe<00:08:53.100> like 00:08:53.509 --> 00:08:53.519 align:start position:0% uh of something so like maybe like 00:08:53.519 --> 00:08:56.449 align:start position:0% uh of something so like maybe like looking<00:08:53.940> at<00:08:54.180> like<00:08:54.480> request<00:08:55.019> timing<00:08:55.560> maybe 00:08:56.449 --> 00:08:56.459 align:start position:0% looking at like request timing maybe 00:08:56.459 --> 00:08:57.590 align:start position:0% looking at like request timing maybe you'll<00:08:56.700> want<00:08:56.820> to<00:08:56.940> see<00:08:57.000> like<00:08:57.240> all<00:08:57.420> right 00:08:57.590 --> 00:08:57.600 align:start position:0% you'll want to see like all right 00:08:57.600 --> 00:08:59.870 align:start position:0% you'll want to see like all right certain<00:08:57.899> requests<00:08:58.380> are<00:08:58.560> happening<00:08:59.100> very<00:08:59.459> very 00:08:59.870 --> 00:08:59.880 align:start position:0% certain requests are happening very very 00:08:59.880 --> 00:09:01.190 align:start position:0% certain requests are happening very very quickly<00:09:00.240> but<00:09:00.420> then<00:09:00.600> others<00:09:00.779> are<00:09:00.899> happening 00:09:01.190 --> 00:09:01.200 align:start position:0% quickly but then others are happening 00:09:01.200 --> 00:09:03.350 align:start position:0% quickly but then others are happening kind<00:09:01.440> of<00:09:01.560> longer 00:09:03.350 --> 00:09:03.360 align:start position:0% kind of longer 00:09:03.360 --> 00:09:04.250 align:start position:0% kind of longer um so<00:09:03.540> the<00:09:03.660> classic<00:09:03.839> mistake<00:09:04.140> with 00:09:04.250 --> 00:09:04.260 align:start position:0% um so the classic mistake with 00:09:04.260 --> 00:09:05.810 align:start position:0% um so the classic mistake with histograms<00:09:04.800> is 00:09:05.810 --> 00:09:05.820 align:start position:0% histograms is 00:09:05.820 --> 00:09:06.710 align:start position:0% histograms is um 00:09:06.710 --> 00:09:06.720 align:start position:0% um 00:09:06.720 --> 00:09:10.130 align:start position:0% um is<00:09:07.320> uh<00:09:07.620> using<00:09:07.980> a<00:09:08.160> label<00:09:08.399> that's<00:09:08.700> unbounded 00:09:10.130 --> 00:09:10.140 align:start position:0% is uh using a label that's unbounded 00:09:10.140 --> 00:09:12.110 align:start position:0% is uh using a label that's unbounded so<00:09:10.500> if<00:09:10.620> you<00:09:10.740> had<00:09:10.860> a<00:09:11.040> metric<00:09:11.459> that<00:09:11.640> was<00:09:11.760> tracking 00:09:12.110 --> 00:09:12.120 align:start position:0% so if you had a metric that was tracking 00:09:12.120 --> 00:09:14.210 align:start position:0% so if you had a metric that was tracking something<00:09:12.300> per<00:09:12.839> peer<00:09:13.380> ID<00:09:13.500> on<00:09:13.980> the<00:09:14.100> network 00:09:14.210 --> 00:09:14.220 align:start position:0% something per peer ID on the network 00:09:14.220 --> 00:09:16.670 align:start position:0% something per peer ID on the network well<00:09:15.060> you<00:09:15.480> might<00:09:15.600> have<00:09:15.779> thousands<00:09:16.260> of<00:09:16.320> peers 00:09:16.670 --> 00:09:16.680 align:start position:0% well you might have thousands of peers 00:09:16.680 --> 00:09:19.250 align:start position:0% well you might have thousands of peers with<00:09:17.160> and<00:09:17.519> if<00:09:17.700> you<00:09:17.880> each<00:09:18.240> new<00:09:18.480> each<00:09:18.720> new<00:09:18.899> peer 00:09:19.250 --> 00:09:19.260 align:start position:0% with and if you each new each new peer 00:09:19.260 --> 00:09:20.930 align:start position:0% with and if you each new each new peer is<00:09:19.380> going<00:09:19.500> to<00:09:19.560> be<00:09:19.620> using<00:09:19.860> a<00:09:19.980> new<00:09:20.040> peer<00:09:20.279> ID<00:09:20.399> it's 00:09:20.930 --> 00:09:20.940 align:start position:0% is going to be using a new peer ID it's 00:09:20.940 --> 00:09:22.430 align:start position:0% is going to be using a new peer ID it's going<00:09:21.120> to<00:09:21.180> blow<00:09:21.420> the<00:09:21.600> metrics<00:09:21.959> up<00:09:22.080> and<00:09:22.260> it's 00:09:22.430 --> 00:09:22.440 align:start position:0% going to blow the metrics up and it's 00:09:22.440 --> 00:09:24.110 align:start position:0% going to blow the metrics up and it's going<00:09:22.560> to<00:09:22.620> affect<00:09:22.860> the<00:09:23.100> performance<00:09:23.880> of 00:09:24.110 --> 00:09:24.120 align:start position:0% going to affect the performance of 00:09:24.120 --> 00:09:26.269 align:start position:0% going to affect the performance of Prometheus<00:09:25.080> and<00:09:25.740> you're<00:09:25.860> not<00:09:25.980> going<00:09:26.100> to<00:09:26.220> be 00:09:26.269 --> 00:09:26.279 align:start position:0% Prometheus and you're not going to be 00:09:26.279 --> 00:09:28.250 align:start position:0% Prometheus and you're not going to be able<00:09:26.459> to<00:09:26.640> run<00:09:27.360> the<00:09:27.540> queries<00:09:27.779> that<00:09:27.959> you<00:09:28.080> want<00:09:28.140> to 00:09:28.250 --> 00:09:28.260 align:start position:0% able to run the queries that you want to 00:09:28.260 --> 00:09:30.170 align:start position:0% able to run the queries that you want to run<00:09:28.380> oh<00:09:29.040> so<00:09:29.220> how<00:09:29.399> do<00:09:29.519> you<00:09:29.580> know<00:09:29.640> which<00:09:29.820> metrics 00:09:30.170 --> 00:09:30.180 align:start position:0% run oh so how do you know which metrics 00:09:30.180 --> 00:09:31.970 align:start position:0% run oh so how do you know which metrics to<00:09:30.300> add<00:09:30.480> so<00:09:30.839> I<00:09:31.320> showed<00:09:31.500> some<00:09:31.680> of<00:09:31.740> these<00:09:31.800> pretty 00:09:31.970 --> 00:09:31.980 align:start position:0% to add so I showed some of these pretty 00:09:31.980 --> 00:09:33.590 align:start position:0% to add so I showed some of these pretty graphs<00:09:32.279> and<00:09:32.459> everything 00:09:33.590 --> 00:09:33.600 align:start position:0% graphs and everything 00:09:33.600 --> 00:09:35.930 align:start position:0% graphs and everything um<00:09:33.720> but<00:09:34.080> like<00:09:34.560> you're<00:09:35.220> in<00:09:35.399> your<00:09:35.580> in<00:09:35.760> your 00:09:35.930 --> 00:09:35.940 align:start position:0% um but like you're in your in your 00:09:35.940 --> 00:09:37.610 align:start position:0% um but like you're in your in your software<00:09:36.300> how<00:09:36.660> do<00:09:36.779> you<00:09:36.899> know<00:09:37.019> what<00:09:37.260> you<00:09:37.380> want 00:09:37.610 --> 00:09:37.620 align:start position:0% software how do you know what you want 00:09:37.620 --> 00:09:40.550 align:start position:0% software how do you know what you want what<00:09:38.279> do<00:09:38.399> you<00:09:38.519> want<00:09:38.640> to<00:09:38.760> track<00:09:39.080> so<00:09:40.080> really<00:09:40.320> what 00:09:40.550 --> 00:09:40.560 align:start position:0% what do you want to track so really what 00:09:40.560 --> 00:09:43.009 align:start position:0% what do you want to track so really what you<00:09:40.680> want<00:09:40.980> to<00:09:41.100> do<00:09:41.220> is<00:09:41.600> think<00:09:42.600> about<00:09:42.779> questions 00:09:43.009 --> 00:09:43.019 align:start position:0% you want to do is think about questions 00:09:43.019 --> 00:09:46.070 align:start position:0% you want to do is think about questions that<00:09:43.320> you<00:09:43.500> couldn't<00:09:43.680> ask<00:09:44.399> otherwise<00:09:44.959> so<00:09:45.959> a 00:09:46.070 --> 00:09:46.080 align:start position:0% that you couldn't ask otherwise so a 00:09:46.080 --> 00:09:48.050 align:start position:0% that you couldn't ask otherwise so a really<00:09:46.200> good<00:09:46.320> example<00:09:46.680> is<00:09:46.860> the<00:09:47.519> size<00:09:47.640> of<00:09:47.880> your 00:09:48.050 --> 00:09:48.060 align:start position:0% really good example is the size of your 00:09:48.060 --> 00:09:49.610 align:start position:0% really good example is the size of your internal<00:09:48.360> caches 00:09:49.610 --> 00:09:49.620 align:start position:0% internal caches 00:09:49.620 --> 00:09:51.110 align:start position:0% internal caches another<00:09:50.100> thing<00:09:50.399> you<00:09:50.459> can<00:09:50.580> think<00:09:50.700> about<00:09:50.820> is 00:09:51.110 --> 00:09:51.120 align:start position:0% another thing you can think about is 00:09:51.120 --> 00:09:53.630 align:start position:0% another thing you can think about is like<00:09:51.420> if<00:09:52.140> this<00:09:52.500> feature<00:09:52.800> or<00:09:52.980> if<00:09:53.160> this<00:09:53.279> part<00:09:53.459> of 00:09:53.630 --> 00:09:53.640 align:start position:0% like if this feature or if this part of 00:09:53.640 --> 00:09:56.509 align:start position:0% like if this feature or if this part of the<00:09:53.760> code<00:09:54.000> would<00:09:54.959> degrade<00:09:55.440> or<00:09:55.620> explode<00:09:56.100> in<00:09:56.399> the 00:09:56.509 --> 00:09:56.519 align:start position:0% the code would degrade or explode in the 00:09:56.519 --> 00:09:59.030 align:start position:0% the code would degrade or explode in the in<00:09:56.700> the<00:09:56.940> kind<00:09:57.180> of<00:09:57.300> bad<00:09:57.540> case<00:09:57.839> that<00:09:58.500> that's<00:09:58.800> a 00:09:59.030 --> 00:09:59.040 align:start position:0% in the kind of bad case that that's a 00:09:59.040 --> 00:10:01.610 align:start position:0% in the kind of bad case that that's a that's<00:09:59.279> a<00:09:59.519> prime<00:09:59.820> candidate<00:10:00.420> for<00:10:00.720> for<00:10:00.959> adding 00:10:01.610 --> 00:10:01.620 align:start position:0% that's a prime candidate for for adding 00:10:01.620 --> 00:10:03.170 align:start position:0% that's a prime candidate for for adding metrics<00:10:02.040> around<00:10:02.220> it 00:10:03.170 --> 00:10:03.180 align:start position:0% metrics around it 00:10:03.180 --> 00:10:04.970 align:start position:0% metrics around it um and<00:10:03.420> then<00:10:03.480> really<00:10:03.720> just<00:10:03.959> like<00:10:04.140> keep<00:10:04.680> asking 00:10:04.970 --> 00:10:04.980 align:start position:0% um and then really just like keep asking 00:10:04.980 --> 00:10:06.829 align:start position:0% um and then really just like keep asking questions<00:10:05.279> and<00:10:06.000> then<00:10:06.060> it's<00:10:06.240> like<00:10:06.360> if<00:10:06.540> you<00:10:06.660> have 00:10:06.829 --> 00:10:06.839 align:start position:0% questions and then it's like if you have 00:10:06.839 --> 00:10:08.690 align:start position:0% questions and then it's like if you have the<00:10:07.019> data<00:10:07.320> to<00:10:07.620> be<00:10:08.040> able<00:10:08.220> to<00:10:08.339> answer<00:10:08.459> the 00:10:08.690 --> 00:10:08.700 align:start position:0% the data to be able to answer the 00:10:08.700 --> 00:10:11.030 align:start position:0% the data to be able to answer the question<00:10:08.940> you're<00:10:09.540> good<00:10:09.720> but<00:10:09.959> otherwise<00:10:10.380> keep 00:10:11.030 --> 00:10:11.040 align:start position:0% question you're good but otherwise keep 00:10:11.040 --> 00:10:12.590 align:start position:0% question you're good but otherwise keep adding<00:10:11.339> metrics<00:10:11.700> until<00:10:11.820> you<00:10:12.000> can<00:10:12.120> until<00:10:12.360> you 00:10:12.590 --> 00:10:12.600 align:start position:0% adding metrics until you can until you 00:10:12.600 --> 00:10:14.449 align:start position:0% adding metrics until you can until you can<00:10:12.720> answer<00:10:12.839> those<00:10:13.080> questions 00:10:14.449 --> 00:10:14.459 align:start position:0% can answer those questions 00:10:14.459 --> 00:10:17.210 align:start position:0% can answer those questions so<00:10:14.940> some<00:10:15.240> examples<00:10:16.019> that<00:10:16.260> we<00:10:16.380> we've<00:10:16.860> kind<00:10:17.040> of 00:10:17.210 --> 00:10:17.220 align:start position:0% so some examples that we we've kind of 00:10:17.220 --> 00:10:19.130 align:start position:0% so some examples that we we've kind of come<00:10:17.519> across<00:10:17.880> how<00:10:18.360> often<00:10:18.600> are<00:10:18.720> our<00:10:18.839> streams 00:10:19.130 --> 00:10:19.140 align:start position:0% come across how often are our streams 00:10:19.140 --> 00:10:20.750 align:start position:0% come across how often are our streams being<00:10:19.260> reset<00:10:19.680> how<00:10:20.040> many<00:10:20.100> peers<00:10:20.399> do<00:10:20.580> we<00:10:20.700> have 00:10:20.750 --> 00:10:20.760 align:start position:0% being reset how many peers do we have 00:10:20.760 --> 00:10:23.090 align:start position:0% being reset how many peers do we have these<00:10:21.120> are<00:10:21.240> kinds<00:10:22.140> of<00:10:22.200> questions<00:10:22.440> that<00:10:22.800> that 00:10:23.090 --> 00:10:23.100 align:start position:0% these are kinds of questions that that 00:10:23.100 --> 00:10:26.329 align:start position:0% these are kinds of questions that that the<00:10:23.519> metrics<00:10:23.940> can<00:10:24.060> help<00:10:24.240> and<00:10:25.080> so<00:10:25.380> I'm<00:10:26.040> going<00:10:26.160> to 00:10:26.329 --> 00:10:26.339 align:start position:0% the metrics can help and so I'm going to 00:10:26.339 --> 00:10:28.970 align:start position:0% the metrics can help and so I'm going to show<00:10:26.459> you<00:10:26.700> just<00:10:27.300> a<00:10:27.480> live<00:10:27.779> demo<00:10:28.140> of<00:10:28.260> or<00:10:28.680> a<00:10:28.860> live 00:10:28.970 --> 00:10:28.980 align:start position:0% show you just a live demo of or a live 00:10:28.980 --> 00:10:31.610 align:start position:0% show you just a live demo of or a live uh<00:10:29.399> example<00:10:29.820> of<00:10:30.060> of<00:10:30.480> our<00:10:30.720> dashboards<00:10:31.260> and<00:10:31.440> we 00:10:31.610 --> 00:10:31.620 align:start position:0% uh example of of our dashboards and we 00:10:31.620 --> 00:10:33.889 align:start position:0% uh example of of our dashboards and we can<00:10:31.800> kind<00:10:31.920> of<00:10:32.040> run<00:10:32.580> through<00:10:32.820> some<00:10:33.060> things 00:10:33.889 --> 00:10:33.899 align:start position:0% can kind of run through some things 00:10:33.899 --> 00:10:35.750 align:start position:0% can kind of run through some things and<00:10:34.380> if<00:10:34.500> anyone<00:10:34.740> has<00:10:34.860> any<00:10:34.980> questions<00:10:35.220> about<00:10:35.519> it 00:10:35.750 --> 00:10:35.760 align:start position:0% and if anyone has any questions about it 00:10:35.760 --> 00:10:37.670 align:start position:0% and if anyone has any questions about it while<00:10:36.180> I'm<00:10:36.300> going<00:10:36.480> through<00:10:36.660> it<00:10:36.779> we<00:10:36.959> can 00:10:37.670 --> 00:10:37.680 align:start position:0% while I'm going through it we can 00:10:37.680 --> 00:10:40.190 align:start position:0% while I'm going through it we can we<00:10:38.040> can<00:10:38.160> do<00:10:38.279> that<00:10:38.399> too<00:10:39.060> okay<00:10:39.420> so<00:10:39.600> this<00:10:39.779> is<00:10:39.899> this 00:10:40.190 --> 00:10:40.200 align:start position:0% we can do that too okay so this is this 00:10:40.200 --> 00:10:45.410 align:start position:0% we can do that too okay so this is this is<00:10:40.320> a<00:10:40.800> our<00:10:41.640> dashboard<00:10:42.620> for<00:10:43.620> our<00:10:43.860> Fleet<00:10:44.579> of<00:10:45.000> of 00:10:45.410 --> 00:10:45.420 align:start position:0% is a our dashboard for our Fleet of of 00:10:45.420 --> 00:10:47.990 align:start position:0% is a our dashboard for our Fleet of of uh<00:10:45.660> Beacon<00:10:46.019> nodes<00:10:46.260> and<00:10:46.440> validators 00:10:47.990 --> 00:10:48.000 align:start position:0% uh Beacon nodes and validators 00:10:48.000 --> 00:10:49.910 align:start position:0% uh Beacon nodes and validators um<00:10:48.540> this<00:10:48.899> is<00:10:49.019> kind<00:10:49.140> of<00:10:49.260> the<00:10:49.380> high<00:10:49.740> level 00:10:49.910 --> 00:10:49.920 align:start position:0% um this is kind of the high level 00:10:49.920 --> 00:10:51.470 align:start position:0% um this is kind of the high level dashboard 00:10:51.470 --> 00:10:51.480 align:start position:0% dashboard 00:10:51.480 --> 00:10:52.970 align:start position:0% dashboard right<00:10:51.839> now<00:10:51.959> we're<00:10:52.079> looking<00:10:52.260> at<00:10:52.440> an<00:10:52.680> unstable 00:10:52.970 --> 00:10:52.980 align:start position:0% right now we're looking at an unstable 00:10:52.980 --> 00:10:56.090 align:start position:0% right now we're looking at an unstable node<00:10:53.279> we<00:10:53.519> can<00:10:53.640> go<00:10:53.760> to<00:10:53.880> like<00:10:54.200> stable<00:10:55.200> a<00:10:55.800> stable 00:10:56.090 --> 00:10:56.100 align:start position:0% node we can go to like stable a stable 00:10:56.100 --> 00:10:58.009 align:start position:0% node we can go to like stable a stable version<00:10:56.339> of<00:10:56.579> lodestar 00:10:58.009 --> 00:10:58.019 align:start position:0% version of lodestar 00:10:58.019 --> 00:11:00.590 align:start position:0% version of lodestar so<00:10:58.320> this<00:10:58.440> is<00:10:58.560> running<00:10:58.740> uh<00:10:59.220> version<00:10:59.399> 1.1<00:11:00.060> or<00:11:00.420> on 00:11:00.590 --> 00:11:00.600 align:start position:0% so this is running uh version 1.1 or on 00:11:00.600 --> 00:11:03.350 align:start position:0% so this is running uh version 1.1 or on the<00:11:00.660> girly<00:11:00.899> Network<00:11:01.140> we're<00:11:02.040> synced<00:11:02.519> it<00:11:03.060> looks 00:11:03.350 --> 00:11:03.360 align:start position:0% the girly Network we're synced it looks 00:11:03.360 --> 00:11:05.930 align:start position:0% the girly Network we're synced it looks like<00:11:03.420> it<00:11:03.480> got<00:11:03.660> restarted<00:11:04.140> about<00:11:04.440> a<00:11:04.620> day<00:11:04.740> ago 00:11:05.930 --> 00:11:05.940 align:start position:0% like it got restarted about a day ago 00:11:05.940 --> 00:11:07.550 align:start position:0% like it got restarted about a day ago some<00:11:06.420> interesting<00:11:06.720> things<00:11:06.899> I<00:11:07.140> guess<00:11:07.260> you<00:11:07.440> know 00:11:07.550 --> 00:11:07.560 align:start position:0% some interesting things I guess you know 00:11:07.560 --> 00:11:09.590 align:start position:0% some interesting things I guess you know peer<00:11:07.920> peer<00:11:08.220> count<00:11:08.399> is<00:11:08.579> looking<00:11:08.760> good<00:11:09.060> all 00:11:09.590 --> 00:11:09.600 align:start position:0% peer peer count is looking good all 00:11:09.600 --> 00:11:10.670 align:start position:0% peer peer count is looking good all right<00:11:09.720> you<00:11:10.019> know<00:11:10.079> there's<00:11:10.440> nothing 00:11:10.670 --> 00:11:10.680 align:start position:0% right you know there's nothing 00:11:10.680 --> 00:11:13.190 align:start position:0% right you know there's nothing everything<00:11:11.100> looks<00:11:11.519> stable<00:11:11.760> there<00:11:12.060> we<00:11:12.959> track 00:11:13.190 --> 00:11:13.200 align:start position:0% everything looks stable there we track 00:11:13.200 --> 00:11:14.449 align:start position:0% everything looks stable there we track different<00:11:13.440> types<00:11:13.680> of<00:11:13.800> peers<00:11:14.100> inbound 00:11:14.449 --> 00:11:14.459 align:start position:0% different types of peers inbound 00:11:14.459 --> 00:11:17.210 align:start position:0% different types of peers inbound outbound<00:11:14.899> and<00:11:15.899> uh<00:11:16.260> for<00:11:16.560> us<00:11:16.680> we<00:11:16.860> really<00:11:17.040> care 00:11:17.210 --> 00:11:17.220 align:start position:0% outbound and uh for us we really care 00:11:17.220 --> 00:11:18.170 align:start position:0% outbound and uh for us we really care about 00:11:18.170 --> 00:11:18.180 align:start position:0% about 00:11:18.180 --> 00:11:19.850 align:start position:0% about um<00:11:18.240> the<00:11:18.420> number<00:11:18.540> of<00:11:18.720> peers<00:11:19.079> that<00:11:19.620> we're<00:11:19.740> going 00:11:19.850 --> 00:11:19.860 align:start position:0% um the number of peers that we're going 00:11:19.860 --> 00:11:21.710 align:start position:0% um the number of peers that we're going to<00:11:19.980> be<00:11:20.040> receiving<00:11:20.579> messages<00:11:20.940> from<00:11:21.300> kind<00:11:21.600> of 00:11:21.710 --> 00:11:21.720 align:start position:0% to be receiving messages from kind of 00:11:21.720 --> 00:11:24.230 align:start position:0% to be receiving messages from kind of continuously<00:11:22.200> those<00:11:22.500> are<00:11:22.680> our<00:11:22.800> gossip<00:11:23.339> uh 00:11:24.230 --> 00:11:24.240 align:start position:0% continuously those are our gossip uh 00:11:24.240 --> 00:11:25.190 align:start position:0% continuously those are our gossip uh peers 00:11:25.190 --> 00:11:25.200 align:start position:0% peers 00:11:25.200 --> 00:11:28.009 align:start position:0% peers so<00:11:25.800> the<00:11:26.040> number<00:11:26.279> of<00:11:26.399> mesh<00:11:26.700> piers<00:11:27.060> on<00:11:27.779> our<00:11:27.899> core 00:11:28.009 --> 00:11:28.019 align:start position:0% so the number of mesh piers on our core 00:11:28.019 --> 00:11:29.090 align:start position:0% so the number of mesh piers on our core topics 00:11:29.090 --> 00:11:29.100 align:start position:0% topics 00:11:29.100 --> 00:11:31.190 align:start position:0% topics staying<00:11:29.579> about<00:11:29.700> six<00:11:29.940> or<00:11:30.120> seven 00:11:31.190 --> 00:11:31.200 align:start position:0% staying about six or seven 00:11:31.200 --> 00:11:33.170 align:start position:0% staying about six or seven it<00:11:31.620> looks<00:11:31.860> like<00:11:31.920> we're<00:11:32.100> connected<00:11:32.820> to<00:11:32.940> a<00:11:33.120> lot 00:11:33.170 --> 00:11:33.180 align:start position:0% it looks like we're connected to a lot 00:11:33.180 --> 00:11:34.670 align:start position:0% it looks like we're connected to a lot of<00:11:33.240> Lighthouse<00:11:33.540> nodes<00:11:33.959> and<00:11:34.140> a<00:11:34.320> lot<00:11:34.380> of<00:11:34.440> prism 00:11:34.670 --> 00:11:34.680 align:start position:0% of Lighthouse nodes and a lot of prism 00:11:34.680 --> 00:11:35.990 align:start position:0% of Lighthouse nodes and a lot of prism nodes 00:11:35.990 --> 00:11:36.000 align:start position:0% nodes 00:11:36.000 --> 00:11:38.990 align:start position:0% nodes and<00:11:36.959> there's<00:11:37.500> another<00:11:37.680> load<00:11:38.100> star<00:11:38.220> out<00:11:38.399> there 00:11:38.990 --> 00:11:39.000 align:start position:0% and there's another load star out there 00:11:39.000 --> 00:11:40.610 align:start position:0% and there's another load star out there let's<00:11:39.360> drill<00:11:39.660> into<00:11:39.720> something<00:11:39.959> a<00:11:40.200> little<00:11:40.320> a 00:11:40.610 --> 00:11:40.620 align:start position:0% let's drill into something a little a 00:11:40.620 --> 00:11:43.370 align:start position:0% let's drill into something a little a little<00:11:40.800> more<00:11:41.100> detailed<00:11:41.880> here 00:11:43.370 --> 00:11:43.380 align:start position:0% little more detailed here 00:11:43.380 --> 00:11:45.350 align:start position:0% little more detailed here so<00:11:43.740> one<00:11:43.980> thing<00:11:44.100> one<00:11:44.459> one<00:11:44.579> area<00:11:44.880> that<00:11:45.060> we<00:11:45.180> really 00:11:45.350 --> 00:11:45.360 align:start position:0% so one thing one one area that we really 00:11:45.360 --> 00:11:48.050 align:start position:0% so one thing one one area that we really struggled<00:11:45.660> with<00:11:45.839> and<00:11:46.079> where<00:11:46.260> we<00:11:46.380> added<00:11:46.880> a<00:11:47.880> lot 00:11:48.050 --> 00:11:48.060 align:start position:0% struggled with and where we added a lot 00:11:48.060 --> 00:11:50.269 align:start position:0% struggled with and where we added a lot of<00:11:48.240> metrics<00:11:48.720> is<00:11:49.019> in<00:11:49.860> our<00:11:49.980> gossip 00:11:50.269 --> 00:11:50.279 align:start position:0% of metrics is in our gossip 00:11:50.279 --> 00:11:53.150 align:start position:0% of metrics is in our gossip sub-implementation<00:11:51.000> we<00:11:51.959> found 00:11:53.150 --> 00:11:53.160 align:start position:0% sub-implementation we found 00:11:53.160 --> 00:11:56.389 align:start position:0% sub-implementation we found through<00:11:53.640> adding<00:11:54.480> a<00:11:54.600> few<00:11:54.720> metrics<00:11:55.140> that<00:11:55.440> we 00:11:56.389 --> 00:11:56.399 align:start position:0% through adding a few metrics that we 00:11:56.399 --> 00:11:59.269 align:start position:0% through adding a few metrics that we weren't<00:11:56.579> getting<00:11:56.959> blocks<00:11:57.959> on<00:11:58.200> time<00:11:58.380> and<00:11:58.920> it 00:11:59.269 --> 00:11:59.279 align:start position:0% weren't getting blocks on time and it 00:11:59.279 --> 00:12:01.310 align:start position:0% weren't getting blocks on time and it turned<00:11:59.519> out<00:11:59.640> that<00:12:00.000> we<00:12:00.240> we<00:12:00.779> didn't<00:12:01.019> have<00:12:01.140> enough 00:12:01.310 --> 00:12:01.320 align:start position:0% turned out that we we didn't have enough 00:12:01.320 --> 00:12:04.250 align:start position:0% turned out that we we didn't have enough peers<00:12:01.800> who<00:12:02.339> were<00:12:02.459> sending<00:12:02.820> us<00:12:02.940> these 00:12:04.250 --> 00:12:04.260 align:start position:0% peers who were sending us these 00:12:04.260 --> 00:12:06.530 align:start position:0% peers who were sending us these um<00:12:04.320> who<00:12:04.920> are<00:12:05.040> setting<00:12:05.279> us<00:12:05.339> blocks<00:12:05.820> and<00:12:06.300> then<00:12:06.420> we 00:12:06.530 --> 00:12:06.540 align:start position:0% um who are setting us blocks and then we 00:12:06.540 --> 00:12:07.850 align:start position:0% um who are setting us blocks and then we started<00:12:06.779> asking<00:12:07.019> questions<00:12:07.260> all<00:12:07.560> right<00:12:07.740> well 00:12:07.850 --> 00:12:07.860 align:start position:0% started asking questions all right well 00:12:07.860 --> 00:12:09.889 align:start position:0% started asking questions all right well why<00:12:08.160> are<00:12:08.279> they<00:12:08.399> not<00:12:08.519> sending<00:12:08.760> us<00:12:08.820> blocks<00:12:09.300> it 00:12:09.889 --> 00:12:09.899 align:start position:0% why are they not sending us blocks it 00:12:09.899 --> 00:12:11.030 align:start position:0% why are they not sending us blocks it turned<00:12:10.140> out<00:12:10.200> it<00:12:10.320> was<00:12:10.440> because<00:12:10.500> our<00:12:10.740> peer 00:12:11.030 --> 00:12:11.040 align:start position:0% turned out it was because our peer 00:12:11.040 --> 00:12:13.610 align:start position:0% turned out it was because our peer scoring<00:12:11.399> was<00:12:11.519> too<00:12:11.700> low<00:12:11.940> all<00:12:12.540> right<00:12:12.720> well<00:12:12.899> why 00:12:13.610 --> 00:12:13.620 align:start position:0% scoring was too low all right well why 00:12:13.620 --> 00:12:16.370 align:start position:0% scoring was too low all right well why why<00:12:13.860> was<00:12:14.160> the<00:12:14.399> score<00:12:14.519> too<00:12:14.820> low<00:12:15.000> uh<00:12:15.899> well<00:12:16.079> it<00:12:16.320> was 00:12:16.370 --> 00:12:16.380 align:start position:0% why was the score too low uh well it was 00:12:16.380 --> 00:12:17.810 align:start position:0% why was the score too low uh well it was it<00:12:16.500> turned<00:12:16.680> out<00:12:16.740> it<00:12:16.920> was<00:12:17.040> because<00:12:17.220> there<00:12:17.519> was<00:12:17.700> a 00:12:17.810 --> 00:12:17.820 align:start position:0% it turned out it was because there was a 00:12:17.820 --> 00:12:19.550 align:start position:0% it turned out it was because there was a specific<00:12:18.120> part<00:12:18.540> of<00:12:18.720> the<00:12:18.839> scoring<00:12:19.200> that<00:12:19.440> was 00:12:19.550 --> 00:12:19.560 align:start position:0% specific part of the scoring that was 00:12:19.560 --> 00:12:21.290 align:start position:0% specific part of the scoring that was that<00:12:20.100> was<00:12:20.279> too<00:12:20.459> loans<00:12:20.760> so<00:12:20.940> we<00:12:21.060> kind<00:12:21.120> of<00:12:21.240> you 00:12:21.290 --> 00:12:21.300 align:start position:0% that was too loans so we kind of you 00:12:21.300 --> 00:12:22.670 align:start position:0% that was too loans so we kind of you know<00:12:21.420> you<00:12:21.540> start<00:12:21.779> asking<00:12:22.079> questions<00:12:22.260> and<00:12:22.560> you 00:12:22.670 --> 00:12:22.680 align:start position:0% know you start asking questions and you 00:12:22.680 --> 00:12:24.230 align:start position:0% know you start asking questions and you can<00:12:22.800> build<00:12:23.040> up 00:12:24.230 --> 00:12:24.240 align:start position:0% can build up 00:12:24.240 --> 00:12:25.790 align:start position:0% can build up um you<00:12:24.480> built<00:12:24.779> you<00:12:24.959> start<00:12:25.079> building<00:12:25.380> the 00:12:25.790 --> 00:12:25.800 align:start position:0% um you built you start building the 00:12:25.800 --> 00:12:28.130 align:start position:0% um you built you start building the answers<00:12:26.100> to<00:12:26.339> that 00:12:28.130 --> 00:12:28.140 align:start position:0% answers to that 00:12:28.140 --> 00:12:30.410 align:start position:0% answers to that okay<00:12:28.620> so<00:12:28.800> what<00:12:28.920> you're<00:12:29.040> seeing<00:12:29.220> here<00:12:29.519> is<00:12:29.940> kind 00:12:30.410 --> 00:12:30.420 align:start position:0% okay so what you're seeing here is kind 00:12:30.420 --> 00:12:32.030 align:start position:0% okay so what you're seeing here is kind of<00:12:30.540> some<00:12:30.660> of<00:12:30.779> what<00:12:30.839> I<00:12:31.019> was<00:12:31.079> talking<00:12:31.260> about 00:12:32.030 --> 00:12:32.040 align:start position:0% of some of what I was talking about 00:12:32.040 --> 00:12:33.949 align:start position:0% of some of what I was talking about um we<00:12:32.339> have<00:12:32.519> breaking<00:12:33.000> out<00:12:33.120> the<00:12:33.300> score<00:12:33.480> into 00:12:33.949 --> 00:12:33.959 align:start position:0% um we have breaking out the score into 00:12:33.959 --> 00:12:36.050 align:start position:0% um we have breaking out the score into different<00:12:34.680> components 00:12:36.050 --> 00:12:36.060 align:start position:0% different components 00:12:36.060 --> 00:12:39.190 align:start position:0% different components um<00:12:36.540> that<00:12:37.380> different<00:12:37.740> parameters<00:12:38.459> that<00:12:38.760> are 00:12:39.190 --> 00:12:39.200 align:start position:0% um that different parameters that are 00:12:39.200 --> 00:12:41.870 align:start position:0% um that different parameters that are going<00:12:40.200> into<00:12:40.440> your<00:12:40.680> kind<00:12:40.920> of<00:12:40.980> aggregated<00:12:41.519> score 00:12:41.870 --> 00:12:41.880 align:start position:0% going into your kind of aggregated score 00:12:41.880 --> 00:12:44.090 align:start position:0% going into your kind of aggregated score for<00:12:42.480> peers 00:12:44.090 --> 00:12:44.100 align:start position:0% for peers 00:12:44.100 --> 00:12:45.470 align:start position:0% for peers um<00:12:44.160> what<00:12:44.399> we<00:12:44.579> were<00:12:44.700> seeing<00:12:44.880> before<00:12:45.060> is<00:12:45.360> you 00:12:45.470 --> 00:12:45.480 align:start position:0% um what we were seeing before is you 00:12:45.480 --> 00:12:47.269 align:start position:0% um what we were seeing before is you know<00:12:45.600> you'd<00:12:45.779> see<00:12:45.899> a<00:12:46.079> graph<00:12:46.320> where<00:12:46.800> the<00:12:46.980> values 00:12:47.269 --> 00:12:47.279 align:start position:0% know you'd see a graph where the values 00:12:47.279 --> 00:12:48.590 align:start position:0% know you'd see a graph where the values are<00:12:47.399> like<00:12:47.579> going<00:12:47.700> down<00:12:47.940> and<00:12:48.120> down<00:12:48.300> and<00:12:48.420> down 00:12:48.590 --> 00:12:48.600 align:start position:0% are like going down and down and down 00:12:48.600 --> 00:12:50.509 align:start position:0% are like going down and down and down over<00:12:48.779> time<00:12:49.019> or<00:12:49.260> if<00:12:49.440> they're<00:12:49.620> like<00:12:49.740> very 00:12:50.509 --> 00:12:50.519 align:start position:0% over time or if they're like very 00:12:50.519 --> 00:12:52.069 align:start position:0% over time or if they're like very negative<00:12:50.880> numbers 00:12:52.069 --> 00:12:52.079 align:start position:0% negative numbers 00:12:52.079 --> 00:12:53.810 align:start position:0% negative numbers another<00:12:52.620> really<00:12:52.860> useful<00:12:53.160> metric<00:12:53.519> that<00:12:53.639> we 00:12:53.810 --> 00:12:53.820 align:start position:0% another really useful metric that we 00:12:53.820 --> 00:12:57.170 align:start position:0% another really useful metric that we were<00:12:53.940> that<00:12:54.300> we<00:12:54.540> check<00:12:54.899> a<00:12:55.200> lot<00:12:55.260> is<00:12:56.040> um 00:12:57.170 --> 00:12:57.180 align:start position:0% were that we check a lot is um 00:12:57.180 --> 00:13:01.970 align:start position:0% were that we check a lot is um things<00:12:57.600> related<00:12:57.899> to<00:12:58.079> the<00:12:58.200> VM<00:12:58.500> so<00:12:59.339> memory<00:13:00.180> CPU 00:13:01.970 --> 00:13:01.980 align:start position:0% things related to the VM so memory CPU 00:13:01.980 --> 00:13:02.569 align:start position:0% things related to the VM so memory CPU um 00:13:02.569 --> 00:13:02.579 align:start position:0% um 00:13:02.579 --> 00:13:06.829 align:start position:0% um and<00:13:03.480> disk<00:13:04.139> storage 00:13:06.829 --> 00:13:06.839 align:start position:0% 00:13:06.839 --> 00:13:08.870 align:start position:0% so<00:13:07.380> looks<00:13:07.740> like<00:13:07.800> our<00:13:07.980> memory<00:13:08.279> is<00:13:08.399> stable 00:13:08.870 --> 00:13:08.880 align:start position:0% so looks like our memory is stable 00:13:08.880 --> 00:13:11.569 align:start position:0% so looks like our memory is stable that's<00:13:09.480> great 00:13:11.569 --> 00:13:11.579 align:start position:0% that's great 00:13:11.579 --> 00:13:12.530 align:start position:0% that's great um 00:13:12.530 --> 00:13:12.540 align:start position:0% um 00:13:12.540 --> 00:13:16.069 align:start position:0% um GC<00:13:13.139> is<00:13:13.380> only<00:13:13.500> taking 00:13:16.069 --> 00:13:16.079 align:start position:0% 00:13:16.079 --> 00:13:18.769 align:start position:0% roughly<00:13:16.800> seven<00:13:17.459> and<00:13:17.700> a<00:13:17.820> half<00:13:17.880> percent<00:13:18.000> this<00:13:18.660> is 00:13:18.769 --> 00:13:18.779 align:start position:0% roughly seven and a half percent this is 00:13:18.779 --> 00:13:30.949 align:start position:0% roughly seven and a half percent this is things<00:13:19.320> are<00:13:19.500> looking<00:13:19.680> good 00:13:30.949 --> 00:13:30.959 align:start position:0% 00:13:30.959 --> 00:13:33.710 align:start position:0% yeah<00:13:31.500> I<00:13:31.680> think<00:13:31.800> that's<00:13:31.860> enough<00:13:32.100> of<00:13:32.339> a 00:13:33.710 --> 00:13:33.720 align:start position:0% yeah I think that's enough of a 00:13:33.720 --> 00:13:36.889 align:start position:0% yeah I think that's enough of a enough<00:13:34.139> of<00:13:34.440> a<00:13:34.560> demo<00:13:34.800> here<00:13:34.980> so<00:13:36.120> um 00:13:36.889 --> 00:13:36.899 align:start position:0% enough of a demo here so um 00:13:36.899 --> 00:13:39.470 align:start position:0% enough of a demo here so um at<00:13:37.380> this<00:13:37.500> point<00:13:38.100> um<00:13:38.100> open<00:13:38.279> to<00:13:38.579> questions<00:13:38.760> and 00:13:39.470 --> 00:13:39.480 align:start position:0% at this point um open to questions and 00:13:39.480 --> 00:13:40.910 align:start position:0% at this point um open to questions and we're<00:13:39.660> also<00:13:39.959> hiring<00:13:40.260> so<00:13:40.380> if<00:13:40.560> any<00:13:40.680> of<00:13:40.800> this 00:13:40.910 --> 00:13:40.920 align:start position:0% we're also hiring so if any of this 00:13:40.920 --> 00:13:43.069 align:start position:0% we're also hiring so if any of this looks<00:13:41.279> cool<00:13:41.459> or<00:13:41.880> interesting<00:13:42.240> and<00:13:42.779> you<00:13:42.959> know 00:13:43.069 --> 00:13:43.079 align:start position:0% looks cool or interesting and you know 00:13:43.079 --> 00:13:44.210 align:start position:0% looks cool or interesting and you know typescript 00:13:44.210 --> 00:13:44.220 align:start position:0% typescript 00:13:44.220 --> 00:13:46.310 align:start position:0% typescript hello<00:13:44.700> and<00:13:44.880> thank<00:13:45.060> you<00:13:45.180> for<00:13:45.240> the<00:13:45.420> talk 00:13:46.310 --> 00:13:46.320 align:start position:0% hello and thank you for the talk 00:13:46.320 --> 00:13:48.290 align:start position:0% hello and thank you for the talk um<00:13:46.380> so<00:13:46.560> I<00:13:46.800> see<00:13:46.980> that<00:13:47.160> you<00:13:47.279> have<00:13:47.519> a<00:13:48.000> ton<00:13:48.120> of 00:13:48.290 --> 00:13:48.300 align:start position:0% um so I see that you have a ton of 00:13:48.300 --> 00:13:51.230 align:start position:0% um so I see that you have a ton of metrics<00:13:48.779> and<00:13:49.019> I<00:13:49.320> believe<00:13:50.279> that<00:13:50.519> most<00:13:51.000> of<00:13:51.120> them 00:13:51.230 --> 00:13:51.240 align:start position:0% metrics and I believe that most of them 00:13:51.240 --> 00:13:53.629 align:start position:0% metrics and I believe that most of them are<00:13:51.420> useful<00:13:51.779> but 00:13:53.629 --> 00:13:53.639 align:start position:0% are useful but 00:13:53.639 --> 00:13:55.910 align:start position:0% are useful but um<00:13:53.700> do<00:13:54.240> you<00:13:54.420> have<00:13:54.540> some<00:13:54.779> kind<00:13:54.959> of<00:13:55.079> an<00:13:55.320> alerting 00:13:55.910 --> 00:13:55.920 align:start position:0% um do you have some kind of an alerting 00:13:55.920 --> 00:14:00.290 align:start position:0% um do you have some kind of an alerting system<00:13:56.420> for<00:13:57.420> you<00:13:57.540> to<00:13:57.839> be<00:13:58.079> able<00:13:58.320> to<00:13:58.800> know<00:13:59.519> when<00:13:59.880> a 00:14:00.290 --> 00:14:00.300 align:start position:0% system for you to be able to know when a 00:14:00.300 --> 00:14:03.590 align:start position:0% system for you to be able to know when a metric<00:14:00.720> has<00:14:01.200> gone<00:14:01.500> wrong<00:14:02.040> because<00:14:02.540> I<00:14:03.540> think 00:14:03.590 --> 00:14:03.600 align:start position:0% metric has gone wrong because I think 00:14:03.600 --> 00:14:05.930 align:start position:0% metric has gone wrong because I think it's<00:14:03.899> just<00:14:04.139> unfeasible<00:14:04.860> to<00:14:05.160> go<00:14:05.279> through<00:14:05.519> each 00:14:05.930 --> 00:14:05.940 align:start position:0% it's just unfeasible to go through each 00:14:05.940 --> 00:14:07.670 align:start position:0% it's just unfeasible to go through each of<00:14:06.120> them<00:14:06.300> like<00:14:06.600> daily<00:14:06.899> or<00:14:07.200> something<00:14:07.380> like 00:14:07.670 --> 00:14:07.680 align:start position:0% of them like daily or something like 00:14:07.680 --> 00:14:09.410 align:start position:0% of them like daily or something like that<00:14:07.800> you<00:14:08.040> I<00:14:08.459> think<00:14:08.639> you<00:14:08.760> need<00:14:08.880> some<00:14:09.060> automated 00:14:09.410 --> 00:14:09.420 align:start position:0% that you I think you need some automated 00:14:09.420 --> 00:14:11.329 align:start position:0% that you I think you need some automated way<00:14:09.720> to<00:14:09.899> to<00:14:10.260> alert<00:14:10.620> you<00:14:10.800> that<00:14:10.980> something<00:14:11.160> is 00:14:11.329 --> 00:14:11.339 align:start position:0% way to to alert you that something is 00:14:11.339 --> 00:14:12.410 align:start position:0% way to to alert you that something is going<00:14:11.459> wrong 00:14:12.410 --> 00:14:12.420 align:start position:0% going wrong 00:14:12.420 --> 00:14:14.389 align:start position:0% going wrong yeah<00:14:12.959> absolutely<00:14:13.380> so<00:14:13.740> that<00:14:13.980> was<00:14:14.040> something<00:14:14.160> I 00:14:14.389 --> 00:14:14.399 align:start position:0% yeah absolutely so that was something I 00:14:14.399 --> 00:14:16.129 align:start position:0% yeah absolutely so that was something I didn't<00:14:14.519> touch<00:14:14.700> on<00:14:14.880> but<00:14:15.660> um<00:14:15.660> it<00:14:15.899> definitely 00:14:16.129 --> 00:14:16.139 align:start position:0% didn't touch on but um it definitely 00:14:16.139 --> 00:14:17.629 align:start position:0% didn't touch on but um it definitely feeds<00:14:16.560> in 00:14:17.629 --> 00:14:17.639 align:start position:0% feeds in 00:14:17.639 --> 00:14:21.650 align:start position:0% feeds in um so<00:14:18.180> we<00:14:18.360> we<00:14:18.899> use<00:14:19.200> uh<00:14:19.920> pagerduty<00:14:20.639> you<00:14:20.880> can<00:14:21.000> you 00:14:21.650 --> 00:14:21.660 align:start position:0% um so we we use uh pagerduty you can you 00:14:21.660 --> 00:14:23.389 align:start position:0% um so we we use uh pagerduty you can you can<00:14:21.720> basically<00:14:21.959> set<00:14:22.200> up 00:14:23.389 --> 00:14:23.399 align:start position:0% can basically set up 00:14:23.399 --> 00:14:25.490 align:start position:0% can basically set up um thresholds<00:14:24.240> for<00:14:24.420> certain<00:14:24.600> metrics<00:14:25.079> that 00:14:25.490 --> 00:14:25.500 align:start position:0% um thresholds for certain metrics that 00:14:25.500 --> 00:14:26.930 align:start position:0% um thresholds for certain metrics that trigger 00:14:26.930 --> 00:14:26.940 align:start position:0% trigger 00:14:26.940 --> 00:14:28.850 align:start position:0% trigger um<00:14:27.000> either<00:14:27.180> slack<00:14:27.720> notifications<00:14:28.260> or<00:14:28.560> Discord 00:14:28.850 --> 00:14:28.860 align:start position:0% um either slack notifications or Discord 00:14:28.860 --> 00:14:30.829 align:start position:0% um either slack notifications or Discord notifications<00:14:29.339> or<00:14:29.820> in<00:14:30.360> the<00:14:30.480> case<00:14:30.600> of<00:14:30.720> a 00:14:30.829 --> 00:14:30.839 align:start position:0% notifications or in the case of a 00:14:30.839 --> 00:14:33.650 align:start position:0% notifications or in the case of a critical<00:14:31.139> kind<00:14:31.620> of<00:14:31.800> issue<00:14:32.339> it'll<00:14:32.760> call<00:14:33.000> you<00:14:33.240> or 00:14:33.650 --> 00:14:33.660 align:start position:0% critical kind of issue it'll call you or 00:14:33.660 --> 00:14:37.009 align:start position:0% critical kind of issue it'll call you or text<00:14:34.200> you<00:14:34.380> yeah<00:14:35.339> we<00:14:35.519> do<00:14:35.639> have<00:14:35.760> that<00:14:35.880> set<00:14:36.060> up<00:14:36.180> I 00:14:37.009 --> 00:14:37.019 align:start position:0% text you yeah we do have that set up I 00:14:37.019 --> 00:14:38.449 align:start position:0% text you yeah we do have that set up I would<00:14:37.139> definitely<00:14:37.320> recommend<00:14:37.680> that<00:14:37.920> if<00:14:38.220> you 00:14:38.449 --> 00:14:38.459 align:start position:0% would definitely recommend that if you 00:14:38.459 --> 00:14:41.269 align:start position:0% would definitely recommend that if you are<00:14:38.940> in<00:14:39.120> running<00:14:39.600> something<00:14:39.899> in<00:14:40.139> production 00:14:41.269 --> 00:14:41.279 align:start position:0% are in running something in production 00:14:41.279 --> 00:14:42.889 align:start position:0% are in running something in production uh<00:14:41.940> thank<00:14:42.180> you 00:14:42.889 --> 00:14:42.899 align:start position:0% uh thank you 00:14:42.899 --> 00:14:45.050 align:start position:0% uh thank you um<00:14:43.019> again<00:14:43.500> there<00:14:43.800> are<00:14:43.920> lots<00:14:44.399> of<00:14:44.459> metrics<00:14:44.880> do 00:14:45.050 --> 00:14:45.060 align:start position:0% um again there are lots of metrics do 00:14:45.060 --> 00:14:47.810 align:start position:0% um again there are lots of metrics do you<00:14:45.180> did<00:14:45.480> you<00:14:45.600> guys<00:14:46.019> um<00:14:46.019> develop<00:14:46.440> over<00:14:46.800> time 00:14:47.810 --> 00:14:47.820 align:start position:0% you did you guys um develop over time 00:14:47.820 --> 00:14:50.150 align:start position:0% you did you guys um develop over time um<00:14:47.880> a<00:14:48.060> way<00:14:48.300> to<00:14:48.480> or<00:14:48.899> you<00:14:49.199> know<00:14:49.260> kind<00:14:49.440> of<00:14:49.620> a<00:14:49.800> mental 00:14:50.150 --> 00:14:50.160 align:start position:0% um a way to or you know kind of a mental 00:14:50.160 --> 00:14:52.550 align:start position:0% um a way to or you know kind of a mental model<00:14:50.339> or<00:14:50.519> framework<00:14:50.940> to<00:14:51.300> decide<00:14:52.139> how<00:14:52.380> to 00:14:52.550 --> 00:14:52.560 align:start position:0% model or framework to decide how to 00:14:52.560 --> 00:14:54.889 align:start position:0% model or framework to decide how to where<00:14:52.800> to<00:14:52.980> put<00:14:53.100> them<00:14:53.279> whenever<00:14:53.639> you<00:14:53.820> have<00:14:54.000> new 00:14:54.889 --> 00:14:54.899 align:start position:0% where to put them whenever you have new 00:14:54.899 --> 00:14:56.870 align:start position:0% where to put them whenever you have new um you<00:14:55.139> know<00:14:55.199> graph<00:14:55.440> or<00:14:55.620> metric<00:14:55.980> instead<00:14:56.760> of 00:14:56.870 --> 00:14:56.880 align:start position:0% um you know graph or metric instead of 00:14:56.880 --> 00:14:59.030 align:start position:0% um you know graph or metric instead of or<00:14:57.240> do<00:14:57.360> you<00:14:57.540> just<00:14:57.660> like<00:14:57.899> decide<00:14:58.440> to<00:14:58.800> put<00:14:58.920> it 00:14:59.030 --> 00:14:59.040 align:start position:0% or do you just like decide to put it 00:14:59.040 --> 00:15:00.889 align:start position:0% or do you just like decide to put it somewhere<00:14:59.279> and<00:14:59.399> then<00:14:59.519> go<00:14:59.760> back<00:14:59.880> and<00:15:00.060> like<00:15:00.360> you 00:15:00.889 --> 00:15:00.899 align:start position:0% somewhere and then go back and like you 00:15:00.899 --> 00:15:03.170 align:start position:0% somewhere and then go back and like you know<00:15:00.959> to<00:15:01.139> make<00:15:01.260> it<00:15:01.440> easy<00:15:01.560> to<00:15:01.860> to<00:15:02.040> read<00:15:02.279> or<00:15:02.519> to<00:15:02.820> uh 00:15:03.170 --> 00:15:03.180 align:start position:0% know to make it easy to to read or to uh 00:15:03.180 --> 00:15:05.509 align:start position:0% know to make it easy to to read or to uh browse<00:15:03.720> the<00:15:03.899> dashboard 00:15:05.509 --> 00:15:05.519 align:start position:0% browse the dashboard 00:15:05.519 --> 00:15:07.129 align:start position:0% browse the dashboard I<00:15:05.880> don't<00:15:06.000> know<00:15:06.120> if<00:15:06.180> that<00:15:06.300> makes<00:15:06.480> sense<00:15:06.720> this<00:15:06.959> is 00:15:07.129 --> 00:15:07.139 align:start position:0% I don't know if that makes sense this is 00:15:07.139 --> 00:15:08.750 align:start position:0% I don't know if that makes sense this is the<00:15:07.199> question<00:15:07.380> about<00:15:07.560> organizing<00:15:08.279> the<00:15:08.459> yeah 00:15:08.750 --> 00:15:08.760 align:start position:0% the question about organizing the yeah 00:15:08.760 --> 00:15:12.170 align:start position:0% the question about organizing the yeah exactly<00:15:09.240> yeah<00:15:10.139> so<00:15:10.980> um<00:15:11.040> I<00:15:11.220> think<00:15:11.399> probably<00:15:11.940> have 00:15:12.170 --> 00:15:12.180 align:start position:0% exactly yeah so um I think probably have 00:15:12.180 --> 00:15:13.850 align:start position:0% exactly yeah so um I think probably have to<00:15:12.300> grow<00:15:12.420> it<00:15:12.540> organically<00:15:13.019> what<00:15:13.199> we<00:15:13.380> did<00:15:13.560> is<00:15:13.680> we 00:15:13.850 --> 00:15:13.860 align:start position:0% to grow it organically what we did is we 00:15:13.860 --> 00:15:16.430 align:start position:0% to grow it organically what we did is we we<00:15:14.100> started<00:15:14.459> with<00:15:14.699> One<00:15:15.240> dashboard<00:15:15.839> That<00:15:16.139> Grew 00:15:16.430 --> 00:15:16.440 align:start position:0% we started with One dashboard That Grew 00:15:16.440 --> 00:15:18.829 align:start position:0% we started with One dashboard That Grew and<00:15:16.500> grew<00:15:16.740> and<00:15:16.800> grew<00:15:17.160> and<00:15:17.519> eventually<00:15:18.060> it 00:15:18.829 --> 00:15:18.839 align:start position:0% and grew and grew and eventually it 00:15:18.839 --> 00:15:20.750 align:start position:0% and grew and grew and eventually it became<00:15:19.139> kind<00:15:19.380> of<00:15:19.500> unsustainable<00:15:20.339> to<00:15:20.639> have 00:15:20.750 --> 00:15:20.760 align:start position:0% became kind of unsustainable to have 00:15:20.760 --> 00:15:23.269 align:start position:0% became kind of unsustainable to have everything<00:15:21.060> just<00:15:21.720> piled<00:15:22.440> together<00:15:22.560> so<00:15:22.920> we<00:15:23.100> we 00:15:23.269 --> 00:15:23.279 align:start position:0% everything just piled together so we we 00:15:23.279 --> 00:15:25.189 align:start position:0% everything just piled together so we we started<00:15:23.579> breaking<00:15:23.880> it<00:15:23.940> apart 00:15:25.189 --> 00:15:25.199 align:start position:0% started breaking it apart 00:15:25.199 --> 00:15:26.990 align:start position:0% started breaking it apart um<00:15:25.320> the<00:15:25.740> other<00:15:25.860> thing<00:15:26.040> that<00:15:26.339> the<00:15:26.579> other<00:15:26.760> kind 00:15:26.990 --> 00:15:27.000 align:start position:0% um the other thing that the other kind 00:15:27.000 --> 00:15:29.750 align:start position:0% um the other thing that the other kind of<00:15:27.120> issue<00:15:27.779> that<00:15:27.899> we<00:15:28.079> had<00:15:28.260> initially<00:15:29.100> is<00:15:29.339> we 00:15:29.750 --> 00:15:29.760 align:start position:0% of issue that we had initially is we 00:15:29.760 --> 00:15:31.129 align:start position:0% of issue that we had initially is we would<00:15:29.940> create<00:15:30.060> a<00:15:30.240> bunch<00:15:30.360> of<00:15:30.420> metrics<00:15:30.839> but<00:15:31.019> we 00:15:31.129 --> 00:15:31.139 align:start position:0% would create a bunch of metrics but we 00:15:31.139 --> 00:15:33.230 align:start position:0% would create a bunch of metrics but we wouldn't<00:15:31.320> add<00:15:31.680> them<00:15:31.800> to<00:15:31.980> the<00:15:32.100> dashboard<00:15:32.459> so 00:15:33.230 --> 00:15:33.240 align:start position:0% wouldn't add them to the dashboard so 00:15:33.240 --> 00:15:35.569 align:start position:0% wouldn't add them to the dashboard so there<00:15:33.480> was<00:15:33.600> no<00:15:33.779> actual<00:15:34.079> like<00:15:34.320> visual<00:15:34.740> cue<00:15:35.220> for 00:15:35.569 --> 00:15:35.579 align:start position:0% there was no actual like visual cue for 00:15:35.579 --> 00:15:37.610 align:start position:0% there was no actual like visual cue for for<00:15:35.940> a<00:15:36.600> lot<00:15:36.660> of<00:15:36.779> it<00:15:36.899> and<00:15:37.019> so<00:15:37.139> it<00:15:37.260> kind<00:15:37.440> of<00:15:37.500> just 00:15:37.610 --> 00:15:37.620 align:start position:0% for a lot of it and so it kind of just 00:15:37.620 --> 00:15:39.290 align:start position:0% for a lot of it and so it kind of just went<00:15:37.860> un 00:15:39.290 --> 00:15:39.300 align:start position:0% went un 00:15:39.300 --> 00:15:41.689 align:start position:0% went un um<00:15:39.360> unnoticed<00:15:39.959> and<00:15:40.680> so<00:15:40.860> one<00:15:41.279> thing<00:15:41.399> we<00:15:41.579> would 00:15:41.689 --> 00:15:41.699 align:start position:0% um unnoticed and so one thing we would 00:15:41.699 --> 00:15:43.009 align:start position:0% um unnoticed and so one thing we would we<00:15:41.940> would<00:15:42.060> definitely<00:15:42.300> recommend<00:15:42.600> would<00:15:42.899> be 00:15:43.009 --> 00:15:43.019 align:start position:0% we would definitely recommend would be 00:15:43.019 --> 00:15:45.530 align:start position:0% we would definitely recommend would be like<00:15:43.139> if<00:15:43.320> you<00:15:43.440> add<00:15:43.740> a<00:15:43.980> metric<00:15:44.519> also<00:15:45.180> add<00:15:45.300> it<00:15:45.420> to 00:15:45.530 --> 00:15:45.540 align:start position:0% like if you add a metric also add it to 00:15:45.540 --> 00:15:48.050 align:start position:0% like if you add a metric also add it to the<00:15:45.660> dashboard<00:15:45.899> at<00:15:46.079> the<00:15:46.260> same<00:15:46.320> time<00:15:46.880> thank<00:15:47.880> you 00:15:48.050 --> 00:15:48.060 align:start position:0% the dashboard at the same time thank you 00:15:48.060 --> 00:15:51.889 align:start position:0% the dashboard at the same time thank you uh<00:15:48.899> the<00:15:49.199> question<00:15:49.440> about<00:15:49.740> storage<00:15:50.660> so<00:15:51.660> again 00:15:51.889 --> 00:15:51.899 align:start position:0% uh the question about storage so again 00:15:51.899 --> 00:15:55.670 align:start position:0% uh the question about storage so again too<00:15:52.620> much<00:15:52.800> metrics<00:15:53.720> or<00:15:54.720> what<00:15:54.839> does<00:15:55.320> it<00:15:55.440> affect 00:15:55.670 --> 00:15:55.680 align:start position:0% too much metrics or what does it affect 00:15:55.680 --> 00:15:57.470 align:start position:0% too much metrics or what does it affect the<00:15:55.860> storage 00:15:57.470 --> 00:15:57.480 align:start position:0% the storage 00:15:57.480 --> 00:16:00.230 align:start position:0% the storage um<00:15:57.600> well<00:15:58.139> so<00:15:58.500> some<00:15:59.339> of<00:15:59.459> it<00:15:59.519> kind<00:15:59.760> of<00:15:59.820> depends<00:16:00.120> on 00:16:00.230 --> 00:16:00.240 align:start position:0% um well so some of it kind of depends on 00:16:00.240 --> 00:16:01.670 align:start position:0% um well so some of it kind of depends on your<00:16:00.360> architecture 00:16:01.670 --> 00:16:01.680 align:start position:0% your architecture 00:16:01.680 --> 00:16:05.329 align:start position:0% your architecture um<00:16:01.740> of<00:16:02.279> how<00:16:02.519> you're<00:16:02.940> deploying<00:16:03.660> this<00:16:03.779> so 00:16:05.329 --> 00:16:05.339 align:start position:0% um of how you're deploying this so 00:16:05.339 --> 00:16:08.090 align:start position:0% um of how you're deploying this so um<00:16:05.579> what<00:16:06.120> we<00:16:06.300> do<00:16:06.420> is<00:16:06.600> we<00:16:06.779> actually<00:16:06.959> have<00:16:07.440> one 00:16:08.090 --> 00:16:08.100 align:start position:0% um what we do is we actually have one 00:16:08.100 --> 00:16:11.210 align:start position:0% um what we do is we actually have one Prometheus<00:16:09.060> and<00:16:09.180> grafana<00:16:09.540> server<00:16:10.220> servicing 00:16:11.210 --> 00:16:11.220 align:start position:0% Prometheus and grafana server servicing 00:16:11.220 --> 00:16:14.569 align:start position:0% Prometheus and grafana server servicing all<00:16:11.579> our<00:16:11.880> entire<00:16:12.120> fleet<00:16:12.540> of<00:16:12.899> of<00:16:13.500> Beacon<00:16:14.220> nodes 00:16:14.569 --> 00:16:14.579 align:start position:0% all our entire fleet of of Beacon nodes 00:16:14.579 --> 00:16:16.670 align:start position:0% all our entire fleet of of Beacon nodes so<00:16:15.180> you<00:16:15.660> know<00:16:15.720> we<00:16:15.839> have<00:16:15.959> a<00:16:16.079> bunch<00:16:16.380> of<00:16:16.440> different 00:16:16.670 --> 00:16:16.680 align:start position:0% so you know we have a bunch of different 00:16:16.680 --> 00:16:18.350 align:start position:0% so you know we have a bunch of different machines<00:16:17.220> and<00:16:17.399> then<00:16:17.519> we<00:16:17.699> have<00:16:17.760> one<00:16:18.060> one 00:16:18.350 --> 00:16:18.360 align:start position:0% machines and then we have one one 00:16:18.360 --> 00:16:22.310 align:start position:0% machines and then we have one one machine<00:16:18.660> for<00:16:19.079> these<00:16:19.440> uh<00:16:20.399> right<00:16:20.880> to<00:16:21.120> to<00:16:21.779> uh 00:16:22.310 --> 00:16:22.320 align:start position:0% machine for these uh right to to uh 00:16:22.320 --> 00:16:23.449 align:start position:0% machine for these uh right to to uh things<00:16:22.560> so 00:16:23.449 --> 00:16:23.459 align:start position:0% things so 00:16:23.459 --> 00:16:26.150 align:start position:0% things so it<00:16:23.940> it<00:16:24.060> doesn't<00:16:24.540> cut<00:16:24.720> into<00:16:24.959> the<00:16:25.260> to<00:16:25.440> the<00:16:25.800> uh 00:16:26.150 --> 00:16:26.160 align:start position:0% it it doesn't cut into the to the uh 00:16:26.160 --> 00:16:29.389 align:start position:0% it it doesn't cut into the to the uh storage<00:16:26.820> we<00:16:27.600> retain<00:16:27.959> for<00:16:28.199> a<00:16:28.320> month<00:16:28.860> quick 00:16:29.389 --> 00:16:29.399 align:start position:0% storage we retain for a month quick 00:16:29.399 --> 00:16:31.970 align:start position:0% storage we retain for a month quick question<00:16:29.579> uh<00:16:30.240> is<00:16:30.540> are<00:16:30.959> the<00:16:31.139> metrics<00:16:31.560> relevant 00:16:31.970 --> 00:16:31.980 align:start position:0% question uh is are the metrics relevant 00:16:31.980 --> 00:16:33.829 align:start position:0% question uh is are the metrics relevant for<00:16:32.279> like<00:16:32.459> other<00:16:32.639> evm<00:16:33.120> chains<00:16:33.420> besides 00:16:33.829 --> 00:16:33.839 align:start position:0% for like other evm chains besides 00:16:33.839 --> 00:16:35.329 align:start position:0% for like other evm chains besides ethereum<00:16:34.199> mainnet 00:16:35.329 --> 00:16:35.339 align:start position:0% ethereum mainnet 00:16:35.339 --> 00:16:37.730 align:start position:0% ethereum mainnet um<00:16:35.399> or<00:16:35.639> is<00:16:35.820> it<00:16:36.000> just<00:16:36.180> like<00:16:36.600> specific<00:16:37.500> to 00:16:37.730 --> 00:16:37.740 align:start position:0% um or is it just like specific to 00:16:37.740 --> 00:16:40.550 align:start position:0% um or is it just like specific to ethereum<00:16:38.100> mainnet<00:16:38.519> so<00:16:39.000> yeah<00:16:39.420> a<00:16:39.720> good<00:16:39.839> question 00:16:40.550 --> 00:16:40.560 align:start position:0% ethereum mainnet so yeah a good question 00:16:40.560 --> 00:16:41.150 align:start position:0% ethereum mainnet so yeah a good question um 00:16:41.150 --> 00:16:41.160 align:start position:0% um 00:16:41.160 --> 00:16:43.009 align:start position:0% um so<00:16:41.519> I'd<00:16:41.699> say<00:16:41.880> like<00:16:42.060> I<00:16:42.540> don't<00:16:42.600> know<00:16:42.660> if<00:16:42.720> it's<00:16:42.899> a 00:16:43.009 --> 00:16:43.019 align:start position:0% so I'd say like I don't know if it's a 00:16:43.019 --> 00:16:44.689 align:start position:0% so I'd say like I don't know if it's a half<00:16:43.139> and<00:16:43.320> half<00:16:43.440> but<00:16:43.740> like<00:16:43.920> half<00:16:44.160> the<00:16:44.339> metrics 00:16:44.689 --> 00:16:44.699 align:start position:0% half and half but like half the metrics 00:16:44.699 --> 00:16:46.249 align:start position:0% half and half but like half the metrics are<00:16:44.820> like<00:16:45.000> more<00:16:45.180> like<00:16:45.360> for<00:16:45.540> like<00:16:45.720> monitoring 00:16:46.249 --> 00:16:46.259 align:start position:0% are like more like for like monitoring 00:16:46.259 --> 00:16:48.829 align:start position:0% are like more like for like monitoring uh<00:16:46.500> the<00:16:46.740> chain<00:16:46.920> something<00:16:47.519> like<00:16:47.820> a<00:16:48.120> like<00:16:48.480> a 00:16:48.829 --> 00:16:48.839 align:start position:0% uh the chain something like a like a 00:16:48.839 --> 00:16:51.530 align:start position:0% uh the chain something like a like a block<00:16:49.139> Explorer<00:16:49.740> kind<00:16:50.040> of<00:16:50.160> kind<00:16:50.579> of<00:16:50.759> thing 00:16:51.530 --> 00:16:51.540 align:start position:0% block Explorer kind of kind of thing 00:16:51.540 --> 00:16:53.150 align:start position:0% block Explorer kind of kind of thing um<00:16:51.600> but<00:16:51.779> I<00:16:51.959> would<00:16:52.019> say<00:16:52.139> like<00:16:52.320> a<00:16:52.620> lot<00:16:52.980> of<00:16:53.100> the 00:16:53.150 --> 00:16:53.160 align:start position:0% um but I would say like a lot of the 00:16:53.160 --> 00:16:55.370 align:start position:0% um but I would say like a lot of the metrics<00:16:53.639> are<00:16:54.480> specific<00:16:54.899> to<00:16:55.199> the 00:16:55.370 --> 00:16:55.380 align:start position:0% metrics are specific to the 00:16:55.380 --> 00:16:57.410 align:start position:0% metrics are specific to the implementation<00:16:55.860> so<00:16:56.279> and<00:16:56.820> those<00:16:56.940> are<00:16:57.120> the<00:16:57.180> ones 00:16:57.410 --> 00:16:57.420 align:start position:0% implementation so and those are the ones 00:16:57.420 --> 00:17:00.230 align:start position:0% implementation so and those are the ones that<00:16:57.540> are<00:16:57.660> I<00:16:57.899> think<00:16:58.079> probably<00:16:58.560> more<00:16:58.860> helpful 00:17:00.230 --> 00:17:00.240 align:start position:0% that are I think probably more helpful 00:17:00.240 --> 00:17:02.449 align:start position:0% that are I think probably more helpful um<00:17:00.300> just<00:17:00.540> the<00:17:00.899> internal<00:17:01.380> caches<00:17:01.800> and<00:17:02.279> like 00:17:02.449 --> 00:17:02.459 align:start position:0% um just the internal caches and like 00:17:02.459 --> 00:17:04.610 align:start position:0% um just the internal caches and like timing<00:17:02.940> timings<00:17:03.720> of<00:17:03.959> things<00:17:04.140> that<00:17:04.500> we're 00:17:04.610 --> 00:17:04.620 align:start position:0% timing timings of things that we're 00:17:04.620 --> 00:17:06.890 align:start position:0% timing timings of things that we're really<00:17:04.860> really<00:17:05.100> wanting<00:17:05.459> to<00:17:05.579> get<00:17:05.640> right<00:17:05.819> hi<00:17:06.419> uh 00:17:06.890 --> 00:17:06.900 align:start position:0% really really wanting to get right hi uh 00:17:06.900 --> 00:17:08.329 align:start position:0% really really wanting to get right hi uh I<00:17:07.079> wanted<00:17:07.140> to<00:17:07.380> know<00:17:07.500> if<00:17:07.559> you<00:17:07.799> think<00:17:08.100> these 00:17:08.329 --> 00:17:08.339 align:start position:0% I wanted to know if you think these 00:17:08.339 --> 00:17:10.250 align:start position:0% I wanted to know if you think these metrics<00:17:08.760> are<00:17:08.939> useful<00:17:09.240> for<00:17:09.540> other<00:17:09.720> clients<00:17:10.079> and 00:17:10.250 --> 00:17:10.260 align:start position:0% metrics are useful for other clients and 00:17:10.260 --> 00:17:12.409 align:start position:0% metrics are useful for other clients and not<00:17:10.500> just<00:17:10.620> the<00:17:10.860> typescript<00:17:11.220> one<00:17:11.579> and<00:17:12.059> if<00:17:12.179> so 00:17:12.409 --> 00:17:12.419 align:start position:0% not just the typescript one and if so 00:17:12.419 --> 00:17:14.510 align:start position:0% not just the typescript one and if so would<00:17:12.780> you<00:17:12.959> maybe<00:17:13.079> consider<00:17:13.799> like<00:17:14.160> you<00:17:14.400> know 00:17:14.510 --> 00:17:14.520 align:start position:0% would you maybe consider like you know 00:17:14.520 --> 00:17:15.949 align:start position:0% would you maybe consider like you know putting<00:17:14.819> them<00:17:14.880> in<00:17:15.059> the<00:17:15.179> public<00:17:15.299> domain<00:17:15.720> so 00:17:15.949 --> 00:17:15.959 align:start position:0% putting them in the public domain so 00:17:15.959 --> 00:17:17.569 align:start position:0% putting them in the public domain so like<00:17:16.140> anybody<00:17:16.439> can<00:17:16.679> see<00:17:16.860> them<00:17:16.980> and<00:17:17.160> apply<00:17:17.400> that 00:17:17.569 --> 00:17:17.579 align:start position:0% like anybody can see them and apply that 00:17:17.579 --> 00:17:19.010 align:start position:0% like anybody can see them and apply that to<00:17:17.819> new<00:17:17.939> clients<00:17:18.240> they<00:17:18.419> might<00:17:18.600> be<00:17:18.720> considering 00:17:19.010 --> 00:17:19.020 align:start position:0% to new clients they might be considering 00:17:19.020 --> 00:17:21.350 align:start position:0% to new clients they might be considering building<00:17:19.260> right<00:17:19.860> I<00:17:20.160> know<00:17:20.280> that<00:17:21.059> um<00:17:21.059> other 00:17:21.350 --> 00:17:21.360 align:start position:0% building right I know that um other 00:17:21.360 --> 00:17:23.569 align:start position:0% building right I know that um other clients<00:17:21.720> do<00:17:21.900> have<00:17:22.020> metrics<00:17:22.559> and<00:17:22.919> I<00:17:23.280> I<00:17:23.400> think 00:17:23.569 --> 00:17:23.579 align:start position:0% clients do have metrics and I I think 00:17:23.579 --> 00:17:25.429 align:start position:0% clients do have metrics and I I think like<00:17:23.819> the<00:17:24.000> ones<00:17:24.179> that<00:17:24.360> are<00:17:24.480> more<00:17:24.660> user<00:17:25.020> focused 00:17:25.429 --> 00:17:25.439 align:start position:0% like the ones that are more user focused 00:17:25.439 --> 00:17:27.650 align:start position:0% like the ones that are more user focused uh<00:17:26.339> those<00:17:26.579> are<00:17:26.760> the<00:17:27.000> ones<00:17:27.179> that<00:17:27.299> look<00:17:27.480> more 00:17:27.650 --> 00:17:27.660 align:start position:0% uh those are the ones that look more 00:17:27.660 --> 00:17:30.110 align:start position:0% uh those are the ones that look more like<00:17:27.839> a<00:17:28.079> block<00:17:28.380> Explorer<00:17:28.980> and<00:17:29.580> maybe<00:17:29.940> with 00:17:30.110 --> 00:17:30.120 align:start position:0% like a block Explorer and maybe with 00:17:30.120 --> 00:17:31.909 align:start position:0% like a block Explorer and maybe with some<00:17:30.299> like<00:17:30.419> validator<00:17:30.840> like<00:17:31.320> showing<00:17:31.799> your 00:17:31.909 --> 00:17:31.919 align:start position:0% some like validator like showing your 00:17:31.919 --> 00:17:34.130 align:start position:0% some like validator like showing your balance<00:17:32.160> ticking<00:17:32.520> up<00:17:32.700> over<00:17:32.940> time<00:17:33.240> and<00:17:33.900> I<00:17:34.020> think 00:17:34.130 --> 00:17:34.140 align:start position:0% balance ticking up over time and I think 00:17:34.140 --> 00:17:35.750 align:start position:0% balance ticking up over time and I think those<00:17:34.320> are<00:17:34.440> those<00:17:34.679> are<00:17:34.860> public<00:17:35.039> and<00:17:35.340> I<00:17:35.520> and 00:17:35.750 --> 00:17:35.760 align:start position:0% those are those are public and I and 00:17:35.760 --> 00:17:37.490 align:start position:0% those are those are public and I and there<00:17:36.000> is<00:17:36.059> an<00:17:36.240> effort<00:17:36.539> to<00:17:36.900> standardize 00:17:37.490 --> 00:17:37.500 align:start position:0% there is an effort to standardize 00:17:37.500 --> 00:17:41.450 align:start position:0% there is an effort to standardize metrics<00:17:38.039> across<00:17:38.600> the<00:17:39.600> consensus<00:17:40.260> clients<00:17:41.100> so 00:17:41.450 --> 00:17:41.460 align:start position:0% metrics across the consensus clients so 00:17:41.460 --> 00:17:43.370 align:start position:0% metrics across the consensus clients so I<00:17:41.580> think<00:17:41.820> some<00:17:41.940> of<00:17:42.120> that's<00:17:42.240> happening 00:17:43.370 --> 00:17:43.380 align:start position:0% I think some of that's happening 00:17:43.380 --> 00:17:45.409 align:start position:0% I think some of that's happening um<00:17:43.460> but<00:17:44.460> again<00:17:44.580> to<00:17:44.820> the<00:17:44.940> to<00:17:45.120> the<00:17:45.299> previous 00:17:45.409 --> 00:17:45.419 align:start position:0% um but again to the to the previous 00:17:45.419 --> 00:17:48.110 align:start position:0% um but again to the to the previous question<00:17:46.160> a<00:17:47.160> lot<00:17:47.220> of<00:17:47.280> the<00:17:47.400> metrics<00:17:47.820> will 00:17:48.110 --> 00:17:48.120 align:start position:0% question a lot of the metrics will 00:17:48.120 --> 00:17:50.450 align:start position:0% question a lot of the metrics will always<00:17:48.299> be<00:17:48.660> implementation<00:17:49.260> specific<00:17:49.740> in 00:17:50.450 --> 00:17:50.460 align:start position:0% always be implementation specific in 00:17:50.460 --> 00:17:52.970 align:start position:0% always be implementation specific in terms<00:17:50.640> of<00:17:50.720> roles<00:17:51.720> and<00:17:51.960> you<00:17:52.200> know<00:17:52.320> in<00:17:52.500> team 00:17:52.970 --> 00:17:52.980 align:start position:0% terms of roles and you know in team 00:17:52.980 --> 00:17:54.830 align:start position:0% terms of roles and you know in team members<00:17:53.340> who<00:17:53.640> generally<00:17:54.000> looks<00:17:54.419> at<00:17:54.480> which 00:17:54.830 --> 00:17:54.840 align:start position:0% members who generally looks at which 00:17:54.840 --> 00:17:57.110 align:start position:0% members who generally looks at which metrics<00:17:55.320> and<00:17:55.440> which<00:17:55.620> graphs 00:17:57.110 --> 00:17:57.120 align:start position:0% metrics and which graphs 00:17:57.120 --> 00:18:00.169 align:start position:0% metrics and which graphs um<00:17:57.179> just<00:17:57.780> kind<00:17:57.960> of<00:17:58.080> to<00:17:58.200> get<00:17:58.320> an<00:17:58.440> idea<00:17:58.679> of<00:17:58.860> uh<00:17:59.820> you 00:18:00.169 --> 00:18:00.179 align:start position:0% um just kind of to get an idea of uh you 00:18:00.179 --> 00:18:02.270 align:start position:0% um just kind of to get an idea of uh you know<00:18:00.240> which<00:18:00.840> metrics<00:18:01.500> and<00:18:01.559> graphs<00:18:01.919> are<00:18:02.039> useful 00:18:02.270 --> 00:18:02.280 align:start position:0% know which metrics and graphs are useful 00:18:02.280 --> 00:18:04.250 align:start position:0% know which metrics and graphs are useful to<00:18:02.460> which<00:18:02.640> team<00:18:02.940> members<00:18:03.299> or<00:18:03.660> depending<00:18:04.140> on 00:18:04.250 --> 00:18:04.260 align:start position:0% to which team members or depending on 00:18:04.260 --> 00:18:06.770 align:start position:0% to which team members or depending on like<00:18:04.320> the<00:18:04.500> role<00:18:04.740> or<00:18:05.039> the<00:18:05.280> the<00:18:05.480> type<00:18:06.480> of<00:18:06.660> work 00:18:06.770 --> 00:18:06.780 align:start position:0% like the role or the the type of work 00:18:06.780 --> 00:18:08.930 align:start position:0% like the role or the the type of work that<00:18:06.960> they<00:18:07.140> do<00:18:07.260> you're<00:18:07.980> asking<00:18:08.280> if<00:18:08.460> if<00:18:08.700> only 00:18:08.930 --> 00:18:08.940 align:start position:0% that they do you're asking if if only 00:18:08.940 --> 00:18:11.270 align:start position:0% that they do you're asking if if only certain<00:18:09.299> members<00:18:09.720> contribute<00:18:10.260> to<00:18:10.380> these<00:18:11.100> are 00:18:11.270 --> 00:18:11.280 align:start position:0% certain members contribute to these are 00:18:11.280 --> 00:18:13.430 align:start position:0% certain members contribute to these are more<00:18:11.520> like<00:18:11.700> the<00:18:12.059> uh<00:18:12.360> looking<00:18:12.660> at<00:18:13.020> the<00:18:13.260> the 00:18:13.430 --> 00:18:13.440 align:start position:0% more like the uh looking at the the 00:18:13.440 --> 00:18:16.370 align:start position:0% more like the uh looking at the the graphs<00:18:13.919> and<00:18:14.340> the<00:18:14.520> browsing<00:18:15.179> the<00:18:15.419> the<00:18:15.539> metrics 00:18:16.370 --> 00:18:16.380 align:start position:0% graphs and the browsing the the metrics 00:18:16.380 --> 00:18:17.990 align:start position:0% graphs and the browsing the the metrics themselves<00:18:16.740> like<00:18:17.340> you<00:18:17.520> know<00:18:17.640> for<00:18:17.760> example 00:18:17.990 --> 00:18:18.000 align:start position:0% themselves like you know for example 00:18:18.000 --> 00:18:19.310 align:start position:0% themselves like you know for example there<00:18:18.179> are<00:18:18.299> lots<00:18:18.539> of<00:18:18.600> metrics<00:18:19.020> so<00:18:19.140> not 00:18:19.310 --> 00:18:19.320 align:start position:0% there are lots of metrics so not 00:18:19.320 --> 00:18:21.230 align:start position:0% there are lots of metrics so not everybody<00:18:19.559> looks<00:18:20.039> at<00:18:20.160> everything<00:18:20.460> so<00:18:21.120> you 00:18:21.230 --> 00:18:21.240 align:start position:0% everybody looks at everything so you 00:18:21.240 --> 00:18:23.510 align:start position:0% everybody looks at everything so you know<00:18:21.419> it's<00:18:21.600> more<00:18:22.380> um<00:18:22.380> what<00:18:23.100> you<00:18:23.280> know<00:18:23.400> what 00:18:23.510 --> 00:18:23.520 align:start position:0% know it's more um what you know what 00:18:23.520 --> 00:18:25.310 align:start position:0% know it's more um what you know what team<00:18:23.700> members<00:18:23.940> look<00:18:24.120> at<00:18:24.299> look<00:18:24.539> at<00:18:24.660> what<00:18:24.960> kinds 00:18:25.310 --> 00:18:25.320 align:start position:0% team members look at look at what kinds 00:18:25.320 --> 00:18:28.850 align:start position:0% team members look at look at what kinds of<00:18:25.380> graphs<00:18:25.799> or<00:18:26.160> right<00:18:26.640> so<00:18:27.000> so<00:18:27.299> for<00:18:27.840> any<00:18:28.740> of 00:18:28.850 --> 00:18:28.860 align:start position:0% of graphs or right so so for any of 00:18:28.860 --> 00:18:31.669 align:start position:0% of graphs or right so so for any of these<00:18:29.039> larger<00:18:29.760> feature<00:18:30.539> pull<00:18:30.780> requests<00:18:31.380> we 00:18:31.669 --> 00:18:31.679 align:start position:0% these larger feature pull requests we 00:18:31.679 --> 00:18:34.669 align:start position:0% these larger feature pull requests we have<00:18:31.860> the<00:18:32.280> the<00:18:32.700> person<00:18:33.000> who<00:18:33.480> is<00:18:33.660> authoring<00:18:34.380> or 00:18:34.669 --> 00:18:34.679 align:start position:0% have the the person who is authoring or 00:18:34.679 --> 00:18:37.909 align:start position:0% have the the person who is authoring or kind<00:18:35.160> of<00:18:35.280> owning<00:18:36.000> that<00:18:36.299> that<00:18:36.480> feature<00:18:37.020> they're 00:18:37.909 --> 00:18:37.919 align:start position:0% kind of owning that that feature they're 00:18:37.919 --> 00:18:40.430 align:start position:0% kind of owning that that feature they're responsible<00:18:38.460> for 00:18:40.430 --> 00:18:40.440 align:start position:0% responsible for 00:18:40.440 --> 00:18:43.490 align:start position:0% responsible for um<00:18:40.500> making<00:18:41.039> the<00:18:41.280> case<00:18:41.580> that<00:18:42.480> that<00:18:42.780> that<00:18:42.900> it's 00:18:43.490 --> 00:18:43.500 align:start position:0% um making the case that that that it's 00:18:43.500 --> 00:18:45.169 align:start position:0% um making the case that that that it's correct<00:18:43.860> and<00:18:44.100> then<00:18:44.220> it<00:18:44.340> works<00:18:44.580> as<00:18:44.700> intended 00:18:45.169 --> 00:18:45.179 align:start position:0% correct and then it works as intended 00:18:45.179 --> 00:18:47.570 align:start position:0% correct and then it works as intended and<00:18:45.960> so<00:18:46.200> they're<00:18:46.799> responsible<00:18:47.160> for<00:18:47.400> looking 00:18:47.570 --> 00:18:47.580 align:start position:0% and so they're responsible for looking 00:18:47.580 --> 00:18:48.770 align:start position:0% and so they're responsible for looking at<00:18:47.760> the<00:18:47.820> metrics<00:18:48.179> and<00:18:48.299> building<00:18:48.419> building 00:18:48.770 --> 00:18:48.780 align:start position:0% at the metrics and building building 00:18:48.780 --> 00:18:51.770 align:start position:0% at the metrics and building building that<00:18:49.080> case<00:18:49.260> and<00:18:49.919> then<00:18:50.039> as<00:18:50.220> far<00:18:50.400> as<00:18:50.460> like 00:18:51.770 --> 00:18:51.780 align:start position:0% that case and then as far as like 00:18:51.780 --> 00:18:54.470 align:start position:0% that case and then as far as like um our<00:18:52.140> release<00:18:52.320> process<00:18:52.860> we<00:18:53.700> have 00:18:54.470 --> 00:18:54.480 align:start position:0% um our release process we have 00:18:54.480 --> 00:18:56.270 align:start position:0% um our release process we have um a<00:18:54.960> checklist<00:18:55.260> of<00:18:55.500> basic<00:18:55.799> things<00:18:55.980> that<00:18:56.220> we 00:18:56.270 --> 00:18:56.280 align:start position:0% um a checklist of basic things that we 00:18:56.280 --> 00:18:57.770 align:start position:0% um a checklist of basic things that we look<00:18:56.460> at 00:18:57.770 --> 00:18:57.780 align:start position:0% look at 00:18:57.780 --> 00:19:00.470 align:start position:0% look at um<00:18:58.380> before<00:18:58.919> we<00:18:59.280> before<00:18:59.700> we<00:19:00.120> actually<00:19:00.240> release 00:19:00.470 --> 00:19:00.480 align:start position:0% um before we before we actually release 00:19:00.480 --> 00:19:02.690 align:start position:0% um before we before we actually release we<00:19:00.780> have<00:19:00.900> a<00:19:01.020> testing<00:19:01.320> period<00:19:01.620> so<00:19:01.799> we<00:19:01.980> cut<00:19:02.160> a<00:19:02.340> cut 00:19:02.690 --> 00:19:02.700 align:start position:0% we have a testing period so we cut a cut 00:19:02.700 --> 00:19:05.990 align:start position:0% we have a testing period so we cut a cut a<00:19:02.820> Beta<00:19:03.179> release<00:19:03.299> we<00:19:03.840> deploy<00:19:04.140> it<00:19:04.260> to<00:19:04.380> our<00:19:04.559> Fleet 00:19:05.990 --> 00:19:06.000 align:start position:0% a Beta release we deploy it to our Fleet 00:19:06.000 --> 00:19:07.970 align:start position:0% a Beta release we deploy it to our Fleet um<00:19:06.059> and<00:19:06.660> we<00:19:06.840> watch<00:19:07.020> it<00:19:07.200> for<00:19:07.320> a<00:19:07.440> few<00:19:07.559> days<00:19:07.679> and 00:19:07.970 --> 00:19:07.980 align:start position:0% um and we watch it for a few days and 00:19:07.980 --> 00:19:10.490 align:start position:0% um and we watch it for a few days and then<00:19:08.100> after<00:19:08.220> a<00:19:08.460> few<00:19:08.580> days<00:19:08.700> we 00:19:10.490 --> 00:19:10.500 align:start position:0% then after a few days we 00:19:10.500 --> 00:19:12.470 align:start position:0% then after a few days we um<00:19:10.559> gather<00:19:11.160> you<00:19:11.520> know<00:19:11.580> check<00:19:11.880> do<00:19:12.360> the 00:19:12.470 --> 00:19:12.480 align:start position:0% um gather you know check do the 00:19:12.480 --> 00:19:15.770 align:start position:0% um gather you know check do the checklist<00:19:13.039> and<00:19:14.039> any<00:19:14.760> other<00:19:15.000> kind<00:19:15.299> of<00:19:15.419> ad<00:19:15.539> hoc 00:19:15.770 --> 00:19:15.780 align:start position:0% checklist and any other kind of ad hoc 00:19:15.780 --> 00:19:19.310 align:start position:0% checklist and any other kind of ad hoc things<00:19:16.080> that<00:19:16.380> were<00:19:16.620> were<00:19:17.400> noticing<00:19:18.360> then<00:19:19.020> okay 00:19:19.310 --> 00:19:19.320 align:start position:0% things that were were noticing then okay 00:19:19.320 --> 00:19:21.529 align:start position:0% things that were were noticing then okay so<00:19:19.559> in<00:19:19.740> web<00:19:19.860> 2<00:19:20.100> there's<00:19:20.400> a<00:19:20.700> standard<00:19:21.000> of<00:19:21.179> how<00:19:21.419> to 00:19:21.529 --> 00:19:21.539 align:start position:0% so in web 2 there's a standard of how to 00:19:21.539 --> 00:19:23.930 align:start position:0% so in web 2 there's a standard of how to consume<00:19:21.960> logs<00:19:22.320> metrics<00:19:22.860> and<00:19:22.980> traces<00:19:23.400> and<00:19:23.760> it's 00:19:23.930 --> 00:19:23.940 align:start position:0% consume logs metrics and traces and it's 00:19:23.940 --> 00:19:26.750 align:start position:0% consume logs metrics and traces and it's done<00:19:24.480> by<00:19:24.780> the<00:19:24.960> cloud<00:19:25.200> native<00:19:25.620> uh<00:19:26.340> Foundation 00:19:26.750 --> 00:19:26.760 align:start position:0% done by the cloud native uh Foundation 00:19:26.760 --> 00:19:29.150 align:start position:0% done by the cloud native uh Foundation to<00:19:27.660> make<00:19:27.840> the<00:19:28.200> same<00:19:28.320> pipeline<00:19:28.799> that<00:19:28.919> you<00:19:29.039> have 00:19:29.150 --> 00:19:29.160 align:start position:0% to make the same pipeline that you have 00:19:29.160 --> 00:19:30.950 align:start position:0% to make the same pipeline that you have here<00:19:29.340> more<00:19:30.000> standard<00:19:30.419> in<00:19:30.600> a<00:19:30.720> way<00:19:30.780> that's 00:19:30.950 --> 00:19:30.960 align:start position:0% here more standard in a way that's 00:19:30.960 --> 00:19:33.770 align:start position:0% here more standard in a way that's consumed<00:19:31.440> by<00:19:31.620> other<00:19:32.100> tools<00:19:32.520> as<00:19:32.640> well<00:19:32.820> uh<00:19:33.480> did 00:19:33.770 --> 00:19:33.780 align:start position:0% consumed by other tools as well uh did 00:19:33.780 --> 00:19:36.470 align:start position:0% consumed by other tools as well uh did you<00:19:33.960> take<00:19:34.620> a<00:19:34.740> look<00:19:34.860> at<00:19:35.100> that<00:19:35.280> I<00:19:35.940> recommend<00:19:36.360> it 00:19:36.470 --> 00:19:36.480 align:start position:0% you take a look at that I recommend it 00:19:36.480 --> 00:19:39.890 align:start position:0% you take a look at that I recommend it if<00:19:36.660> not<00:19:37.340> the<00:19:38.340> the<00:19:38.340> standardization<00:19:39.299> effort<00:19:39.720> is 00:19:39.890 --> 00:19:39.900 align:start position:0% if not the the standardization effort is 00:19:39.900 --> 00:19:43.490 align:start position:0% if not the the standardization effort is to<00:19:40.080> to<00:19:40.380> build<00:19:40.740> out<00:19:41.240> some<00:19:42.240> kind<00:19:42.419> of<00:19:42.539> Baseline<00:19:43.260> of 00:19:43.490 --> 00:19:43.500 align:start position:0% to to build out some kind of Baseline of 00:19:43.500 --> 00:19:46.250 align:start position:0% to to build out some kind of Baseline of functionality<00:19:44.100> like<00:19:44.520> you<00:19:44.940> can<00:19:45.059> have<00:19:45.179> one<00:19:45.720> one 00:19:46.250 --> 00:19:46.260 align:start position:0% functionality like you can have one one 00:19:46.260 --> 00:19:48.490 align:start position:0% functionality like you can have one one Prometheus<00:19:47.039> and<00:19:47.220> grafana<00:19:47.640> running<00:19:48.000> and<00:19:48.360> then 00:19:48.490 --> 00:19:48.500 align:start position:0% Prometheus and grafana running and then 00:19:48.500 --> 00:19:51.289 align:start position:0% Prometheus and grafana running and then whichever<00:19:49.500> client<00:19:49.919> you<00:19:50.220> want<00:19:50.460> plugged<00:19:51.179> in 00:19:51.289 --> 00:19:51.299 align:start position:0% whichever client you want plugged in 00:19:51.299 --> 00:19:53.390 align:start position:0% whichever client you want plugged in here<00:19:51.539> you<00:19:52.080> kind<00:19:52.260> of<00:19:52.380> have<00:19:52.500> some<00:19:52.799> guarantees 00:19:53.390 --> 00:19:53.400 align:start position:0% here you kind of have some guarantees 00:19:53.400 --> 00:19:55.669 align:start position:0% here you kind of have some guarantees like<00:19:53.700> uh<00:19:54.480> it's<00:19:54.660> not<00:19:54.780> breaking<00:19:55.140> and<00:19:55.320> like<00:19:55.500> it's 00:19:55.669 --> 00:19:55.679 align:start position:0% like uh it's not breaking and like it's 00:19:55.679 --> 00:19:57.950 align:start position:0% like uh it's not breaking and like it's it's<00:19:55.980> kind<00:19:56.220> of<00:19:56.340> working<00:19:56.460> so<00:19:56.880> yeah<00:19:57.179> that's 00:19:57.950 --> 00:19:57.960 align:start position:0% it's kind of working so yeah that's 00:19:57.960 --> 00:20:00.289 align:start position:0% it's kind of working so yeah that's exactly<00:19:58.320> what<00:19:58.500> open<00:19:58.860> Telemetry<00:19:59.520> does<00:19:59.700> so<00:19:59.940> take 00:20:00.289 --> 00:20:00.299 align:start position:0% exactly what open Telemetry does so take 00:20:00.299 --> 00:20:02.720 align:start position:0% exactly what open Telemetry does so take a<00:20:00.360> look<00:20:00.480> at<00:20:00.600> it